From 30e63fb6e0cfb5140022bdda6ea3cef180d986c4 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 16 Aug 2022 12:27:30 +0200 Subject: [PATCH 01/11] test(deps): update dependency org.mockito:mockito-core to v4.7.0 (#2231) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * test(deps): update dependency org.mockito:mockito-core to v4.7.0 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot --- README.md | 6 +++--- pom.xml | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index bc481d2de1..d71364f416 100644 --- a/README.md +++ b/README.md @@ -52,20 +52,20 @@ If you are using Maven without BOM, add this to your dependencies: If you are using Gradle 5.x or later, add this to your dependencies: ```Groovy -implementation platform('com.google.cloud:libraries-bom:26.0.0') +implementation platform('com.google.cloud:libraries-bom:26.1.0') implementation 'com.google.cloud:google-cloud-bigquery' ``` If you are using Gradle without BOM, add this to your dependencies: ```Groovy -implementation 'com.google.cloud:google-cloud-bigquery:2.14.5' +implementation 'com.google.cloud:google-cloud-bigquery:2.14.6' ``` If you are using SBT, add this to your dependencies: ```Scala -libraryDependencies += "com.google.cloud" % "google-cloud-bigquery" % "2.14.5" +libraryDependencies += "com.google.cloud" % "google-cloud-bigquery" % "2.14.6" ``` ## Authentication diff --git a/pom.xml b/pom.xml index 1f36b5ecbc..9fed16acdd 100644 --- a/pom.xml +++ b/pom.xml @@ -170,7 +170,7 @@ org.mockito mockito-core - 4.6.1 + 4.7.0 test From f4b6ce75e12546560e8bf5b4a81455ac2d4ef633 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 16 Aug 2022 12:27:54 +0200 Subject: [PATCH 02/11] chore(deps): update dependency com.google.cloud:libraries-bom to v26.1.0 (#2230) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(deps): update dependency com.google.cloud:libraries-bom to v26.1.0 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot --- README.md | 2 +- samples/native-image-sample/pom.xml | 2 +- samples/snippets/pom.xml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index d71364f416..2e749205c6 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ See https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google com.google.cloud libraries-bom - 26.0.0 + 26.1.0 pom import diff --git a/samples/native-image-sample/pom.xml b/samples/native-image-sample/pom.xml index 702f489522..0f4267c2a8 100644 --- a/samples/native-image-sample/pom.xml +++ b/samples/native-image-sample/pom.xml @@ -39,7 +39,7 @@ com.google.cloud libraries-bom - 26.0.0 + 26.1.0 pom import diff --git a/samples/snippets/pom.xml b/samples/snippets/pom.xml index 1711a5a707..310417efd9 100644 --- a/samples/snippets/pom.xml +++ b/samples/snippets/pom.xml @@ -47,7 +47,7 @@ com.google.cloud libraries-bom - 26.0.0 + 26.1.0 pom import From 286125e7488b880f2c8010408c034891387c908e Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 16 Aug 2022 12:28:15 +0200 Subject: [PATCH 03/11] chore(deps): update dependency com.google.cloud:google-cloud-bigquery to v2.14.6 (#2229) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(deps): update dependency com.google.cloud:google-cloud-bigquery to v2.14.6 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot --- README.md | 2 +- samples/install-without-bom/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 2e749205c6..c2af592689 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ If you are using Maven without BOM, add this to your dependencies: com.google.cloud google-cloud-bigquery - 2.14.5 + 2.14.6 ``` diff --git a/samples/install-without-bom/pom.xml b/samples/install-without-bom/pom.xml index 00b02fac60..14727dab21 100644 --- a/samples/install-without-bom/pom.xml +++ b/samples/install-without-bom/pom.xml @@ -45,7 +45,7 @@ com.google.cloud google-cloud-bigquery - 2.14.5 + 2.14.6 From abad7f924113e917c98e741c37494a8aa09fe9c0 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 16 Aug 2022 18:18:16 +0200 Subject: [PATCH 04/11] build(deps): update dependency org.apache.maven.plugins:maven-project-info-reports-plugin to v3.4.1 (#2233) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [org.apache.maven.plugins:maven-project-info-reports-plugin](https://maven.apache.org/plugins/) | `3.4.0` -> `3.4.1` | [![age](https://badges.renovateapi.com/packages/maven/org.apache.maven.plugins:maven-project-info-reports-plugin/3.4.1/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/org.apache.maven.plugins:maven-project-info-reports-plugin/3.4.1/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/org.apache.maven.plugins:maven-project-info-reports-plugin/3.4.1/compatibility-slim/3.4.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/org.apache.maven.plugins:maven-project-info-reports-plugin/3.4.1/confidence-slim/3.4.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, click this checkbox. ⚠ **Warning**: custom changes will be lost. --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-bigquery). --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 9fed16acdd..facd5716bb 100644 --- a/pom.xml +++ b/pom.xml @@ -206,7 +206,7 @@ org.apache.maven.plugins maven-project-info-reports-plugin - 3.4.0 + 3.4.1 From e4766acd179e0585aadcdbf09bab949482374e98 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 16 Aug 2022 22:18:22 +0200 Subject: [PATCH 05/11] build(deps): update dependency org.apache.maven.plugins:maven-javadoc-plugin to v3.4.1 (#2232) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [org.apache.maven.plugins:maven-javadoc-plugin](https://maven.apache.org/plugins/) ([source](https://togithub.com/apache/maven-javadoc-plugin)) | `3.4.0` -> `3.4.1` | [![age](https://badges.renovateapi.com/packages/maven/org.apache.maven.plugins:maven-javadoc-plugin/3.4.1/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/org.apache.maven.plugins:maven-javadoc-plugin/3.4.1/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/org.apache.maven.plugins:maven-javadoc-plugin/3.4.1/compatibility-slim/3.4.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/org.apache.maven.plugins:maven-javadoc-plugin/3.4.1/confidence-slim/3.4.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, click this checkbox. ⚠ **Warning**: custom changes will be lost. --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-bigquery). --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index facd5716bb..a9e531a548 100644 --- a/pom.xml +++ b/pom.xml @@ -233,7 +233,7 @@ org.apache.maven.plugins maven-javadoc-plugin - 3.4.0 + 3.4.1 html From 01f826db243cac69fde09312be67f292a8b3dde3 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 17 Aug 2022 15:52:14 +0200 Subject: [PATCH 06/11] chore(deps): update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.20.0 (#2237) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [com.google.cloud:google-cloud-bigquerystorage-bom](https://togithub.com/googleapis/java-bigquerystorage) | `2.19.1` -> `2.20.0` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigquerystorage-bom/2.20.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigquerystorage-bom/2.20.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigquerystorage-bom/2.20.0/compatibility-slim/2.19.1)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-bigquerystorage-bom/2.20.0/confidence-slim/2.19.1)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/java-bigquerystorage ### [`v2.20.0`](https://togithub.com/googleapis/java-bigquerystorage/blob/HEAD/CHANGELOG.md#​2200-httpsgithubcomgoogleapisjava-bigquerystoragecomparev2191v2200-2022-08-17) [Compare Source](https://togithub.com/googleapis/java-bigquerystorage/compare/v2.19.1...v2.20.0) ##### Features - introduce InflightLimitExceededException ([#​1746](https://togithub.com/googleapis/java-bigquerystorage/issues/1746)) ([449353b](https://togithub.com/googleapis/java-bigquerystorage/commit/449353bd3b1fc78d46990c8203e1607d83f7ad03)) ##### Bug Fixes - Add documentation to Exceptions ([#​1745](https://togithub.com/googleapis/java-bigquerystorage/issues/1745)) ([3bc7aca](https://togithub.com/googleapis/java-bigquerystorage/commit/3bc7acaa4a110a0ce487e3976b5fd4ddb865ea24)) - Add missing field error to row error message ([#​1752](https://togithub.com/googleapis/java-bigquerystorage/issues/1752)) ([186d213](https://togithub.com/googleapis/java-bigquerystorage/commit/186d2135cb7e18410b89c93e909b03dc8e92ac6c)) - Close based on whether connection is active, not status variable ([#​1750](https://togithub.com/googleapis/java-bigquerystorage/issues/1750)) ([ce7f3e0](https://togithub.com/googleapis/java-bigquerystorage/commit/ce7f3e04046be6b4c1321be877034f5d0ab007f4)) ##### Dependencies - fixing the test scope dependency to runtime ([#​1742](https://togithub.com/googleapis/java-bigquerystorage/issues/1742)) ([878020b](https://togithub.com/googleapis/java-bigquerystorage/commit/878020b3667040aef1b8d610b37cd678ef932870)) - update dependency com.google.cloud:google-cloud-bigquery to v2.14.4 ([#​1740](https://togithub.com/googleapis/java-bigquerystorage/issues/1740)) ([f842d51](https://togithub.com/googleapis/java-bigquerystorage/commit/f842d517b70847b8e67359257a213129586513f6)) - update dependency com.google.cloud:google-cloud-bigquery to v2.14.6 ([#​1747](https://togithub.com/googleapis/java-bigquerystorage/issues/1747)) ([329bd9d](https://togithub.com/googleapis/java-bigquerystorage/commit/329bd9da268f48a4d18158845895b07a43c766ab))
--- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-bigquery). --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index a9e531a548..378615d693 100644 --- a/pom.xml +++ b/pom.xml @@ -72,7 +72,7 @@ com.google.cloud google-cloud-bigquerystorage-bom - 2.19.1 + 2.20.0 pom import From de313bdaf5bec240723279fbe331561ac9eeb2ca Mon Sep 17 00:00:00 2001 From: Neenu Shaji Date: Fri, 19 Aug 2022 07:00:28 -0400 Subject: [PATCH 07/11] chore: add set user agent sample (#2239) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore: add set user agent sample * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * chore: add comments and fix variable names Co-authored-by: Owl Bot --- README.md | 1 + .../com/example/bigquery/SetUserAgent.java | 61 ++++++++++++++ .../example/bigquery/SetUserAgentTest.java | 79 +++++++++++++++++++ 3 files changed, 141 insertions(+) create mode 100644 samples/snippets/src/main/java/com/example/bigquery/SetUserAgent.java create mode 100644 samples/snippets/src/test/java/com/example/bigquery/SetUserAgentTest.java diff --git a/README.md b/README.md index c2af592689..58b11c9c20 100644 --- a/README.md +++ b/README.md @@ -225,6 +225,7 @@ Samples are in the [`samples/`](https://github.com/googleapis/java-bigquery/tree | Resource Clean Up | [source code](https://github.com/googleapis/java-bigquery/blob/main/samples/snippets/src/main/java/com/example/bigquery/ResourceCleanUp.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-bigquery&page=editor&open_in_editor=samples/snippets/src/main/java/com/example/bigquery/ResourceCleanUp.java) | | Run Legacy Query | [source code](https://github.com/googleapis/java-bigquery/blob/main/samples/snippets/src/main/java/com/example/bigquery/RunLegacyQuery.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-bigquery&page=editor&open_in_editor=samples/snippets/src/main/java/com/example/bigquery/RunLegacyQuery.java) | | Save Query To Table | [source code](https://github.com/googleapis/java-bigquery/blob/main/samples/snippets/src/main/java/com/example/bigquery/SaveQueryToTable.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-bigquery&page=editor&open_in_editor=samples/snippets/src/main/java/com/example/bigquery/SaveQueryToTable.java) | +| Set User Agent | [source code](https://github.com/googleapis/java-bigquery/blob/main/samples/snippets/src/main/java/com/example/bigquery/SetUserAgent.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-bigquery&page=editor&open_in_editor=samples/snippets/src/main/java/com/example/bigquery/SetUserAgent.java) | | Simple App | [source code](https://github.com/googleapis/java-bigquery/blob/main/samples/snippets/src/main/java/com/example/bigquery/SimpleApp.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-bigquery&page=editor&open_in_editor=samples/snippets/src/main/java/com/example/bigquery/SimpleApp.java) | | Simple Query | [source code](https://github.com/googleapis/java-bigquery/blob/main/samples/snippets/src/main/java/com/example/bigquery/SimpleQuery.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-bigquery&page=editor&open_in_editor=samples/snippets/src/main/java/com/example/bigquery/SimpleQuery.java) | | Table Exists | [source code](https://github.com/googleapis/java-bigquery/blob/main/samples/snippets/src/main/java/com/example/bigquery/TableExists.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-bigquery&page=editor&open_in_editor=samples/snippets/src/main/java/com/example/bigquery/TableExists.java) | diff --git a/samples/snippets/src/main/java/com/example/bigquery/SetUserAgent.java b/samples/snippets/src/main/java/com/example/bigquery/SetUserAgent.java new file mode 100644 index 0000000000..2794305f6a --- /dev/null +++ b/samples/snippets/src/main/java/com/example/bigquery/SetUserAgent.java @@ -0,0 +1,61 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.example.bigquery; + +// [START bigquery_set_user_agent] +import com.google.api.gax.rpc.FixedHeaderProvider; +import com.google.api.gax.rpc.HeaderProvider; +import com.google.auth.oauth2.GoogleCredentials; +import com.google.cloud.bigquery.BigQuery; +import com.google.cloud.bigquery.BigQueryOptions; +import com.google.common.collect.ImmutableMap; +import java.io.IOException; + +public class SetUserAgent { + + private static final String USER_AGENT_HEADER = "user-agent"; + + public static void main(String[] args) throws IOException { + // TODO(developer): Replace these variables before running the sample. + String projectId = "my-project-id"; + String customUserAgentValue = "my-custom-user-agent-value"; + setUserAgent(projectId, customUserAgentValue); + } + + public static void setUserAgent(String projectId, String customUserAgentValue) + throws IOException { + // Setup the credentials + GoogleCredentials googleCredentials = GoogleCredentials.getApplicationDefault(); + + // Initialize the HeaderProvider object with custom user agent value + HeaderProvider headerProvider = + FixedHeaderProvider.create(ImmutableMap.of(USER_AGENT_HEADER, customUserAgentValue)); + + // Initialize client that will be used to send requests. This client only needs to be created + // once, and can be reused for multiple requests. + BigQuery bigQuery = + BigQueryOptions.newBuilder() + .setProjectId(projectId) + .setCredentials(googleCredentials) + .setHeaderProvider(headerProvider) + .build() + .getService(); + + System.out.println(bigQuery.getOptions().getUserAgent()); + } +} +// [END bigquery_set_user_agent] diff --git a/samples/snippets/src/test/java/com/example/bigquery/SetUserAgentTest.java b/samples/snippets/src/test/java/com/example/bigquery/SetUserAgentTest.java new file mode 100644 index 0000000000..6c5c9cf6d3 --- /dev/null +++ b/samples/snippets/src/test/java/com/example/bigquery/SetUserAgentTest.java @@ -0,0 +1,79 @@ +/* + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.example.bigquery; + +import static com.google.common.truth.Truth.assertThat; +import static junit.framework.TestCase.assertNotNull; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.PrintStream; +import java.util.UUID; +import java.util.logging.Level; +import java.util.logging.Logger; +import org.junit.After; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; + +public class SetUserAgentTest { + + private final Logger log = Logger.getLogger(this.getClass().getName()); + private String customUserAgentValue; + private ByteArrayOutputStream bout; + private PrintStream out; + private PrintStream originalPrintStream; + + private static final String PROJECT_ID = requireEnvVar("GOOGLE_CLOUD_PROJECT"); + + private static String requireEnvVar(String varName) { + String value = System.getenv(varName); + assertNotNull( + "Environment variable " + varName + " is required to perform these tests.", + System.getenv(varName)); + return value; + } + + @BeforeClass + public static void checkRequirements() { + requireEnvVar("GOOGLE_CLOUD_PROJECT"); + } + + @Before + public void setUp() { + customUserAgentValue = "CUSTOM_USER_AGENT_" + UUID.randomUUID().toString().substring(0, 8); + bout = new ByteArrayOutputStream(); + out = new PrintStream(bout); + originalPrintStream = System.out; + System.setOut(out); + } + + @After + public void tearDown() { + // Clean up + // restores print statements in the original method + System.out.flush(); + System.setOut(originalPrintStream); + log.log(Level.INFO, "\n" + bout.toString()); + } + + @Test + public void setUserAgentTest() throws IOException { + SetUserAgent.setUserAgent(PROJECT_ID, customUserAgentValue); + assertThat(bout.toString()).contains("CUSTOM_USER_AGENT_"); + } +} From 487656973fe3e06d838c1b495ac024ab2c6810f6 Mon Sep 17 00:00:00 2001 From: Prashant Mishra <11733935+prash-mi@users.noreply.github.com> Date: Tue, 23 Aug 2022 07:44:47 +0530 Subject: [PATCH 08/11] fix: table-not-found issue with executeSelect while running long queries (#2222) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Added exponential-back-off to create read session to avoid table-not-found error * Added testForTableNotFound IT * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Set recordCnt to 5Mil * Add polling logic @ getQueryResultsFirstPage, Removed retrial logic on table_not_found * Removed getTableNotFoundRetrySettings * Updated getQueryResultsWithRowLimit - Added timeoutMs param * Updated testGetQueryResultsFirstPage * Updated getQueryResultsWithRowLimit - Add timeoutMs * Updated getQueryResultsFirstPage - Modified polling logic and refactor * Removed prev differences. Add getQueryResultsWithRowLimit * Removed prev differences. Add getQueryResultsWithRowLimit Co-authored-by: Owl Bot --- .../clirr-ignored-differences.xml | 16 ++-- .../google/cloud/bigquery/ConnectionImpl.java | 74 ++++++++++++------- .../cloud/bigquery/spi/v2/BigQueryRpc.java | 2 +- .../bigquery/spi/v2/HttpBigQueryRpc.java | 3 +- .../cloud/bigquery/ConnectionImplTest.java | 12 ++- .../bigquery/it/ITNightlyBigQueryTest.java | 62 ++++++++++++++++ 6 files changed, 133 insertions(+), 36 deletions(-) diff --git a/google-cloud-bigquery/clirr-ignored-differences.xml b/google-cloud-bigquery/clirr-ignored-differences.xml index 1d3d826812..2ad26f9464 100644 --- a/google-cloud-bigquery/clirr-ignored-differences.xml +++ b/google-cloud-bigquery/clirr-ignored-differences.xml @@ -2,14 +2,16 @@ - - 7012 - com/google/cloud/bigquery/LoadConfiguration - java.util.List getDecimalTargetTypes() + + 7004 + com/google/cloud/bigquery/spi/v2/BigQueryRpc + com.google.api.services.bigquery.model.GetQueryResultsResponse getQueryResultsWithRowLimit(java.lang.String, java.lang.String, java.lang.String, java.lang.Integer) + getQueryResultsWithRowLimit is just used by ConnectionImpl at the moment so it should be fine to update the signature instead of writing an overloaded method - 7012 - com/google/cloud/bigquery/LoadConfiguration$Builder - com.google.cloud.bigquery.LoadConfiguration$Builder setDecimalTargetTypes(java.util.List) + 7004 + com/google/cloud/bigquery/spi/v2/HttpBigQueryRpc + com.google.api.services.bigquery.model.GetQueryResultsResponse getQueryResultsWithRowLimit(java.lang.String, java.lang.String, java.lang.String, java.lang.Integer) + getQueryResultsWithRowLimit is just used by ConnectionImpl at the moment so it should be fine to update the signature instead of writing an overloaded method \ No newline at end of file diff --git a/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ConnectionImpl.java b/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ConnectionImpl.java index 3a9937a074..da79bd6ba1 100644 --- a/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ConnectionImpl.java +++ b/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ConnectionImpl.java @@ -835,8 +835,8 @@ BigQueryResult highThroughPutRead( .setMaxStreamCount(1) // Currently just one stream is allowed // DO a regex check using order by and use multiple streams ; - ReadSession readSession = bqReadClient.createReadSession(builder.build()); + bufferRow = new LinkedBlockingDeque<>(getBufferSize()); Map arrowNameToIndex = new HashMap<>(); // deserialize and populate the buffer async, so that the client isn't blocked @@ -995,33 +995,57 @@ GetQueryResultsResponse getQueryResultsFirstPage(JobId jobId) { jobId.getLocation() == null && bigQueryOptions.getLocation() != null ? bigQueryOptions.getLocation() : jobId.getLocation()); - try { - GetQueryResultsResponse results = - BigQueryRetryHelper.runWithRetries( - () -> - bigQueryRpc.getQueryResultsWithRowLimit( - completeJobId.getProject(), - completeJobId.getJob(), - completeJobId.getLocation(), - connectionSettings.getMaxResultPerPage()), - bigQueryOptions.getRetrySettings(), - BigQueryBaseService.BIGQUERY_EXCEPTION_HANDLER, - bigQueryOptions.getClock(), - retryConfig); - if (results.getErrors() != null) { - List bigQueryErrors = - results.getErrors().stream() - .map(BigQueryError.FROM_PB_FUNCTION) - .collect(Collectors.toList()); - // Throwing BigQueryException since there may be no JobId and we want to stay consistent - // with the case where there there is a HTTP error - throw new BigQueryException(bigQueryErrors); + // Implementing logic to poll the Job's status using getQueryResults as + // we do not get rows, rows count and schema unless the job is complete + // Ref: b/241134681 + // This logic relies on backend for poll and wait.BigQuery guarantees that jobs make forward + // progress (a job won't get stuck in pending forever). + boolean jobComplete = false; + GetQueryResultsResponse results = null; + long timeoutMs = + 60000; // defaulting to 60seconds. TODO(prashant): It should be made user configurable + + while (!jobComplete) { + try { + results = + BigQueryRetryHelper.runWithRetries( + () -> + bigQueryRpc.getQueryResultsWithRowLimit( + completeJobId.getProject(), + completeJobId.getJob(), + completeJobId.getLocation(), + connectionSettings.getMaxResultPerPage(), + timeoutMs), + bigQueryOptions.getRetrySettings(), + BigQueryBaseService.BIGQUERY_EXCEPTION_HANDLER, + bigQueryOptions.getClock(), + retryConfig); + + if (results.getErrors() != null) { + List bigQueryErrors = + results.getErrors().stream() + .map(BigQueryError.FROM_PB_FUNCTION) + .collect(Collectors.toList()); + // Throwing BigQueryException since there may be no JobId, and we want to stay consistent + // with the case where there is a HTTP error + throw new BigQueryException(bigQueryErrors); + } + } catch (BigQueryRetryHelper.BigQueryRetryHelperException e) { + throw BigQueryException.translateAndThrow(e); } - return results; - } catch (BigQueryRetryHelper.BigQueryRetryHelperException e) { - throw BigQueryException.translateAndThrow(e); + jobComplete = results.getJobComplete(); + + // This log msg at Level.FINE might indicate that the job is still running and not stuck for + // very long running jobs. + logger.log( + Level.FINE, + String.format( + "jobComplete: %s , Polling getQueryResults with timeoutMs: %s", + jobComplete, timeoutMs)); } + + return results; } @VisibleForTesting diff --git a/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/BigQueryRpc.java b/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/BigQueryRpc.java index 871590ca4b..eecf5d36f5 100644 --- a/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/BigQueryRpc.java +++ b/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/BigQueryRpc.java @@ -315,7 +315,7 @@ GetQueryResultsResponse getQueryResults( * @throws BigQueryException upon failure */ GetQueryResultsResponse getQueryResultsWithRowLimit( - String projectId, String jobId, String location, Integer preFetchedRowLimit); + String projectId, String jobId, String location, Integer preFetchedRowLimit, Long timeoutMs); /** * Runs a BigQuery SQL query synchronously and returns query results if the query completes within diff --git a/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java b/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java index d6b57a3daf..a9ef3a817a 100644 --- a/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java +++ b/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java @@ -697,7 +697,7 @@ public GetQueryResultsResponse getQueryResults( @Override public GetQueryResultsResponse getQueryResultsWithRowLimit( - String projectId, String jobId, String location, Integer maxResultPerPage) { + String projectId, String jobId, String location, Integer maxResultPerPage, Long timeoutMs) { try { return bigquery .jobs() @@ -705,6 +705,7 @@ public GetQueryResultsResponse getQueryResultsWithRowLimit( .setPrettyPrint(false) .setLocation(location) .setMaxResults(Long.valueOf(maxResultPerPage)) + .setTimeoutMs(timeoutMs) .execute(); } catch (IOException ex) { throw translate(ex); diff --git a/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/ConnectionImplTest.java b/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/ConnectionImplTest.java index 9543ccebf7..4b1b93487b 100644 --- a/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/ConnectionImplTest.java +++ b/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/ConnectionImplTest.java @@ -315,14 +315,22 @@ public void testNextPageTask() throws InterruptedException { @Test public void testGetQueryResultsFirstPage() { when(bigqueryRpcMock.getQueryResultsWithRowLimit( - any(String.class), any(String.class), any(String.class), any(Integer.class))) + any(String.class), + any(String.class), + any(String.class), + any(Integer.class), + any(Long.class))) .thenReturn(GET_QUERY_RESULTS_RESPONSE); GetQueryResultsResponse response = connection.getQueryResultsFirstPage(QUERY_JOB); assertNotNull(response); assertEquals(GET_QUERY_RESULTS_RESPONSE, response); verify(bigqueryRpcMock, times(1)) .getQueryResultsWithRowLimit( - any(String.class), any(String.class), any(String.class), any(Integer.class)); + any(String.class), + any(String.class), + any(String.class), + any(Integer.class), + any(Long.class)); } // calls executeSelect with a nonFast query and exercises createQueryJob diff --git a/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/it/ITNightlyBigQueryTest.java b/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/it/ITNightlyBigQueryTest.java index 73bd21a30d..006c126b66 100644 --- a/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/it/ITNightlyBigQueryTest.java +++ b/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/it/ITNightlyBigQueryTest.java @@ -23,9 +23,11 @@ import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; +import com.google.cloud.ServiceOptions; import com.google.cloud.bigquery.BigQuery; import com.google.cloud.bigquery.BigQueryError; import com.google.cloud.bigquery.BigQueryException; +import com.google.cloud.bigquery.BigQueryOptions; import com.google.cloud.bigquery.BigQueryResult; import com.google.cloud.bigquery.BigQuerySQLException; import com.google.cloud.bigquery.Connection; @@ -60,6 +62,7 @@ import java.util.List; import java.util.Map; import java.util.TimeZone; +import java.util.UUID; import java.util.concurrent.ExecutionException; import java.util.logging.Level; import java.util.logging.Logger; @@ -484,6 +487,65 @@ public void testPositionalParams() assertEquals(MULTI_LIMIT_RECS, cnt); } + @Test + // This testcase reads rows in bulk for a public table to make sure we do not get + // table-not-found exception. Ref: b/241134681 . This exception has been seen while reading data + // in bulk + public void testForTableNotFound() throws SQLException { + int recordCnt = 50000000; // 5Mil + String query = + String.format( + "SELECT * FROM `bigquery-samples.wikipedia_benchmark.Wiki10B` LIMIT %s", recordCnt); + + String dataSet = RemoteBigQueryHelper.generateDatasetName(); + String table = "TAB_" + UUID.randomUUID(); + createDataset(dataSet); + TableId targetTable = + TableId.of( + ServiceOptions.getDefaultProjectId(), + dataSet, + table); // table will be created implicitly + + ConnectionSettings conSet = + ConnectionSettings.newBuilder() + .setUseReadAPI(true) // enable read api + .setDestinationTable(targetTable) + .setAllowLargeResults(true) + .build(); + + Connection connection = + BigQueryOptions.getDefaultInstance().getService().createConnection(conSet); + BigQueryResult bigQueryResultSet = connection.executeSelect(query); + assertNotNull(getResultHashWiki(bigQueryResultSet)); // this iterated through all the rows + assertTrue( + (recordCnt == bigQueryResultSet.getTotalRows()) + || (-1 + == bigQueryResultSet + .getTotalRows())); // either job should return the actual count or -1 if the job + // is still running + try { + deleteTable(dataSet, table); + deleteDataset(dataSet); + } catch (Exception e) { + logger.log( + Level.WARNING, + String.format( + "Error [ %s ] while deleting dataset: %s , table: %s", + e.getMessage(), dataSet, table)); + } + } + + // this iterated through all the rows (just reads the title column) + private Long getResultHashWiki(BigQueryResult bigQueryResultSet) throws SQLException { + ResultSet rs = bigQueryResultSet.getResultSet(); + long hash = 0L; + System.out.print("\n Running"); + while (rs.next()) { + hash += rs.getString("title") == null ? 0 : rs.getString("title").hashCode(); + } + return hash; + } + // asserts the value of each row private static void testForAllDataTypeValues(ResultSet rs, int cnt) throws SQLException { // Testing JSON type From e42a5440bd882a40251380b66c42f8312e7bd740 Mon Sep 17 00:00:00 2001 From: "release-please[bot]" <55107282+release-please[bot]@users.noreply.github.com> Date: Tue, 23 Aug 2022 14:04:01 -0400 Subject: [PATCH 09/11] chore(main): release 2.14.7-SNAPSHOT (#2228) Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> --- benchmark/pom.xml | 2 +- google-cloud-bigquery/pom.xml | 4 ++-- pom.xml | 4 ++-- samples/snapshot/pom.xml | 2 +- versions.txt | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/benchmark/pom.xml b/benchmark/pom.xml index 221cf8ee97..c67b15083c 100644 --- a/benchmark/pom.xml +++ b/benchmark/pom.xml @@ -6,7 +6,7 @@ google-cloud-bigquery-parent com.google.cloud - 2.14.6 + 2.14.7-SNAPSHOT diff --git a/google-cloud-bigquery/pom.xml b/google-cloud-bigquery/pom.xml index 7c701ca4b8..d69dd2bfc2 100644 --- a/google-cloud-bigquery/pom.xml +++ b/google-cloud-bigquery/pom.xml @@ -3,7 +3,7 @@ 4.0.0 com.google.cloud google-cloud-bigquery - 2.14.6 + 2.14.7-SNAPSHOT jar BigQuery https://github.com/googleapis/java-bigquery @@ -11,7 +11,7 @@ com.google.cloud google-cloud-bigquery-parent - 2.14.6 + 2.14.7-SNAPSHOT google-cloud-bigquery diff --git a/pom.xml b/pom.xml index 378615d693..a7177c701a 100644 --- a/pom.xml +++ b/pom.xml @@ -4,7 +4,7 @@ com.google.cloud google-cloud-bigquery-parent pom - 2.14.6 + 2.14.7-SNAPSHOT BigQuery Parent https://github.com/googleapis/java-bigquery @@ -118,7 +118,7 @@ com.google.cloud google-cloud-bigquery - 2.14.6 + 2.14.7-SNAPSHOT diff --git a/samples/snapshot/pom.xml b/samples/snapshot/pom.xml index 535f1b3cb6..c1696f1654 100644 --- a/samples/snapshot/pom.xml +++ b/samples/snapshot/pom.xml @@ -44,7 +44,7 @@ com.google.cloud google-cloud-bigquery - 2.14.6 + 2.14.7-SNAPSHOT diff --git a/versions.txt b/versions.txt index 44c0bb9f27..dcd10147bb 100644 --- a/versions.txt +++ b/versions.txt @@ -1,4 +1,4 @@ # Format: # module:released-version:current-version -google-cloud-bigquery:2.14.6:2.14.6 \ No newline at end of file +google-cloud-bigquery:2.14.6:2.14.7-SNAPSHOT \ No newline at end of file From a449ee6a30ca5706fedeee3ae635c08332b2e424 Mon Sep 17 00:00:00 2001 From: Nikunj Bhartia Date: Wed, 24 Aug 2022 04:12:20 +0530 Subject: [PATCH 10/11] samples / fixing authorize dataset logic by updating source dataset acl (#2189) Fixes#2188 Fixed the logic for authorizing a bigquery user dataset but updating the source dataset acl with that of the user dataset information. --- .../java/com/example/bigquery/AuthorizeDataset.java | 10 +++++----- .../java/com/example/bigquery/AuthorizeDatasetIT.java | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/samples/snippets/src/main/java/com/example/bigquery/AuthorizeDataset.java b/samples/snippets/src/main/java/com/example/bigquery/AuthorizeDataset.java index f7f28c2182..949e583e48 100644 --- a/samples/snippets/src/main/java/com/example/bigquery/AuthorizeDataset.java +++ b/samples/snippets/src/main/java/com/example/bigquery/AuthorizeDataset.java @@ -38,7 +38,7 @@ public static void main(String[] args) { DatasetId.of(projectId, sourceDatasetName), DatasetId.of(projectId, userDatasetName)); } - // This method will update userDataset's ACL with sourceDataset's ACL + // This method will update sourceDataset's ACL with userDataset's ACL public static void authorizeDataset(DatasetId sourceDatasetId, DatasetId userDatasetId) { try { // Initialize client that will be used to send requests. This client only needs to be created @@ -58,12 +58,12 @@ public static void authorizeDataset(DatasetId sourceDatasetId, DatasetId userDat new Acl.DatasetAclEntity(userDatasetId, targetTypes); sourceDatasetAcl.add(Acl.of(userDatasetAclEntity)); - // update the user dataset with source dataset's ACL - Dataset updatedUserDataset = - userDataset.toBuilder().setAcl(sourceDatasetAcl).build().update(); + // update the source dataset with user dataset's ACL + Dataset updatedSourceDataset = + sourceDataset.toBuilder().setAcl(sourceDatasetAcl).build().update(); System.out.printf( - "Dataset %s updated with the added authorization\n", updatedUserDataset.getDatasetId()); + "Dataset %s updated with the added authorization\n", updatedSourceDataset.getDatasetId()); } catch (BigQueryException e) { System.out.println("Dataset Authorization failed due to error: \n" + e); diff --git a/samples/snippets/src/test/java/com/example/bigquery/AuthorizeDatasetIT.java b/samples/snippets/src/test/java/com/example/bigquery/AuthorizeDatasetIT.java index b13c6bb73a..c4facd5ef4 100644 --- a/samples/snippets/src/test/java/com/example/bigquery/AuthorizeDatasetIT.java +++ b/samples/snippets/src/test/java/com/example/bigquery/AuthorizeDatasetIT.java @@ -80,6 +80,6 @@ public void tearDown() { @Test public void testCreateDataset() { AuthorizeDataset.authorizeDataset(sourceDatasetId, userDatasetId); - assertThat(bout.toString()).contains(userDatasetId + " updated with the added authorization"); + assertThat(bout.toString()).contains(sourceDatasetId + " updated with the added authorization"); } } From 1477562c48e9f993d506d13b140d8d2d9c96cacb Mon Sep 17 00:00:00 2001 From: "release-please[bot]" <55107282+release-please[bot]@users.noreply.github.com> Date: Wed, 24 Aug 2022 08:21:36 +0530 Subject: [PATCH 11/11] chore(main): release 2.14.7 (#2245) Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> --- CHANGELOG.md | 7 +++++++ benchmark/pom.xml | 2 +- google-cloud-bigquery/pom.xml | 4 ++-- pom.xml | 4 ++-- samples/snapshot/pom.xml | 2 +- versions.txt | 2 +- 6 files changed, 14 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 68cff6cc02..7867073585 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,12 @@ # Changelog +## [2.14.7](https://github.com/googleapis/java-bigquery/compare/v2.14.6...v2.14.7) (2022-08-23) + + +### Bug Fixes + +* table-not-found issue with executeSelect while running long queries ([#2222](https://github.com/googleapis/java-bigquery/issues/2222)) ([4876569](https://github.com/googleapis/java-bigquery/commit/487656973fe3e06d838c1b495ac024ab2c6810f6)) + ## [2.14.6](https://github.com/googleapis/java-bigquery/compare/v2.14.5...v2.14.6) (2022-08-12) diff --git a/benchmark/pom.xml b/benchmark/pom.xml index c67b15083c..3c299ff36e 100644 --- a/benchmark/pom.xml +++ b/benchmark/pom.xml @@ -6,7 +6,7 @@ google-cloud-bigquery-parent com.google.cloud - 2.14.7-SNAPSHOT + 2.14.7 diff --git a/google-cloud-bigquery/pom.xml b/google-cloud-bigquery/pom.xml index d69dd2bfc2..0f87a2b7b3 100644 --- a/google-cloud-bigquery/pom.xml +++ b/google-cloud-bigquery/pom.xml @@ -3,7 +3,7 @@ 4.0.0 com.google.cloud google-cloud-bigquery - 2.14.7-SNAPSHOT + 2.14.7 jar BigQuery https://github.com/googleapis/java-bigquery @@ -11,7 +11,7 @@ com.google.cloud google-cloud-bigquery-parent - 2.14.7-SNAPSHOT + 2.14.7 google-cloud-bigquery diff --git a/pom.xml b/pom.xml index a7177c701a..442a2dc4f0 100644 --- a/pom.xml +++ b/pom.xml @@ -4,7 +4,7 @@ com.google.cloud google-cloud-bigquery-parent pom - 2.14.7-SNAPSHOT + 2.14.7 BigQuery Parent https://github.com/googleapis/java-bigquery @@ -118,7 +118,7 @@ com.google.cloud google-cloud-bigquery - 2.14.7-SNAPSHOT + 2.14.7 diff --git a/samples/snapshot/pom.xml b/samples/snapshot/pom.xml index c1696f1654..4e9ddfd3ea 100644 --- a/samples/snapshot/pom.xml +++ b/samples/snapshot/pom.xml @@ -44,7 +44,7 @@ com.google.cloud google-cloud-bigquery - 2.14.7-SNAPSHOT + 2.14.7 diff --git a/versions.txt b/versions.txt index dcd10147bb..ddf3790c1f 100644 --- a/versions.txt +++ b/versions.txt @@ -1,4 +1,4 @@ # Format: # module:released-version:current-version -google-cloud-bigquery:2.14.6:2.14.7-SNAPSHOT \ No newline at end of file +google-cloud-bigquery:2.14.7:2.14.7 \ No newline at end of file