Java idiomatic client for BigQuery JDBC.

• Product Documentation
• Client Library Documentation
• Driver User Guide
• Storage APIs Deep-Dive Guide

If you are using Maven, add this to your pom.xml file:

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-bigquery-jdbc</artifactId>
  <version>LATEST_VERSION</version>
</dependency>

If you are using Gradle without BOM, add this to your dependencies:

implementation 'com.google.cloud:google-cloud-bigquery-jdbc:LATEST_VERSION'

If you are using SBT, add this to your dependencies:

libraryDependencies += "com.google.cloud" % "google-cloud-bigquery-jdbc" % "LATEST_VERSION"

See the Authentication section in the base directory's README.

The client application making API calls must be granted authorization scopes required for the desired BigQuery JDBC APIs, and the authenticated principal must have the IAM role(s) required to access GCP resources using the BigQuery JDBC API calls.

The JDBC driver for BigQuery includes client-side telemetry that automatically collects anonymous usage and diagnostic metrics to help Google improve driver stability and features. Client-side usage telemetry collection is designed to run in the background without affecting driver performance or JDBC functionality. Any telemetry transmission failures fail silently and don't surface errors to the application or incur any cost.

Client-side usage telemetry never collects SQL query text, parameter values, database schema information, or data contents. Telemetry collection doesn't work in environments using custom endpoint URLs or restricted network access that blocks outbound connections to Google's public logging services.

The driver collects the following information:

• Environment information: The Driver version, JVM major version, operating system platform and version, and an anonymous device correlation ID stored locally in the ~/.bigquery-jdbc/telemetry-tag folder.
• Connection metrics: Connection outcome status (success, failure, or cancellation) and authentication method used (such as service account, ADC, or token).
• Execution metrics: High-level query operation types (such as SELECT, INSERT, and CREATE TABLE), execution engine substrate used (standard REST API or high-throughput API), and duration latency distributions.
• Error and feature metrics: Internal exception codes, origin methods, and feature usage counters (such as prepared statements, batching, and connection pooling).

Client-side usage telemetry behaves as a global background service for the application's JVM. Settings can be configured using JDBC connection properties, environment variables, or Java system properties. The driver resolves settings in the following priority: system property > environment variable > JDBC connection property.

Due to its global nature, client-side usage telemetry connection properties exhibit the following lifecycle behaviors:

• Opt-Out Control: If client-side usage telemetry is disabled at any point using a connection property (EnableDiagnosticTelemetry=0), environment variable, or system property, the service immediately shuts down and remains permanently off for the remainder of the JVM lifecycle. It will only restart when the JVM restarts.
• Global Worker Settings: Because the telemetry batcher is a shared global resource, properties that configure its behavior (TelemetryUploadInterval and TelemetryBatchSize) are exclusively evaluated upon establishing the first connection. Subsequent overrides for these specific properties are ignored.

To configure these properties, you can use the following keys:

• Opt-Out Control:
  • Connection Property: EnableDiagnosticTelemetry=0;
  • Environment Variable: GOOGLE_BIGQUERY_JDBC_TELEMETRY_ENABLED=false
  • System Property: -Dgoogle.bigquery.jdbc.telemetry.enabled=false
• Upload Interval:
  • Connection Property: TelemetryUploadInterval=300000;
  • Environment Variable: GOOGLE_BIGQUERY_JDBC_TELEMETRY_INTERVAL_MS=300000
  • System Property: -Dgoogle.bigquery.jdbc.telemetry.interval_ms=300000
• Batch Size:
  • Connection Property: TelemetryBatchSize=5000;
  • Environment Variable: GOOGLE_BIGQUERY_JDBC_TELEMETRY_BATCH_SIZE=5000
  • System Property: -Dgoogle.bigquery.jdbc.telemetry.batch_size=5000

You need to have either Java with Maven installed or Docker. You might want to install Make to simplify running commands, otherwise please look into Makefile to check for specific configurations.

make install primarily relies on mvn install command. All following commands are primarily applicable for the google-cloud-bigquery-jdbc project. You can also use make clean to clean the project and make lint to format the code.

Run all unittests

make unittest

Run specific unittests

make unittest test=<filter>

Please reference Maven documentation for details about <filter>

Example: make unittest test=BigQueryArrowStructTest

IMPORTANT: Running integration tests will skip unit tests by default. To include unit tests, run make integration-test skipSurefire=false. Primary focus of this command is to run specific set of tests without a lot of overhead.

Set the following environment variables to run the integration tests:

# Default gcloud auth setup
export GOOGLE_APPLICATION_CREDENTIALS=<path-to-service-account-json-config-file>

# Test specfic envs
export SA_EMAIL=email@email.com
export SA_SECRET=<path-to-service-account-json-config-file>
# Alternatively it can be json content:
export SA_SECRET=`cat <path-to-service-account-json-config-file>`

Run all integration tests (currently takes 15+ minutes, so this is discouraged).

make integration-test

Run specific integration test

make integration-test test=<filter>

Please reference Maven documentation for details about <filter>

Example: make integration-test test=ITBigQueryJDBCTest#testValidServiceAccountAuthenticationOAuthPvtKey

If you don't have Java or Maven, or if you want to test changes with a different Java version, you can leverage dockerized environment.

One-time run commands are similar to local development make commands:

make docker-build
make docker-unittest
make docker-integration-test

Please note that running unit or integration tests within docker doesn't leverage maven cache because it is not persisted. If you want to run multiple commands, you can start a session and treat this shell session as your local environment.

make docker-session

All Docker commands rely only on GOOGLE_APPLICATION_CREDENTIALS env being present, it will create rest of env vars as needed.

There are a few ways to package the Google JDBC Driver. The output of the packaging commands can be found in the target directory.

make package or make docker-package will create both a thin jar and a zip file with all dependencies. make package-all-dependencies or make docker-package-all-dependencies will create a single jar with all dependencies included. make package-all-dependencies-shaded or make docker-package-all-dependencies-shaded will create a single shaded jar with all dependencies included.

The thin jar is created with make package. The thin jar is packaged as a zip file with its dependencies listed in a dependencies.txt file, compatible with pom.xml.

The jar with all dependencies is also created. This jar includes all dependencies and can be used as a standalone jar with tools like R-Studio. You can run make package-all-dependencies or make docker-package-all-dependencies to build only the jar with all dependencies.

You can also build a shaded jar with all dependencies. This can be done by running make package-all-dependencies-shaded or make docker-package-all-dependencies-shaded.

The nightly build runs the full set of integration tests, including the ITBigQueryJDBCTest and ITNightlyBigQueryTest test suites. It also includes some long-running tests (takes 20+ minutes to complete).

Note: These builds are intended for testing and development purposes only and are not recommended for production use.

Nightly Integration Tests include a step to build a full and thin jars and upload them to Google Storage.

They can be retrieved via following commands:

gsutil cp gs://bq_devtools_release_private/drivers/jdbc/google-cloud-bigquery-jdbc-latest-all.jar .
gsutil cp gs://bq_devtools_release_private/drivers/jdbc/google-cloud-bigquery-jdbc-latest.zip .

Cloud Build Pipeline is uploading latest full jar to the internal location for perf tests once a week.

We're using JaCoCo to track Code Coverage. Makefile has 2 separate set of commands for unittests and integration tests reports.

You can run make unit-test-coverage to generate a coverage report for unit tests. The output will be in jacoco-unittests.zip. You can run make full-coverage to generate a coverage report for both unit and integration tests. The output will be in jacoco-full.zip.

You can also run make docker-coverage which will produce both results. You can find jacoco-unittests.zip and jacoco-full.zip files in the root with results.

You will need a Google Cloud Platform Console project with the BigQuery JDBC [API enabled][enable-api].

Follow these instructions to get your project set up. You will also need to set up the local development environment by installing the Google Cloud Command Line Interface and running the following commands in command line: gcloud auth login and gcloud config set project [YOUR PROJECT ID].

You'll need to obtain the google-cloud-bigquery-jdbc library. See the Quickstart section to add google-cloud-bigquery-jdbc as a dependency in your code.

BigQuery JDBC

See the BigQuery JDBC client library docs to learn how to use this BigQuery JDBC Client Library.

To get help, follow the instructions in the shared Troubleshooting document.

Java 8 or above is required for using this client.

Google's Java client libraries, Google Cloud Client Libraries and Google Cloud API Libraries, follow the Oracle Java SE support roadmap (see the Oracle Java SE Product Releases section).

In general, new feature development occurs with support for the lowest Java LTS version covered by Oracle's Premier Support (which typically lasts 5 years from initial General Availability). If the minimum required JVM for a given library is changed, it is accompanied by a semver major release.

Java 11 and (in September 2021) Java 17 are the best choices for new development.

Google tests its client libraries with all current LTS versions covered by Oracle's Extended Support (which typically lasts 8 years from initial General Availability).

Google's client libraries support legacy versions of Java runtimes with long term stable libraries that don't receive feature updates on a best efforts basis as it may not be possible to backport all patches.

Google provides updates on a best efforts basis to apps that continue to use Java 7, though apps might need to upgrade to current versions of the library that supports their JVM.

The latest versions and the supported Java versions are identified on the individual GitHub repository github.com/GoogleAPIs/java-SERVICENAME and on google-cloud-java.

This library follows Semantic Versioning.

Contributions to this library are always welcome and highly encouraged.

See CONTRIBUTING for more information how to get started.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See Code of Conduct for more information.

Apache 2.0 - See LICENSE for more information.

Java is a registered trademark of Oracle and/or its affiliates.