Skip to content

Commit 3f2799d

Browse files
ajkannanajkannan
committed
Merge pull request #503 from GoogleCloudPlatform/bigquery
Merge gcloud-java-bigquery into master
2 parents 9bcebd3 + a1299c7 commit 3f2799d

File tree

76 files changed

+15603
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

76 files changed

+15603
-0
lines changed

‎README.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ This client supports the following Google Cloud Platform services:
1515
- [Google Cloud Datastore] (#google-cloud-datastore)
1616
- [Google Cloud Storage] (#google-cloud-storage)
1717
- [Google Cloud Resource Manager] (#google-cloud-resource-manager)
18+
- [Google Cloud BigQuery] (#google-cloud-bigquery)
1819

1920
> Note: This client is a work-in-progress, and may occasionally
2021
> make backwards-incompatible changes.
@@ -214,6 +215,51 @@ while (projectIterator.hasNext()) {
214215
}
215216
```
216217
218+
Google Cloud BigQuery
219+
----------------------
220+
221+
- [API Documentation][bigquery-api]
222+
- [Official Documentation][cloud-bigquery-docs]
223+
224+
#### Preview
225+
226+
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you
227+
must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.
228+
229+
```java
230+
import com.google.gcloud.bigquery.BaseTableInfo;
231+
import com.google.gcloud.bigquery.BigQuery;
232+
import com.google.gcloud.bigquery.BigQueryOptions;
233+
import com.google.gcloud.bigquery.Field;
234+
import com.google.gcloud.bigquery.JobStatus;
235+
import com.google.gcloud.bigquery.LoadJobInfo;
236+
import com.google.gcloud.bigquery.Schema;
237+
import com.google.gcloud.bigquery.TableId;
238+
import com.google.gcloud.bigquery.TableInfo;
239+
240+
BigQuery bigquery = BigQueryOptions.defaultInstance().service();
241+
TableId tableId = TableId.of("dataset", "table");
242+
BaseTableInfo info = bigquery.getTable(tableId);
243+
if (info == null) {
244+
System.out.println("Creating table " + tableId);
245+
Field integerField = Field.of("fieldName", Field.Type.integer());
246+
bigquery.create(TableInfo.of(tableId, Schema.of(integerField)));
247+
} else {
248+
System.out.println("Loading data into table " + tableId);
249+
LoadJobInfo loadJob = LoadJobInfo.of(tableId, "gs://bucket/path");
250+
loadJob = bigquery.create(loadJob);
251+
while (loadJob.status().state() != JobStatus.State.DONE) {
252+
Thread.sleep(1000L);
253+
loadJob = bigquery.getJob(loadJob.jobId());
254+
}
255+
if (loadJob.status().error() != null) {
256+
System.out.println("Job completed with errors");
257+
} else {
258+
System.out.println("Job succeeded");
259+
}
260+
}
261+
```
262+
217263
Troubleshooting
218264
---------------
219265
@@ -276,3 +322,7 @@ Apache 2.0 - See [LICENSE] for more information.
276322
277323
[resourcemanager-api]:http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/resourcemanager/package-summary.html
278324
[cloud-resourcemanager-docs]:https://cloud.google.com/resource-manager/
325+
326+
[cloud-bigquery]: https://cloud.google.com/bigquery/
327+
[cloud-bigquery-docs]: https://cloud.google.com/bigquery/docs/overview
328+
[bigquery-api]: http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/bigquery/package-summary.html

‎TESTING.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ This library provides tools to help write tests for code that uses the following
55
- [Datastore] (#testing-code-that-uses-datastore)
66
- [Storage] (#testing-code-that-uses-storage)
77
- [Resource Manager] (#testing-code-that-uses-resource-manager)
8+
- [BigQuery] (#testing-code-that-uses-bigquery)
89

910
### Testing code that uses Datastore
1011

@@ -103,5 +104,34 @@ You can test against a temporary local Resource Manager by following these steps
103104

104105
This method will block until the server thread has been terminated.
105106

107+
### Testing code that uses BigQuery
108+
109+
Currently, there isn't an emulator for Google BigQuery, so an alternative is to create a test
110+
project. `RemoteBigQueryHelper` contains convenience methods to make setting up and cleaning up the
111+
test project easier. To use this class, follow the steps below:
112+
113+
1. Create a test Google Cloud project.
114+
115+
2. Download a [JSON service account credentials file][create-service-account] from the Google
116+
Developer's Console.
117+
118+
3. Create a `RemoteBigQueryHelper` object using your project ID and JSON key.
119+
Here is an example that uses the `RemoteBigQueryHelper` to create a dataset.
120+
```java
121+
RemoteBigQueryHelper bigqueryHelper =
122+
RemoteBigQueryHelper.create(PROJECT_ID, new FileInputStream("/path/to/my/JSON/key.json"));
123+
BigQuery bigquery = bigqueryHelper.options().service();
124+
String dataset = RemoteBigQueryHelper.generateDatasetName();
125+
bigquery.create(DatasetInfo.builder(dataset).build());
126+
```
127+
128+
4. Run your tests.
129+
130+
5. Clean up the test project by using `forceDelete` to clear any datasets used.
131+
Here is an example that clears the dataset created in Step 3.
132+
```java
133+
RemoteBigQueryHelper.forceDelete(bigquery, dataset);
134+
```
106135

107136
[cloud-platform-storage-authentication]:https://cloud.google.com/storage/docs/authentication?hl=en#service_accounts
137+
[create-service-account]:https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount

0 commit comments

Comments
 (0)