Skip to content

Commit a900b97

Browse files
mziccardmziccard
committed
Add bigquery module README
1 parent 7eee528 commit a900b97

File tree

1 file changed

+240
-7
lines changed

1 file changed

+240
-7
lines changed

‎gcloud-java-bigquery/README.md‎

Lines changed: 240 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -8,16 +8,22 @@ Java idiomatic client for [Google Cloud BigQuery] (https://cloud.google.com/bigq
88
<!-- TODO(mziccard): add in the maven shield once the artifact is pushed to maven -->
99

1010
- [Homepage] (https://googlecloudplatform.github.io/gcloud-java/)
11-
+<!-- TODO(mziccard): add link to API documentatin -->
11+
12+
<!-- TODO(mziccard): add link to API documentation -->
1213

1314
> Note: This client is a work-in-progress, and may occasionally
1415
> make backwards-incompatible changes.
1516
1617
Quickstart
1718
----------
18-
Add this to your pom.xml file
19-
<!-- TODO(mziccard): add dependency code -->
19+
If you are using Maven, add this to your pom.xml file
20+
<!-- TODO(mziccard): add mvn dependency code -->
21+
22+
If you are using Gradle, add this to your dependencies
23+
<!-- TODO(mziccard): add gradle dependency code -->
2024

25+
If you are using SBT, add this to your dependencies
26+
<!-- TODO(mziccard): add sbt dependency code -->
2127

2228
Example Application
2329
-------------------
@@ -36,14 +42,239 @@ About Google Cloud BigQuery
3642
Data can be streamed into BigQuery at millions of rows per second to enable real-time analysis.
3743
With BigQuery you can easily deploy Petabyte-scale Databases.
3844

39-
Be sure to activate the Google Cloud BigQuery API on the Developer's Console to use BigQuery from your project.
45+
Be sure to activate the Google Cloud BigQuery API on the Developer's Console to use BigQuery from
46+
your project.
4047

4148
See the ``gcloud-java`` API [bigquery documentation][bigquery-api] to learn how to interact
4249
with Google Cloud BigQuery using this Client Library.
4350

44-
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](https://github.com/GoogleCloudPlatform/gcloud-java#authentication) and a project ID if running this snippet elsewhere.
51+
Getting Started
52+
---------------
53+
#### Prerequisites
54+
For this tutorial, you will need a
55+
[Google Developers Console](https://console.developers.google.com/) project with the BigQuery API
56+
enabled. You will need to [enable billing](https://support.google.com/cloud/answer/6158867?hl=en) to
57+
use Google Cloud BigQuery.
58+
[Follow these instructions](https://cloud.google.com/docs/authentication#preparation) to get your
59+
project set up. You will also need to set up the local development environment by [installing the
60+
Google Cloud SDK](https://cloud.google.com/sdk/) and running the following commands in command line:
61+
`gcloud auth login` and `gcloud config set project [YOUR PROJECT ID]`.
62+
63+
#### Installation and setup
64+
You'll need to obtain the `gcloud-java-bigquery` library. See the [Quickstart](#quickstart) section
65+
to add `gcloud-java-bigquery` as a dependency in your code.
66+
67+
#### Creating an authorized service object
68+
To make authenticated requests to Google Cloud BigQuery, you must create a service object with
69+
credentials. You can then make API calls by calling methods on the BigQuery service object. The
70+
simplest way to authenticate is to use
71+
[Application Default Credentials](https://developers.google.com/identity/protocols/application-default-credentials).
72+
These credentials are automatically inferred from your environment, so you only need the following
73+
code to create your service object:
74+
75+
```java
76+
import com.google.gcloud.bigquery.BigQuery;
77+
import com.google.gcloud.bigquery.BigQueryOptions;
78+
79+
BigQuery bigquery = BigQueryOptions.defaultInstance().service();
80+
```
81+
82+
For other authentication options, see the
83+
[Authentication](https://github.com/GoogleCloudPlatform/gcloud-java#authentication) page.
84+
85+
#### Creating a dataset
86+
With BigQuery you can create datasets. A dataset is a grouping mechanism that holds zero or more
87+
tables. Add the following import at the top of your file:
88+
89+
```java
90+
import com.google.gcloud.bigquery.DatasetInfo;
91+
```
92+
Then, to create the dataset, use the following code:
93+
94+
```java
95+
// Create a dataset
96+
String datasetId = "my_dataset_id";
97+
bigquery.create(DatasetInfo.builder(datasetId).build());
98+
```
99+
100+
#### Creating a table
101+
With BigQuery you can create different types of tables: normal tables with an associated schema,
102+
external tables backed by data stored on Google Cloud Storage and view tables that are created from
103+
a BigQuery SQL query. In this code snippet we show how to create a normal table with only one string
104+
field. Add the following imports at the top of your file:
105+
106+
```java
107+
import com.google.gcloud.bigquery.Field;
108+
import com.google.gcloud.bigquery.Schema;
109+
import com.google.gcloud.bigquery.TableId;
110+
import com.google.gcloud.bigquery.BaseTableInfo;
111+
import com.google.gcloud.bigquery.TableInfo;
112+
```
113+
Then add the following code to create the table:
114+
115+
```java
116+
TableId tableId = TableId.of(datasetId, "my_table_id");
117+
// Table field definition
118+
Field stringField = Field.builder("StringField", Field.Type.string())
119+
.mode(Field.Mode.NULLABLE)
120+
.build();
121+
// Table schema definition
122+
Schema schema = Schema.of(stringField);
123+
// Create a table
124+
BaseTableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, schema));
125+
```
126+
127+
#### Loading data into a table
128+
BigQuery provides several ways to load data into a table: streaming rows or loading data from a
129+
Google Cloud Storage file. In this code snippet we show how to stream rows into a table.
130+
Add the following imports at the top of your file:
131+
132+
```java
133+
import com.google.gcloud.bigquery.InsertAllRequest;
134+
import com.google.gcloud.bigquery.InsertAllResponse;
135+
136+
import java.util.HashMap;
137+
import java.util.Map;
138+
```
139+
Then add the following code to insert data:
45140

46-
<!-- TODO(mziccard): add code snippet -->
141+
```java
142+
Map<String, Object> firstRow = new HashMap<>();
143+
Map<String, Object> secondRow = new HashMap<>();
144+
firstRow.put("StringField", "value1");
145+
secondRow.put("StringField", "value2");
146+
// Create an insert request
147+
InsertAllRequest insertRequest = InsertAllRequest.builder(tableId)
148+
.addRow(firstRow)
149+
.addRow(secondRow)
150+
.build();
151+
// Insert rows
152+
InsertAllResponse insertResponse = bigquery.insertAll(insertRequest);
153+
// Check if errors occurred
154+
if (insertResponse.hasErrors()) {
155+
System.out.println("Errors occurred while inserting rows");
156+
}
157+
```
158+
159+
#### Querying data
160+
BigQuery enables querying data by running queries and waiting for the result. Queries can be run
161+
directly or through a Query Job. In this code snippet we show how to run a query directly and wait
162+
for the result. Add the following imports at the top of your file:
163+
164+
```java
165+
import com.google.gcloud.bigquery.FieldValue;
166+
import com.google.gcloud.bigquery.QueryRequest;
167+
import com.google.gcloud.bigquery.QueryResponse;
168+
169+
import java.util.Iterator;
170+
import java.util.List;
171+
```
172+
Then add the following code to run the query and wait for the result:
173+
174+
```java
175+
// Create a query request
176+
QueryRequest queryRequest = QueryRequest.builder("SELECT * FROM my_dataset_id.my_table_id")
177+
.maxWaitTime(60000L)
178+
.maxResults(1000L)
179+
.build();
180+
// Request query to be executed and wait for results
181+
QueryResponse queryResponse = bigquery.query(queryRequest);
182+
while (!queryResponse.jobComplete()) {
183+
Thread.sleep(1000);
184+
queryResponse = bigquery.getQueryResults(queryResponse.jobId());
185+
}
186+
// Read rows
187+
Iterator<List<FieldValue>> rowIterator = queryResponse.result().iterateAll();
188+
System.out.println("Table rows:");
189+
while (rowIterator.hasNext()) {
190+
System.out.println(rowIterator.next());
191+
}
192+
```
193+
#### Complete source code
194+
195+
Here we put together all the code shown above into one program. This program assumes that you are
196+
running on Compute Engine or from your own desktop. To run this example on App Engine, simply move
197+
the code from the main method to your application's servlet class and change the print statements to
198+
display on your webpage.
199+
200+
```java
201+
import com.google.gcloud.bigquery.BaseTableInfo;
202+
import com.google.gcloud.bigquery.BigQuery;
203+
import com.google.gcloud.bigquery.BigQueryOptions;
204+
import com.google.gcloud.bigquery.DatasetInfo;
205+
import com.google.gcloud.bigquery.Field;
206+
import com.google.gcloud.bigquery.FieldValue;
207+
import com.google.gcloud.bigquery.InsertAllRequest;
208+
import com.google.gcloud.bigquery.InsertAllResponse;
209+
import com.google.gcloud.bigquery.QueryRequest;
210+
import com.google.gcloud.bigquery.QueryResponse;
211+
import com.google.gcloud.bigquery.Schema;
212+
import com.google.gcloud.bigquery.TableId;
213+
import com.google.gcloud.bigquery.TableInfo;
214+
215+
import java.util.HashMap;
216+
import java.util.Iterator;
217+
import java.util.List;
218+
import java.util.Map;
219+
220+
public class GcloudBigQueryExample {
221+
222+
public static void main(String[] args) throws InterruptedException {
223+
224+
// Create a service instance
225+
BigQuery bigquery = BigQueryOptions.defaultInstance().service();
226+
227+
// Create a dataset
228+
String datasetId = "my_dataset_id";
229+
bigquery.create(DatasetInfo.builder(datasetId).build());
230+
231+
TableId tableId = TableId.of(datasetId, "my_table_id");
232+
// Table field definition
233+
Field stringField = Field.builder("StringField", Field.Type.string())
234+
.mode(Field.Mode.NULLABLE)
235+
.build();
236+
// Table schema definition
237+
Schema schema = Schema.of(stringField);
238+
// Create a table
239+
BaseTableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, schema));
240+
241+
// Define rows to insert
242+
Map<String, Object> firstRow = new HashMap<>();
243+
Map<String, Object> secondRow = new HashMap<>();
244+
firstRow.put("StringField", "value1");
245+
secondRow.put("StringField", "value2");
246+
// Create an insert request
247+
InsertAllRequest insertRequest = InsertAllRequest.builder(tableId)
248+
.addRow(firstRow)
249+
.addRow(secondRow)
250+
.build();
251+
// Insert rows
252+
InsertAllResponse insertResponse = bigquery.insertAll(insertRequest);
253+
// Check if errors occurred
254+
if (insertResponse.hasErrors()) {
255+
System.out.println("Errors occurred while inserting rows");
256+
}
257+
258+
// Create a query request
259+
QueryRequest queryRequest = QueryRequest.builder("SELECT * FROM my_dataset_id.my_table_id")
260+
.maxWaitTime(60000L)
261+
.maxResults(1000L)
262+
.build();
263+
// Request query to be executed and wait for results
264+
QueryResponse queryResponse = bigquery.query(queryRequest);
265+
while (!queryResponse.jobComplete()) {
266+
Thread.sleep(1000);
267+
queryResponse = bigquery.getQueryResults(queryResponse.jobId());
268+
}
269+
// Read rows
270+
Iterator<List<FieldValue>> rowIterator = queryResponse.result().iterateAll();
271+
System.out.println("Table rows:");
272+
while (rowIterator.hasNext()) {
273+
System.out.println(rowIterator.next());
274+
}
275+
}
276+
}
277+
```
47278

48279
Java Versions
49280
-------------
@@ -53,7 +284,9 @@ Java 7 or above is required for using this client.
53284
Testing
54285
-------
55286

56-
<!-- TODO(mziccard): add this in once the RemoteGCBQMHelper class is functional -->
287+
This library has tools to help make tests for code using Cloud BigQuery.
288+
289+
See [TESTING] to read more about testing.
57290

58291
Versioning
59292
----------

0 commit comments

Comments
 (0)