Skip to content
This repository was archived by the owner on Mar 6, 2026. It is now read-only.

Commit 66014c3

Browse files
tswasttswast
authored
chore: merge changes from master (#872)
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly: - [ ] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/python-bigquery/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea - [ ] Ensure the tests and linter pass - [ ] Code coverage does not decrease (if any source code was changed) - [ ] Appropriate docs were updated (if necessary) Fixes #<issue_number_goes_here> 🦕
1 parent e26d879 commit 66014c3

28 files changed

Lines changed: 1005 additions & 97 deletions

‎.github/.OwlBot.lock.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
docker:
22
image: gcr.io/repo-automation-bots/owlbot-python:latest
3-
digest: sha256:aea14a583128771ae8aefa364e1652f3c56070168ef31beb203534222d842b8b
3+
digest: sha256:50e35228649c47b6ca82aa0be3ff9eb2afce51c82b66c4a03fe4afeb5ff6c0fc

‎.github/CODEOWNERS‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,4 +8,4 @@
88
* @googleapis/api-bigquery @googleapis/yoshi-python
99

1010
# The python-samples-reviewers team is the default owner for samples changes
11-
/samples/ @googleapis/api-bigquery @googleapis/python-samples-owners
11+
/samples/ @googleapis/api-bigquery @googleapis/python-samples-owners @googleapis/yoshi-python

‎CHANGELOG.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,28 @@
55
[1]: https://pypi.org/project/google-cloud-bigquery/#history
66

77

8+
## [2.24.0](https://www.github.com/googleapis/python-bigquery/compare/v2.23.3...v2.24.0) (2021-08-11)
9+
10+
11+
### Features
12+
13+
* add support for transaction statistics ([#849](https://www.github.com/googleapis/python-bigquery/issues/849)) ([7f7b1a8](https://www.github.com/googleapis/python-bigquery/commit/7f7b1a808d50558772a0deb534ca654da65d629e))
14+
* make the same `Table*` instances equal to each other ([#867](https://www.github.com/googleapis/python-bigquery/issues/867)) ([c1a3d44](https://www.github.com/googleapis/python-bigquery/commit/c1a3d4435739a21d25aa154145e36d3a7c42eeb6))
15+
* retry failed query jobs in `result()` ([#837](https://www.github.com/googleapis/python-bigquery/issues/837)) ([519d99c](https://www.github.com/googleapis/python-bigquery/commit/519d99c20e7d1101f76981f3de036fdf3c7a4ecc))
16+
* support `ScalarQueryParameterType` for `type_` argument in `ScalarQueryParameter` constructor ([#850](https://www.github.com/googleapis/python-bigquery/issues/850)) ([93d15e2](https://www.github.com/googleapis/python-bigquery/commit/93d15e2e5405c2cc6d158c4e5737361344193dbc))
17+
18+
19+
### Bug Fixes
20+
21+
* make unicode characters working well in load_table_from_json ([#865](https://www.github.com/googleapis/python-bigquery/issues/865)) ([ad9c802](https://www.github.com/googleapis/python-bigquery/commit/ad9c8026f0e667f13dd754279f9dc40d06f4fa78))
22+
23+
### [2.23.3](https://www.github.com/googleapis/python-bigquery/compare/v2.23.2...v2.23.3) (2021-08-06)
24+
25+
26+
### Bug Fixes
27+
28+
* increase default retry deadline to 10 minutes ([#859](https://www.github.com/googleapis/python-bigquery/issues/859)) ([30770fd](https://www.github.com/googleapis/python-bigquery/commit/30770fd0575fbd5aaa70c14196a4cc54627aecd2))
29+
830
### [2.23.2](https://www.github.com/googleapis/python-bigquery/compare/v2.23.1...v2.23.2) (2021-07-29)
931

1032

‎docs/reference.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ Job-Related Types
6868
job.SourceFormat
6969
job.WriteDisposition
7070
job.SchemaUpdateOption
71+
job.TransactionInfo
7172

7273

7374
Dataset
@@ -137,6 +138,7 @@ Query
137138

138139
query.ArrayQueryParameter
139140
query.ScalarQueryParameter
141+
query.ScalarQueryParameterType
140142
query.StructQueryParameter
141143
query.UDFResource
142144

‎google/cloud/bigquery/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@
6969
from google.cloud.bigquery.job import ScriptOptions
7070
from google.cloud.bigquery.job import SourceFormat
7171
from google.cloud.bigquery.job import UnknownJob
72+
from google.cloud.bigquery.job import TransactionInfo
7273
from google.cloud.bigquery.job import WriteDisposition
7374
from google.cloud.bigquery.model import Model
7475
from google.cloud.bigquery.model import ModelReference
@@ -148,6 +149,7 @@
148149
"GoogleSheetsOptions",
149150
"ParquetOptions",
150151
"ScriptOptions",
152+
"TransactionInfo",
151153
"DEFAULT_RETRY",
152154
# Enum Constants
153155
"enums",

‎google/cloud/bigquery/client.py‎

Lines changed: 82 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@
7373
from google.cloud.bigquery.model import ModelReference
7474
from google.cloud.bigquery.model import _model_arg_to_model_ref
7575
from google.cloud.bigquery.query import _QueryResults
76-
from google.cloud.bigquery.retry import DEFAULT_RETRY
76+
from google.cloud.bigquery.retry import DEFAULT_RETRY, DEFAULT_JOB_RETRY
7777
from google.cloud.bigquery.routine import Routine
7878
from google.cloud.bigquery.routine import RoutineReference
7979
from google.cloud.bigquery.schema import SchemaField
@@ -2709,7 +2709,7 @@ def load_table_from_json(
27092709

27102710
destination = _table_arg_to_table_ref(destination, default_project=self.project)
27112711

2712-
data_str = "\n".join(json.dumps(item) for item in json_rows)
2712+
data_str = "\n".join(json.dumps(item, ensure_ascii=False) for item in json_rows)
27132713
encoded_str = data_str.encode()
27142714
data_file = io.BytesIO(encoded_str)
27152715
return self.load_table_from_file(
@@ -3110,6 +3110,7 @@ def query(
31103110
project: str = None,
31113111
retry: retries.Retry = DEFAULT_RETRY,
31123112
timeout: float = None,
3113+
job_retry: retries.Retry = DEFAULT_JOB_RETRY,
31133114
) -> job.QueryJob:
31143115
"""Run a SQL query.
31153116
@@ -3139,30 +3140,59 @@ def query(
31393140
Project ID of the project of where to run the job. Defaults
31403141
to the client's project.
31413142
retry (Optional[google.api_core.retry.Retry]):
3142-
How to retry the RPC.
3143+
How to retry the RPC. This only applies to making RPC
3144+
calls. It isn't used to retry failed jobs. This has
3145+
a reasonable default that should only be overridden
3146+
with care.
31433147
timeout (Optional[float]):
31443148
The number of seconds to wait for the underlying HTTP transport
31453149
before using ``retry``.
3150+
job_retry (Optional[google.api_core.retry.Retry]):
3151+
How to retry failed jobs. The default retries
3152+
rate-limit-exceeded errors. Passing ``None`` disables
3153+
job retry.
3154+
3155+
Not all jobs can be retried. If ``job_id`` is
3156+
provided, then the job returned by the query will not
3157+
be retryable, and an exception will be raised if a
3158+
non-``None`` (and non-default) value for ``job_retry``
3159+
is also provided.
3160+
3161+
Note that errors aren't detected until ``result()`` is
3162+
called on the job returned. The ``job_retry``
3163+
specified here becomes the default ``job_retry`` for
3164+
``result()``, where it can also be specified.
31463165
31473166
Returns:
31483167
google.cloud.bigquery.job.QueryJob: A new query job instance.
31493168
31503169
Raises:
31513170
TypeError:
3152-
If ``job_config`` is not an instance of :class:`~google.cloud.bigquery.job.QueryJobConfig`
3153-
class.
3171+
If ``job_config`` is not an instance of
3172+
:class:`~google.cloud.bigquery.job.QueryJobConfig`
3173+
class, or if both ``job_id`` and non-``None`` non-default
3174+
``job_retry`` are provided.
31543175
"""
31553176
job_id_given = job_id is not None
3156-
job_id = _make_job_id(job_id, job_id_prefix)
3177+
if (
3178+
job_id_given
3179+
and job_retry is not None
3180+
and job_retry is not DEFAULT_JOB_RETRY
3181+
):
3182+
raise TypeError(
3183+
"`job_retry` was provided, but the returned job is"
3184+
" not retryable, because a custom `job_id` was"
3185+
" provided."
3186+
)
3187+
3188+
job_id_save = job_id
31573189

31583190
if project is None:
31593191
project = self.project
31603192

31613193
if location is None:
31623194
location = self.location
31633195

3164-
job_config = copy.deepcopy(job_config)
3165-
31663196
if self._default_query_job_config:
31673197
if job_config:
31683198
_verify_job_config_type(
@@ -3172,6 +3202,8 @@ def query(
31723202
# that is in the default,
31733203
# should be filled in with the default
31743204
# the incoming therefore has precedence
3205+
#
3206+
# Note that _fill_from_default doesn't mutate the receiver
31753207
job_config = job_config._fill_from_default(
31763208
self._default_query_job_config
31773209
)
@@ -3180,34 +3212,54 @@ def query(
31803212
self._default_query_job_config,
31813213
google.cloud.bigquery.job.QueryJobConfig,
31823214
)
3183-
job_config = copy.deepcopy(self._default_query_job_config)
3215+
job_config = self._default_query_job_config
31843216

3185-
job_ref = job._JobReference(job_id, project=project, location=location)
3186-
query_job = job.QueryJob(job_ref, query, client=self, job_config=job_config)
3217+
# Note that we haven't modified the original job_config (or
3218+
# _default_query_job_config) up to this point.
3219+
job_config_save = job_config
31873220

3188-
try:
3189-
query_job._begin(retry=retry, timeout=timeout)
3190-
except core_exceptions.Conflict as create_exc:
3191-
# The thought is if someone is providing their own job IDs and they get
3192-
# their job ID generation wrong, this could end up returning results for
3193-
# the wrong query. We thus only try to recover if job ID was not given.
3194-
if job_id_given:
3195-
raise create_exc
3221+
def do_query():
3222+
# Make a copy now, so that original doesn't get changed by the process
3223+
# below and to facilitate retry
3224+
job_config = copy.deepcopy(job_config_save)
3225+
3226+
job_id = _make_job_id(job_id_save, job_id_prefix)
3227+
job_ref = job._JobReference(job_id, project=project, location=location)
3228+
query_job = job.QueryJob(job_ref, query, client=self, job_config=job_config)
31963229

31973230
try:
3198-
query_job = self.get_job(
3199-
job_id,
3200-
project=project,
3201-
location=location,
3202-
retry=retry,
3203-
timeout=timeout,
3204-
)
3205-
except core_exceptions.GoogleAPIError: # (includes RetryError)
3206-
raise create_exc
3231+
query_job._begin(retry=retry, timeout=timeout)
3232+
except core_exceptions.Conflict as create_exc:
3233+
# The thought is if someone is providing their own job IDs and they get
3234+
# their job ID generation wrong, this could end up returning results for
3235+
# the wrong query. We thus only try to recover if job ID was not given.
3236+
if job_id_given:
3237+
raise create_exc
3238+
3239+
try:
3240+
query_job = self.get_job(
3241+
job_id,
3242+
project=project,
3243+
location=location,
3244+
retry=retry,
3245+
timeout=timeout,
3246+
)
3247+
except core_exceptions.GoogleAPIError: # (includes RetryError)
3248+
raise create_exc
3249+
else:
3250+
return query_job
32073251
else:
32083252
return query_job
3209-
else:
3210-
return query_job
3253+
3254+
future = do_query()
3255+
# The future might be in a failed state now, but if it's
3256+
# unrecoverable, we'll find out when we ask for it's result, at which
3257+
# point, we may retry.
3258+
if not job_id_given:
3259+
future._retry_do_query = do_query # in case we have to retry later
3260+
future._job_retry = job_retry
3261+
3262+
return future
32113263

32123264
def insert_rows(
32133265
self,

‎google/cloud/bigquery/enums.py‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -259,23 +259,23 @@ class SqlTypeNames(str, enum.Enum):
259259
class SqlParameterScalarTypes:
260260
"""Supported scalar SQL query parameter types as type objects."""
261261

262-
STRING = ScalarQueryParameterType("STRING")
262+
BOOL = ScalarQueryParameterType("BOOL")
263+
BOOLEAN = ScalarQueryParameterType("BOOL")
264+
BIGDECIMAL = ScalarQueryParameterType("BIGNUMERIC")
265+
BIGNUMERIC = ScalarQueryParameterType("BIGNUMERIC")
263266
BYTES = ScalarQueryParameterType("BYTES")
264-
INTEGER = ScalarQueryParameterType("INT64")
265-
INT64 = ScalarQueryParameterType("INT64")
267+
DATE = ScalarQueryParameterType("DATE")
268+
DATETIME = ScalarQueryParameterType("DATETIME")
269+
DECIMAL = ScalarQueryParameterType("NUMERIC")
266270
FLOAT = ScalarQueryParameterType("FLOAT64")
267271
FLOAT64 = ScalarQueryParameterType("FLOAT64")
268-
NUMERIC = ScalarQueryParameterType("NUMERIC")
269-
BIGNUMERIC = ScalarQueryParameterType("BIGNUMERIC")
270-
DECIMAL = ScalarQueryParameterType("NUMERIC")
271-
BIGDECIMAL = ScalarQueryParameterType("BIGNUMERIC")
272-
BOOLEAN = ScalarQueryParameterType("BOOL")
273-
BOOL = ScalarQueryParameterType("BOOL")
274272
GEOGRAPHY = ScalarQueryParameterType("GEOGRAPHY")
275-
TIMESTAMP = ScalarQueryParameterType("TIMESTAMP")
276-
DATE = ScalarQueryParameterType("DATE")
273+
INT64 = ScalarQueryParameterType("INT64")
274+
INTEGER = ScalarQueryParameterType("INT64")
275+
NUMERIC = ScalarQueryParameterType("NUMERIC")
276+
STRING = ScalarQueryParameterType("STRING")
277277
TIME = ScalarQueryParameterType("TIME")
278-
DATETIME = ScalarQueryParameterType("DATETIME")
278+
TIMESTAMP = ScalarQueryParameterType("TIMESTAMP")
279279

280280

281281
class WriteDisposition(object):

‎google/cloud/bigquery/job/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
from google.cloud.bigquery.job.base import ReservationUsage
2323
from google.cloud.bigquery.job.base import ScriptStatistics
2424
from google.cloud.bigquery.job.base import ScriptStackFrame
25+
from google.cloud.bigquery.job.base import TransactionInfo
2526
from google.cloud.bigquery.job.base import UnknownJob
2627
from google.cloud.bigquery.job.copy_ import CopyJob
2728
from google.cloud.bigquery.job.copy_ import CopyJobConfig
@@ -81,5 +82,6 @@
8182
"QueryPriority",
8283
"SchemaUpdateOption",
8384
"SourceFormat",
85+
"TransactionInfo",
8486
"WriteDisposition",
8587
]

‎google/cloud/bigquery/job/base.py‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
import http
2020
import threading
2121
import typing
22+
from typing import Dict, Optional
2223

2324
from google.api_core import exceptions
2425
import google.api_core.future.polling
@@ -88,6 +89,22 @@ def _error_result_to_exception(error_result):
8889
)
8990

9091

92+
class TransactionInfo(typing.NamedTuple):
93+
"""[Alpha] Information of a multi-statement transaction.
94+
95+
https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#TransactionInfo
96+
97+
.. versionadded:: 2.24.0
98+
"""
99+
100+
transaction_id: str
101+
"""Output only. ID of the transaction."""
102+
103+
@classmethod
104+
def from_api_repr(cls, transaction_info: Dict[str, str]) -> "TransactionInfo":
105+
return cls(transaction_info["transactionId"])
106+
107+
91108
class _JobReference(object):
92109
"""A reference to a job.
93110
@@ -336,6 +353,18 @@ def reservation_usage(self):
336353
for usage in usage_stats_raw
337354
]
338355

356+
@property
357+
def transaction_info(self) -> Optional[TransactionInfo]:
358+
"""Information of the multi-statement transaction if this job is part of one.
359+
360+
.. versionadded:: 2.24.0
361+
"""
362+
info = self._properties.get("statistics", {}).get("transactionInfo")
363+
if info is None:
364+
return None
365+
else:
366+
return TransactionInfo.from_api_repr(info)
367+
339368
@property
340369
def error_result(self):
341370
"""Error information about the job as a whole.

0 commit comments

Comments
 (0)