Repository navigation
BigQuery: error when using created table ref as query destination #6938
Description
Environment details
BigQuery API, Linux, Python 2.7
Steps to reproduce
I'm getting a 400 error when using a newly-created table ref as a destination:
Code example
from google.cloud import bigquery
client = biquery.Client()
export_ref = client.create_table('project_name.dataset.table')
config = bigquery.QueryJobConfig()
config.destination = export_ref
client.query('SELECT MY_FIELDS', job_config=config)Stack trace
Traceback (most recent call last):
File "bq_export.py", line 112, in <module>
main(args.table_ref[0])
File "bq_export.py", line 102, in main
export_block(client, table_ref, block_id, block[0], block[1])
File "bq_export.py", line 50, in export_block
q = client.query(query, job_config=q_config)
File "/home/user/.local/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 1439, in query
query_job._begin(retry=retry)
File "/home/user/.local/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 553, in _begin
retry, method="POST", path=path, data=self.to_api_repr()
File "/home/user/.local/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 382, in _call_api
return call()
File "/home/user/.local/lib/python2.7/site-packages/google/api_core/retry.py", line 270, in retry_wrapped_func
on_error=on_error,
File "/home/user/.local/lib/python2.7/site-packages/google/api_core/retry.py", line 179, in retry_target
return target()
File "/home/user/.local/lib/python2.7/site-packages/google/cloud/_http.py", line 319, in api_request
raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://www.googleapis.com/bigquery/v2/projects/my_bucket/jobs: Required parameter is missing
What appears to be happening when I enable httplib logging is that the table ref internal representation includes a bunch of extra fields. However, the error message is incredibly confusing, as it says something required is missing, when the docs mention very clear that all parameters are optional.
Activity
shollyman commentedon Dec 18, 2018
create_table() returns the Table, not a TableReference. To confirm, does it work when you change the setting of the destination to export_ref.reference?
Perhaps we should extract the TableReference when the destination is passed a Table.
[-]Error when using created table ref as query destination[/-][+]BigQuery: error when using created table ref as query destination[/+]tseaver commentedon Dec 18, 2018
@shollyman That seems sensible, given that
Client.create_tableitself is willing to take any of aTable, aTableReference, or a string for itstableargument. We've explicitly avoided doing client-side validation for the various job configuration fields, but that conversion would be a different thing.Likely we'd want to do the same for any job / job configuration arguments currently expecting
DatasetReference(s) orTableReference(s):LoadJob.destinationCopyJob.sourcesCopyJob.destinationExtractJob.sourceQueryJobConfig.default_datasetQueryJobConfig.destinationQueryJobConfig.ddl_target_tabletswast commentedon Apr 2, 2019
Thanks for the exhaustive list @tseaver
The specific request of query destination was fixed in #7560 It looks like the others were tackled as part of the client methods update in #6164