This repository was archived by the owner on Mar 6, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 6, 2026. It is now read-only.
Returned API errors are an HTML rendering, causing exception in client #1296
Description
If an error occurs whilst inserting rows, an exception is raised by the client with rendered HTML, instead of the errors being returned as documented here.
This is probably an integration testing problem. I guess that the BigQuery API has changed its default error response format to rendered HTML since this code was written. It may alternatively be that the client only handles row-level errors correctly but raises non-row-level exceptions. If the latter, then:
- these additional raises should be documented (current docs say
:raises: ValueError – if table’s schema is not set or rows is not a Sequence), and - the error raised should include meaningful information, not just
<!DOCTYPE html>
Environment details
- OS type and version: whatever google cloud functions run on (debian?)
- Python version: 3.9
- pip version: n/a
google-cloud-bigqueryversion: 2.34.3
Steps to reproduce
NOTE: I'm only assuming this is a problem with maximum row size, due to the 413 error I receive and the fact that the same code succeeds on smaller inserts. It has to be an assumption because the whole problem is I can't see the error!
- Prepare a table (in this case I call it sensor_data) with whatever schema
- Using the following code sample... https://cloud.google.com/bigquery/docs/samples/bigquery-table-insert-rows
- Insert data into the table using
client.insert_rows:
rows = [
# ...create a lot of data, enough to violate the limits here: https://cloud.google.com/bigquery/quotas#streaming_inserts
]
client = bigquery.Client()
table=self.client.get_table(self.table_names["sensor_data"])
errors = client.insert_rows(table=table, rows=rows)Stack trace
Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 2073, in wsgi_app response = self.full_dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1518, in full_dispatch_request rv = self.handle_user_exception(e)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request rv = self.dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1502, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/__init__.py", line 171, in view_func function(data, context)
File "/workspace/main.py", line 38, in upload_window window_handler.persist_window(window, window_metadata)
File "/workspace/window_handler.py", line 91, in persist_window self.dataset.add_sensor_data(
File "/workspace/big_query.py", line 99, in add_sensor_data errors = self.client.insert_rows(table=self.client.get_table(self.table_names["sensor_data"]), rows=rows)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 3466, in insert_rows return self.insert_rows_json(table, json_rows, **kwargs)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 3646, in insert_rows_json response = self._call_api(
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 782, in _call_api return call()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func return retry_target(
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/retry.py", line 190, in retry_target return target()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 494, in api_request raise exceptions.from_http_response(response) google.api_core.exceptions.GoogleAPICall
Error: 413 POST https://bigquery.googleapis.com/bigquery/v2/projects/aerosense-twined/datasets/greta/tables/sensor_data/insertAll?prettyPrint=false: <!DOCTYPE html>
This prevents us from ever seeing what the actual error is that would be shown on the page.
Thanks!
Activity
shollyman commentedon Jul 28, 2022
This is best addressed by augmenting the docs for the python library.
Additional context: If you try to insert vastly oversized volumes of data in a single request, the error (and the html output) is emitted by intermediate components in the serving path.
There's an existing test in the go library that tests this behavior: https://github.com/googleapis/google-cloud-go/blob/5b7bfebcd4a0fd3cbe355d9d290e6b5101810b7e/bigquery/integration_test.go#L841
Assigning to one of the python maintainers to follow up here.
thclark commentedon Nov 18, 2022
Thanks for the clarification and docs, both.
From the perspective of the python client user that’s still not that useful, since the traceback still hides the error so you’re left guessing about what happened.
It sounds like the errors must be returned from the API in html for good reasons, but perhaps we could look into parsing that for meaning before injecting it into a traceback?
Anything I can do to help with that?
aribray commentedon Nov 21, 2022
I replicated this error using the
insert_rows()method, and my traceback showed the full error message: