BigQuery: populate RowIterator.total_rows after running a query job #6117
Description
Request to have total_rows property public, under QueryJob() along with other stats
{{ message }}
Request to have total_rows property public, under QueryJob() along with other stats
Activity
shollyman commentedon Sep 27, 2018
My initial preference would be against exposing this directly as property of the QueryJob, as its part of the jobs.getQueryResults / tabledata.list response and not a member of the query stats metadata.
Is the interest in fetching the table statistics for the results without the extra fetch of destination table metadata, or are you interested in it related to progress while fetching row data?
yan-hic commentedon Sep 27, 2018
The interest is to determine how many rows a query, when completed, has generated.
Fetching the destination table metadata is not valid if records are appended to an existing table - I would get the total number of rows, not the addition.
Note that
totalRowsexists in bothjobs.getQueryResultsandjobs.query. There is no need to call the former (except if job is inserted) so request is not unreasonable. Also, https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#totalBytesProcessed made it as property ofQueryJob()so why nottotal_rows.shollyman commentedon Sep 27, 2018
Thanks for clarifying the interest is in the effective delta.
For the case where you run a query with a destination table and a
WRITE_APPENDdisposition, the results (and total_rows) will represent the whole table. For mutation DML queries (e.g. UPDATE/DELETE),num_dml_affected_rowsmay help, but we should consider exposing something like thestatistics.load.outputRowsequivalent for query jobs (likely contingent on statement type).If that sounds like what you're after, I'll request it for inclusion in the backend response. There's insufficient information in the existing response to correctly report the delta for the append case.
yan-hic commentedon Sep 27, 2018
Let's cancel the request as I just suggested making the existing - and perfectly working -
_query_results.total_rowsa public property ofQueryJob()or another object. I thought https://googlecloudplatform.github.io/google-cloud-python/latest/bigquery/generated/google.cloud.bigquery.job.QueryJob.result.html#google.cloud.bigquery.job.QueryJob.result would work but it doesn't,total_rowsisNone(myqueryjob.result().total_rows).This was reported on SO too: https://stackoverflow.com/questions/48799898/bigquery-py-client-library-v0-28-fetch-result-from-table-query-job
I will just use the private
_query_results.total_rows.Thanks.
tswast commentedon Jan 30, 2019
Let's write the
total_rowsvalue from theQueryResultsto theRowIteratorso that it's available without an additional (and awkward) API requests totabledata.listthat I do in the code sample at #7217.yan-hic commentedon Jan 30, 2019
Thanks Tim ! To add some additional context, I have also reported an unexpected behavior of the underlying
totalRecordsAPI property here[-]BigQuery: make QueryJob._query_results.total_rows public[/-][+]BigQuery: populate RowIterator.total_rows after running a query job[/+]