Skip to content

BigQuery: preview / list_rows command for %%bigquery Jupyter magic #9105

Closed

Description

@tswast

Is your feature request related to a problem? Please describe.

When you want to get all rows in a table (or just a preview), it's more expensive to run a SELECT * query than the alternatives.

Describe the solution you'd like

  • If there is no whitespace in the query text, assume it's a table ID. Call list_rows (possibly with the BigQuery Storage API option, if that's set).
  • To allow for a preview, add a --max_results option to the %%bigquery magic. If not set, get all rows. If set, download at most max_results rows.

Describe alternatives you've considered

  • Add a special %%bigquery preview command. I prefer looking to see if the query text is just a table name, as that will be more consistent with pandas-gbq. Also, we'd prefer to limit the %%bigquery magic to just "queries". I think entering just a table ID as query text is clear that it's different from SELECT * (anti-pattern) but also the intention is clear.
  • Client-side query processing to automatically detect SELECT * [LIMIT N] queries client-side. I think it could make sense to parse queries client-side in just the %%bigquery magic (or opt-in via QueryJobConfig option), but this is much more complex than checking if there is any whitespace in query text.

Additional context

Related feature requests in pandas-gbq:

Activity

  1. tswast commented on Aug 26, 2019

    @tswast
    ContributorAuthor

    CC @shubha-rajan

    If you're interested, I think this could be a natural next task to get more familiar with the BigQuery API(s) and the Jupyter features.

  2. added
    api: bigqueryIssues related to the BigQuery API.
    type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.
    on Aug 26, 2019
  3. shubha-rajan commented on Aug 27, 2019

    @shubha-rajan
    Contributor

    @tswast sure, I'll take this one!

  4. plamut commented on Sep 25, 2019

    @plamut
    Contributor

    With #9170 now merged, this issue can be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: bigqueryIssues related to the BigQuery API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions