feat: Bigquery as OLAP engine by k-anshul · Pull Request #9161 · rilldata/rill

k-anshul · 2026-04-01T12:11:40Z

closes https://linear.app/rilldata/issue/PLAT-450/metrics-views-on-bigquery

Added

TODOs to be done with follow ups:

Exports are broken
remove conversion of civil.Date to time.Time in the rill driver and handle it wherever required

Checklist:

Covered by tests
Ran it and it works as intended
Reviewed the diff before requesting a review
Checked for unhandled edge cases
Linked the issues it closes
Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
Intend to cherry-pick into the release branch
I'm proud of this work!

k-anshul · 2026-04-02T11:54:59Z

+	}
+
+	rangeSQL := fmt.Sprintf(
+		"SELECT min(%[1]s) as `min`, max(%[1]s) as `max`, %[2]s as `watermark` FROM %[3]s %[4]s",


This is not an efficient query even when running on partition column

An optimization can be done where we check if this is the partition column in the table and directly check on min/max partition metadata.
Given this is an often executed query I think it can done in a follow-up. @begelundmuller thoughts ?

If the optimization can be done in a fast/cheap/safe way, then yeah it sounds good to me

Can be fast but to ensure that we do not query information_schema again and again, we need to cache the information that this is the partition column in the table so require some changes. Will take it up separately .

begelundmuller · 2026-04-03T16:02:03Z

@@ -180,33 +181,157 @@ func (q *TableHead) generalExport(ctx context.Context, rt *runtime.Runtime, inst
 }

 func (q *TableHead) buildTableHeadSQL(ctx context.Context, olap drivers.OLAPStore) (string, error) {


It seems like there's a huge complexity increase in this function. Two questions:

We don't run TableHead very often, so is it necessary to optimize it so hard? In general, I would assume people who connect a BI tool to a data warehouse are fine with a SELECT * FROM tbl LIMIT 100 query being run.

If it really is necessary, is it possible to combine it into one nested query and push it into the dialect somehow?

It is used in data preview. On a 100 TB table this can cost a user 600 dollars. This can be a silent "trap" for a user given BigQuery returns result very fast (as reported by users running such queries on big tables).
I agree that users should not use bytes processed based pricing when connecting to a BI tool but we should not leave such traps for users.
For example, I found this issue in superset where the reporter refused to use superset with BigQuery till this kind of queries are removed : Select * Limit is DANGEROUS in BigQuery apache/superset#17299

For partition pruning the filter has to be a static filter and using dynamic filter is not allowed.

If you are worried about dialect specific complexity in runtime/queries then we can take one of the following approaches:

Disable data preview for BigQuery in UI and return an error in the API.

Use preview table API which is free : https://docs.cloud.google.com/bigquery/docs/samples/bigquery-browse-table#bigquery_browse_table-go

Both approaches make this more optimised given we don't have to scan even 1 partition (which can still be big).

That makes sense. Yeah I'm just a little worried about the driver-specificity in TableHead, especially given we are not adding many new OLAP drivers.

I don't think we should disable previews, but it would just be nice if we could push this into the driver somehow. I'm good with any of these:

Rewrite SELECT * FROM tbl LIMIT n into preview API calls inside OLAPStore.Query itself (similar to the code we have here:

rill/runtime/drivers/bigquery/warehouse.go

Lines 38 to 39 in 93b278f

// Regex to parse BigQuery SELECT ALL statement: SELECT * FROM `project_id.dataset.table`

var selectQueryRegex = regexp.MustCompile(

)

Add a Head function on the OLAPStore interface (other drives can implement using a normal SELECT *)

Add to the drivers.Dialect somehow (will become clean with Naman's refactors)

I implemented 2nd option. It leads to some duplicate code but seemed cleanest/safest.

begelundmuller · 2026-04-14T11:29:40Z

  type: "object",
  title: "BigQuery",
-  "x-category": "warehouse",
+  "x-category": "olap",


This might be problematic; most people will probably still wan't to use BigQuery as source for now? I believe applications would work on a way to give users a choice between OLAP and source for Snowflake and BigQuery, but if that hasn't landed yet, probably we should stick with the old default?

Actually @nishantmonu51 asked to move Snowflake to OLAP engine for now and handle it as warehouse for duckdb in subsequent OLAP works.
I followed the same approach here.
But on reflection, given BigQuery OLAP connector has been a mixed experience, I am okay to leave it as warehouse.
Thoughts @begelundmuller @nishantmonu51 ?

Are you aware of this thread? I think Applications had to patch that change as it was breaking some flows. https://rilldata.slack.com/archives/C093UBT5NLV/p1775582170699349?thread_ts=1775491222.470509&cid=C093UBT5NLV

However, I see their patch didn't involve this specific flag, so maybe you need to change something else. I'll let you look at the thread and the patch and make any necessary changes. I do think we should not break the BigQuery as warehouse flows just yet.

No. I wasn't ware of this. Reverted all UI changes per the patch fix.

begelundmuller

Looks good!

k-anshul added 4 commits April 1, 2026 14:00

feat: bigquery as olap engine

e900703

revert table partition column as timeseries

d40b83a

better error msg for max bytes billed

6752fc2

self review

942cebe

k-anshul self-assigned this Apr 1, 2026

k-anshul added 3 commits April 2, 2026 11:31

unit tests fix

814148e

more fixes

d3cd995

full join fix

e07d8f0

k-anshul commented Apr 2, 2026

View reviewed changes

also add other partition

1ad44e3

k-anshul requested a review from begelundmuller April 2, 2026 13:08

k-anshul added 3 commits April 3, 2026 13:00

timezone related changes

a20da27

add unit tests

1e3d684

small query change

b37d12a

begelundmuller requested changes Apr 3, 2026

View reviewed changes

k-anshul added 5 commits April 6, 2026 14:59

review comments - 1

7459d89

review comments - 2

8a7c21c

handle civil types explicitly

f246b3d

remove time comparison for schema validation for bigquery

90300e3

nits

ee65102

begelundmuller requested changes Apr 14, 2026

View reviewed changes

k-anshul added 6 commits April 15, 2026 10:54

nits

7eb2e3d

docs generate

90528cb

cast to timestamp in metrics view timerange

1a40e3a

revert UI changes

55062ad

Merge remote-tracking branch 'origin/main' into bigquery_olap

3fa908c

Merge remote-tracking branch 'origin/main' into bigquery_olap

1277b16

begelundmuller approved these changes Apr 16, 2026

View reviewed changes

k-anshul merged commit 586a914 into main Apr 17, 2026
15 of 18 checks passed

k-anshul deleted the bigquery_olap branch April 17, 2026 04:48

		@@ -180,33 +181,157 @@ func (q TableHead) generalExport(ctx context.Context, rt runtime.Runtime, inst
		}

		func (q *TableHead) buildTableHeadSQL(ctx context.Context, olap drivers.OLAPStore) (string, error) {

	// Regex to parse BigQuery SELECT ALL statement: SELECT * FROM `project_id.dataset.table`
	var selectQueryRegex = regexp.MustCompile(

Conversation

k-anshul commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k-anshul Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

begelundmuller left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

k-anshul commented Apr 1, 2026 •

edited

Loading

k-anshul Apr 6, 2026 •

edited

Loading