Skip to content

Add lazy SQL → xarray round-trip via XarrayDataFrame.to_dataset (closes #58) - #167

Merged
alxmrs merged 20 commits into
xqlsystems:mainfrom
ghostiee-11:feat/lazy-sql-to-xarray
Jun 20, 2026
Merged

Add lazy SQL → xarray round-trip via XarrayDataFrame.to_dataset (closes #58)#167
alxmrs merged 20 commits into
xqlsystems:mainfrom
ghostiee-11:feat/lazy-sql-to-xarray

Conversation

@ghostiee-11

@ghostiee-11 ghostiee-11 commented May 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #58. ctx.sql(...) now returns an XarrayDataFrame that adds .to_dataset() on top of the existing .to_pandas(). The Dataset is lazy by default slicing pushes down into SQL WHERE clauses so only the requested slab gets materialized.

Why

Going Dataset → SQL worked great. Going the other way meant hand-rolling set_index().to_xarray() and waving goodbye to attrs, encoding, and non-dim coords. This closes the loop. API shape follows the wrapper proposal in the issue thread; lazy-default via BackendArray + LazilyIndexedArray is the direction that came out of the design discussion.

Before / After

Before

df = ctx.sql('SELECT * FROM "air"').to_pandas()        # +194 MB RSS
ds = df.set_index(['time', 'lat', 'lon']).to_xarray()  # attrs / encoding / non-dim coords gone

After

out  = ctx.sql('SELECT * FROM "air"').to_dataset()     # +2 MB RSS, lazy
slab = out['air'].isel(time=0).values                  # +15 MB, single wrapped SQL with WHERE time=...
# attrs, encoding (dtype-bound keys stripped), non-dim coords, dim dtype all back

RSS on xr.tutorial.open_dataset('air_temperature') (~31 MB):

Path RSS delta
ctx.sql(...).to_pandas() +194 MB
Hand-written WHERE query +18 MB
Lazy to_dataset() + isel(time=0).values +17 MB

About 11× lighter than the eager path, basically tied with hand-written filtered SQL.

Scope

  • xarray_sql/ds.py: XarrayDataFrame + SQLBackendArray (pushes indexers down to SQL WHERE) + template-based metadata recovery
  • XarrayContext.sql() returns the wrapper; .to_pandas() stays bit-for-bit identical (there's a regression test for this)
  • Lazy via BackendArray + LazilyIndexedArray with IndexingSupport.OUTER; dask stays orthogonal
  • sparse_extent="result"|"template" for filtered queries
  • 48 new tests, README example

Follow-ups per the issue thread: chunking / rechunker primitive, hypothesis property tests, simulation testing, promoting the standalone helper to public.

Test plan

  • uv run pytest tests/ - 158/158
  • uv run ruff check xarray_sql/ tests/ clean
  • uv run ruff format -check xarray_sql/ tests/ clean
  • uv run python -m mypy xarray_sql/ - 4 errors, all pre-existing (net −1 vs main)
  • README example runs end-to-end

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inverse problem: Read a table or SQL query into an Xarray Dataset

2 participants