[adapters] Support JSON functions in ad-hoc queries#6645
Conversation
mythical-fred
left a comment
There was a problem hiding this comment.
Approve. Registers datafusion_functions_json::register_all on the single spot every ad-hoc SessionContext flows through (create_session_context in adapterlib), so the fix reaches ad-hoc queries and the Python test_shared_pipeline path without touching call sites. The layering is right and the surface is minimal.
Testing is thorough for what this changes:
- Rust unit test
create_session_context_registers_json_functionsenumerates the 11 registered functions on a fresh context — that's a good tripwire against upstream renames. json_functions_work_in_adhoc_queries(adapters/adhoc) starts with a negative assertion that stock DataFusion does not providejson_get_str, then covers typed getters, missing-key/NULL-document semantics, JSON predicates, the->>operator, and theCAST(json_get(...) AS BIGINT)rewrite. That negative-first shape is exactly right — if DataFusion ever gains native JSON functions the test will yell.- Python
test_adhoc_json_functionsmirrors the Rust coverage end-to-end through the pipeline manager: VARIANT round-trip, typed getters, nested paths (object → array-index),json_contains,json_get_bool,->>, and the cast rewrite.
Docs read cleanly — the function table, the 0-based-vs-1-based array-index footgun (called out in the "Differences" bullet and the section body), the -> / ? operator limitation, and the json_get union-return caveat with the concrete workaround are all there. That is the level of care ad-hoc-vs-Feldera-SQL asymmetries need.
Commits are three focused pieces, all Signed-off-by, no AI trailers. CI main is green at tip.
One transitive-dep nit below on jiter / pyo3 bloat — not a blocker.
| # JSON functions (json_get_str, json_contains, ...) for ad-hoc queries over | ||
| # VARIANT columns, which reach DataFusion as JSON-encoded strings. | ||
| datafusion-functions-json = "0.53.1" | ||
| dbsp = { path = "crates/dbsp", version = "0.322.0" } |
There was a problem hiding this comment.
Nit (transitive-dep bloat, not a blocker): datafusion-functions-json 0.53.1 depends on jiter with default features on, and jiter's python feature (which pulls pyo3, pyo3-build-config, pyo3-ffi, pyo3-macros, pyo3-macros-backend, target-lexicon) is enabled by default. You can see the fallout in Cargo.lock — five new pyo3-* crates arrive in a pure-Rust adapter. If upstream is willing to accept default-features = false on their jiter dep (or expose a python-off feature), we'd cut a chunk of compile time and binary size on every build. Worth an upstream nudge; not something to fix here.
| // strings apart. Of the crate's operators only `->>` is reachable: | ||
| // `->` and `?` do not parse in the default (generic) SQL dialect. | ||
| datafusion_functions_json::register_all(&mut state) | ||
| .expect("registering JSON functions on a fresh session state cannot fail"); |
There was a problem hiding this comment.
Nit: expect("registering JSON functions on a fresh session state cannot fail") reads clearly, but as a matter of style the message describes why rather than what would be violated. Something like "datafusion_functions_json::register_all on a fresh SessionState should never fail — no name collisions possible" gives the future reader a hint about which invariant just broke if this ever fires. Cosmetic; leave as-is if you disagree.
Shorten dependency and test comments, drop issue references, and remove output-format details and the mention of unavailable operators from the ad-hoc JSON functions documentation (PR #6645 review).
Register the datafusion-functions-json UDF family (json_get_str, json_get_int, json_contains, ..., and the ->> operator) in every DataFusion session context Feldera creates, so ad-hoc queries can take apart VARIANT columns, which reach DataFusion as JSON-encoded strings. Casting json_get is rewritten to the matching typed getter. The -> and ? operators from the crate do not parse in the default (generic) SQL dialect and stay unavailable. Fixes #6644 Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Feed JSON documents into a materialized table with a VARIANT column and exercise the datafusion-functions-json surface end to end: typed getters, nested paths with 0-based array indexes, JSON predicates, the ->> operator, and the cast-to-typed-getter rewrite (issue #6644). Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Add a section on querying VARIANT columns with datafusion-functions-json to the ad-hoc query page: the function table, path semantics (0-based array indexes, unlike 1-based VARIANT subscripts in Feldera SQL), and the limits (json_get's union return type, unavailable -> and ? operators). Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Shorten dependency and test comments, drop issue references, and remove output-format details and the mention of unavailable operators from the ad-hoc JSON functions documentation (PR #6645 review).
|
This seems to have failed due to a download problem, so I just requeued it. |
Register the datafusion-functions-json UDF family (json_get_str,
json_get_int, json_contains, ..., and the ->> operator) in every
DataFusion session context Feldera creates, so ad-hoc queries can take
apart VARIANT columns, which reach DataFusion as JSON-encoded strings.
Casting json_get is rewritten to the matching typed getter. The ->
and ? operators from the crate do not parse in the default (generic)
SQL dialect and stay unavailable.
Fixes #6644
Describe Manual Test Plan
Tested manually with a simple pipeline using webconsole.
Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes