[PyLimit] adding skip and fetch with datafusion expr support - #1674
Open
hakunamatata-sb wants to merge 1 commit into
Open
[PyLimit] adding skip and fetch with datafusion expr support#1674hakunamatata-sb wants to merge 1 commit into
hakunamatata-sb wants to merge 1 commit into
Conversation
Author
|
@onestn @timsaucer requesting review and approval for workflows. |
Member
|
This is a good start, but if you want to really use these functions then I propose we add a full class wrapper for Limit instead of just the Rust exposed function. That's a bigger change because we currently don't have an expression module but I could see the repo going that way. If that's too large a lift, then maybe we open an issue to track wrapper coverage for all expressions. I'm a bit torn, because as a user I haven't needed access to these parts of the Expressions. @kosiew do you have thoughts on the idea? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1673
Rationale for this change
PyLimit(the Python wrapper for aLogicalPlan::Limitnode) currently exposes no way to read the actualLIMIT/OFFSETvalue from Python.skip()andfetch()were removed in #905 ("Upgrade to Datafusion 43") because upstream changedLimit.skip/Limit.fetchfromusize/Option<usize>toOption<Box<Expr>>(apache/datafusion#13028 — note the existingTODOcomment inlimit.rscites #12836, which is unrelated; #13028 is the actual causal PR), and the old methods no longer compiled against the new field types. Rather than update them, they were deleted, leaving aTODOand no replacement.This means any consumer walking a logical plan from Python — e.g. a custom SQL compiler/backend built on
datafusion-python— cannot determine whatLIMIT/OFFSETa query specified, even for a plain literal likeLIMIT 10. The value is still present and correctly parsed internally (Display for PyLimitprints it fine viaself.limit.skip/self.limit.fetch), it's just inaccessible as structured data. The only current workaround is regex-parsingrepr()/str()output.What changes are included in this PR?
crates/core/src/expr/limit.rs: restoreskip()/fetch()onPyLimit, now returningOption<PyExpr>instead of the oldOption<usize>, matching the new upstream field types (Option<Box<Expr>>). Conversion follows the same pattern already used elsewhere in this crate forExpr/Vec<Expr>fields (e.g.PyProjection::projections(),PyTableScan::py_filters()):This does not attempt to simplify/resolve non-literal expressions (e.g.
LIMIT $1, computed expressions) — it exposes whateverExprvariant the planner produced, unchanged. Callers should check the variant (e.g.Expr.variant_name() == "Literal") before assuming a value is resolvable, mirroring how DataFusion's own physical planner relies onSimplifyExpressionsand errors if it can't fold to a constant.python/tests/test_expr.py: updatedtest_limitto assert on the new accessors directly instead of only string-matchingrepr():Note:
Expr.python_value()returns a PyArrow scalar, whose__eq__only compares against other PyArrow scalars (pa.scalar(10) == 10isFalse) —.as_py()is used to get a plain Python value for comparison.Testing performed:
cargo check -p datafusion-python— compiles clean.maturin develop— builds the real extension (not just type-checked), installs editable into a venv with this repo's pinned dev deps.git submodule update --init testingwas required to populate test fixture data before running tests.pytest python/tests/test_expr.py::test_limit -v— passes against the live build.pytest python/tests/test_expr.py -v— full suite, 176 tests, all pass, no regressions.Are there any user-facing changes?
Yes. This adds two new public methods to
datafusion.expr.Limit:skip() -> Optional[Expr]fetch() -> Optional[Expr]There are no removals or signature changes to existing methods, so this is additive only — no breaking changes to public APIs. (No
api changelabel needed.)