diff --git a/.ai/skills/audit-skill-md/SKILL.md b/.ai/skills/audit-skill-md/SKILL.md new file mode 100644 index 000000000..ba5255a59 --- /dev/null +++ b/.ai/skills/audit-skill-md/SKILL.md @@ -0,0 +1,290 @@ + + +--- +name: audit-skill-md +description: Audit the user-facing skill at skills/datafusion_python/SKILL.md against the current public Python API. Find new APIs that should be documented, stale mentions of removed/renamed APIs, examples that drifted from current idiomatic style, and places that need a "requires datafusion-python NN or newer" note. Run after upstream syncs and before each release. +argument-hint: [scope] (e.g., "session-context", "dataframe", "expr", "functions", "patterns", "pitfalls", "version-notes", "all") +--- + +# Audit `skills/datafusion_python/SKILL.md` + +You are auditing the user-facing skill at +[`skills/datafusion_python/SKILL.md`](../../skills/datafusion_python/SKILL.md) +against the current state of the Python API. The skill is the source of truth +for how AI coding assistants are taught to write `datafusion-python` code, so +it must match what the project actually ships. This skill identifies gaps +caused by upstream syncs, refactors, or renames, and (if asked) applies the +edits directly to `SKILL.md`. + +The skill is most usefully run **after** the `check-upstream` step of an +upstream sync (see `dev/release/upstream-sync.md`) — once any new APIs are +exposed, this skill makes sure they get documented. + +## What the skill covers + +The user-facing `SKILL.md` documents these public surfaces. This list is not +exhaustive — if a new top-level area is added (e.g., a new `Catalog` API +exposed at the package root), include it. + +| Surface | Module | Sections in SKILL.md | +|---|---|---| +| `SessionContext` | `python/datafusion/context.py` | "Data Loading" | +| `DataFrame` | `python/datafusion/dataframe.py` | "DataFrame Operations Quick Reference", "Executing and Collecting Results", "Idiomatic Patterns" | +| `Expr` | `python/datafusion/expr.py` | "Expression Building", "Common Pitfalls" | +| `functions` | `python/datafusion/functions/__init__.py` | "Available Functions (Categorized)", scattered uses throughout | +| `functions.spark` | `python/datafusion/functions/spark.py` | "Available Functions (Categorized)" → "Spark-Compatible Functions" subsection | +| Top-level helpers (`col`, `lit`, `WindowFrame`, ...) | `python/datafusion/__init__.py` | "Import Conventions", "Core Abstractions" | + +## Scope argument + +The user may specify a scope via `$ARGUMENTS` to limit the audit. If no scope +is given or `all` is specified, audit every area. + +| Scope | Audit target | +|---|---| +| `session-context` | `SessionContext` methods and the "Data Loading" section | +| `dataframe` | `DataFrame` methods and the operations / executing / patterns sections | +| `expr` | `Expr` methods/operators and the "Expression Building" section | +| `functions` | `functions/__init__.py` `__all__` and the "Available Functions (Categorized)" section | +| `spark-functions` | `functions/spark.py` `__all__`, the "Spark-Compatible Functions" subsection, and the divergent-semantics table | +| `patterns` | "Idiomatic Patterns" section — confirm patterns still match recommended style | +| `pitfalls` | "Common Pitfalls" — confirm each pitfall still reproduces, drop ones fixed upstream | +| `version-notes` | Cross-check version annotations (see below) | +| `all` | Everything above | + +## Inputs to read + +Before producing the report: + +1. `skills/datafusion_python/SKILL.md` — the document being audited. +2. The relevant Python module(s) for the chosen scope. Public surface is the + `__all__` list (where defined) plus `class` and `def` symbols not prefixed + with `_`. +3. `Cargo.toml` (root) for the current `datafusion-python` version — read + the `version` field under `[workspace.package]` (format `NN.0.0`). The + major version always matches the upstream `datafusion` crate, so a + single `datafusion-python` version expresses both. + `python/datafusion/__init__.py`'s `__version__` is the same value + exposed at runtime. +4. Recent commits touching the relevant module(s) for context on what + changed since the last sync: + ```bash + git log --oneline -- python/datafusion/dataframe.py | head -20 + ``` + +## What to look for + +Walk through each scoped area and flag four kinds of issues. + +### 1. New APIs not mentioned + +For each public symbol in the module's `__all__` (or each public class +method), check whether it appears anywhere in `SKILL.md`. A symbol is +"covered" if it shows up in: + +- A code block (the strongest signal — it's demonstrated). +- The "Available Functions (Categorized)" list. +- The SQL-to-DataFrame Reference table. + +**Decide whether each missing symbol deserves an entry.** Not every public +symbol belongs in `SKILL.md` — the skill is curated for the patterns users +hit daily, not exhaustive API reference. Use these heuristics: + +- **Add it** if it replaces or supersedes something already in the skill + (e.g., a new operation that is the idiomatic alternative to a documented + workaround). +- **Add it** if it fits a category already present (a new aggregate function + goes in the aggregate list; a new join type goes in the joining section). +- **Add it** if it changes how a documented pattern should be written. +- **Skip it** if it is genuinely niche / advanced / experimental. +- **Skip it** if it is internal plumbing exposed for FFI but not user-facing. + +When you flag a missing symbol, include a one-line proposed insertion point +(which section / which table row) so a reviewer can decide quickly. + +### 2. Stale mentions + +For each function name, method name, or import shown in `SKILL.md`, verify it +still exists in the current API: + +- Function names mentioned in prose or in the categorized list should appear + in `python/datafusion/functions/__init__.py`'s `__all__`. +- Spark function names mentioned in the "Spark-Compatible Functions" + subsection should appear in `python/datafusion/functions/spark.py`'s + `__all__`. Also confirm the divergent-semantics table still matches the + current spark vs. main signatures. +- Method calls in code blocks should resolve against the current class. +- Imports (`from datafusion import ...`) should succeed against the current + `__init__.py`. + +A quick way to check imports without running them: + +```bash +python -c "from datafusion import SessionContext, col, lit; from datafusion import functions as F; print('ok')" +``` + +For each stale mention, propose either: +- a rename to the current name, or +- removal if the API is gone with no replacement. + +### 3. Examples that drifted from idiomatic style + +The skill teaches a Pythonic style: prefer plain strings to `col(...)` when a +column reference is all you need; prefer raw Python values to `lit(...)` +where auto-wrapping applies. Recent refactors (see the `make-pythonic` +skill) keep moving more functions toward accepting native types. + +For each code example in `SKILL.md`, check: + +- Does it use `lit(value)` where a raw value would work? Comparison RHS, + arithmetic with a column, etc. all auto-wrap. (Reserve `lit()` for the + cases listed in pitfall #2.) +- Does it use `col("name")` where a plain string would work? `select(...)`, + `aggregate([keys], ...)`, `sort(...)`, `sort_by(...)` all accept plain + name strings. +- Do `functions.py` calls match the current pythonic signature for that + function? If `make-pythonic` recently changed a signature (e.g., + `repeat(string, n: Expr | int)`), the example should pass `3` rather than + `lit(3)`. +- Does any example use a deprecated or removed parameter name? + +For drift, propose the updated snippet. If the change is purely stylistic +and the older form still works, mark the suggestion as **non-blocking**. + +### 4. Missing or stale version notes + +When an API depends on a specific version, the skill should say so — +otherwise an agent referencing the skill in an older project will write +code that fails at import or at runtime. + +`datafusion-python` shares its major version number with the upstream +`datafusion` crate (e.g., `datafusion-python 53.x` tracks upstream +`datafusion 53`). Always express version requirements in terms of +`datafusion-python` only — there is no need to call out upstream and +package versions separately. + +Add a version note when: + +- A method or function shown in the skill was added in a specific release + (e.g., a new `DataFrame` method that didn't exist before 53). +- A breaking change altered behavior in a specific release (signature + change, default-value change, new required argument). +- A pitfall was fixed in a specific release. Either annotate the pitfall + block with "fixed in datafusion-python NN, kept here for users on older + versions" or remove it once the supported floor moves past that version. + +Format for version notes (inline, italicized): + +```markdown +*Requires datafusion-python 53 or newer.* +``` + +For each missing/stale version note, propose the exact line and where it +belongs. + +## How to discover changes since the last audit + +If the user supplies a previous version or commit SHA where the audit was +last run, diff against it: + +```bash +# Public-API-relevant changes since SHA +git log --oneline ..HEAD -- python/datafusion/ + +# Whose signatures actually moved +git diff ..HEAD -- python/datafusion/functions.py | grep '^[+-]def ' +``` + +If no prior audit point is given, fall back to "since the last upstream +sync" by inspecting commits that touch `Cargo.toml`'s `datafusion` pin: + +```bash +git log --oneline -- Cargo.toml | grep -i datafusion | head -5 +``` + +## Output Format + +Produce a report grouped by scope. Each finding is one bullet with a +proposed action, so a maintainer can review the list quickly and apply +edits in order. + +``` +## SKILL.md Audit (scope: ) + +Audited against: +- skills/datafusion_python/SKILL.md @ +- datafusion-python + +### New APIs to cover +- `DataFrame.foo()` — added in datafusion-python 53. Insert in "DataFrame Operations Quick Reference" under . + Proposed snippet: + ```python + df.foo(...) + ``` + +### Stale mentions +- "old_function_name" referenced in the categorized list (line N) — renamed to "new_function_name". Replace. + +### Drifted examples +- "Filtering" section, `df.filter(col("a") > lit(10))` — drop `lit(10)`, auto-wrap applies. (non-blocking) +- "Aggregation" section, `df.aggregate([col("region")], ...)` — pass `"region"` as a plain string per "Projection" guidance. + +### Version notes +- `DataFrame.foo()` block needs *Requires datafusion-python 53 or newer.* +- "Common Pitfalls" #N — fixed in datafusion-python 53; remove the pitfall and update the SQL-to-DataFrame row to no longer flag the workaround. + +### No-change confirmed +- `SessionContext` data-loading section — all entries match current API. +``` + +If asked to apply the changes, edit `skills/datafusion_python/SKILL.md` +directly with `Edit` tool calls, one finding at a time, and re-run the +relevant doctest sanity check at the end: + +```bash +pytest --doctest-modules python/datafusion -q +``` + +## What NOT to flag + +- **Internal helpers / underscored names.** Private symbols are not part of + the user-facing surface. +- **Functions intentionally omitted.** Niche / advanced APIs (custom + catalogs, raw FFI plumbing, low-level execution plan accessors) live in + the API reference, not the skill. If an omission was deliberate and a + comment / commit explains why, leave it out. +- **Style nits inside explanatory prose.** The skill mixes example code and + prose; only enforce the pythonic style on actual code blocks. +- **Function-by-function coverage of every `functions.py` symbol.** The + "Available Functions (Categorized)" list is curated by category, not + exhaustive. Adding a single new aggregate to the aggregate list is + enough — the user follows the pointer to the API reference for the rest. + +## Coordination with other skills + +- Run `/check-upstream` first to expose any missing upstream APIs into the + Python layer. Without that, this skill cannot recommend documenting + something that is not yet exposed. +- Run `/make-pythonic` before this skill if a Pythonic-signature pass is + planned for a release — that way this skill can update examples to the + final signature in one shot rather than churning them twice. +- The order during an upstream sync (PR 3 of `dev/release/upstream-sync.md`) + is therefore: `/check-upstream` → `/make-pythonic` (optional) → + `/audit-skill-md`. diff --git a/.ai/skills/check-upstream/SKILL.md b/.ai/skills/check-upstream/SKILL.md new file mode 100644 index 000000000..a3d82a670 --- /dev/null +++ b/.ai/skills/check-upstream/SKILL.md @@ -0,0 +1,477 @@ + + +--- +name: check-upstream +description: Check if upstream Apache DataFusion features (functions, DataFrame ops, SessionContext methods, FFI types) are exposed in this Python project. Use when adding missing functions, auditing API coverage, or ensuring parity with upstream. +argument-hint: [area] (e.g., "scalar functions", "aggregate functions", "window functions", "dataframe", "session context", "ffi types", "all") +--- + +# Check Upstream DataFusion Feature Coverage + +You are auditing the datafusion-python project to find features from the upstream Apache DataFusion Rust library that are **not yet exposed** in this Python binding project. Your goal is to identify gaps and, if asked, implement the missing bindings. + +**IMPORTANT: The Python API is the source of truth for coverage.** A function or method is considered "exposed" if it exists in the Python API (e.g., `python/datafusion/functions.py`), even if there is no corresponding entry in the Rust bindings. Many upstream functions are aliases of other functions — the Python layer can expose these aliases by calling a different underlying Rust binding. Do NOT report a function as missing if it appears in the Python `__all__` list and has a working implementation, regardless of whether a matching `#[pyfunction]` exists in Rust. + +**IMPORTANT: audit the total upstream surface, not the delta since the last pin.** Gaps accumulate across syncs. A patch-release bump with a "bug fixes only" changelog does not mean there is nothing to find — pre-existing gaps from earlier majors still need to be surfaced. Always run the full comparison. + +## Compile-Signal Triggers + +If a recent upstream bump required *any* of the following while fixing +compile errors in `crates/core/` or the FFI example, treat that as a +**hard signal** that user-facing surface area grew and run this skill +before considering the bump done. Each pattern corresponds to a class of +gap that frequently shows up in the audit: + +| Signal during PR 1 compile fix | Likely gap to check | +|---|---| +| New `Expr::*` variant added to a non-exhaustive `match` (`HigherOrderFunction`, `Lambda`, `LambdaVariable`, …) | New lambda / higher-order scalar functions (`any_match`, `array_transform`, `list_transform`, …) | +| New `ScalarValue::*` variant (`ListView`, `LargeListView`, …) | New scalar / array functions that consume or produce the type | +| New required trait method on `ExecutionPlan` / `TableProvider` / `*UDFImpl` (`apply_expressions`, …) | Corresponding capability on the Python wrapper class | +| Renamed or restructured struct field (e.g. `Cast.data_type` → `Cast.field: FieldRef`) | Any Python accessor / SKILL.md doc that read the old field | +| Newly deprecated trait method with a `_with_args` / `_with_options` replacement | The `*_with_options` variant frequently warrants a separate Python entry point | + +PR 1 of `dev/release/upstream-sync.md` asks you to log these signals as +they appear. When you run this skill, use that log as a checklist: every +entry must either show up in the audit output or be explicitly skipped +with a reason. + +## Areas to Check + +The user may specify an area via `$ARGUMENTS`. If no area is specified or "all" is given, check all areas. + +### 1. Scalar Functions + +**Upstream source of truth:** +- Rust docs: https://docs.rs/datafusion/latest/datafusion/functions/index.html +- User docs: https://datafusion.apache.org/user-guide/sql/scalar_functions.html + +**Where they are exposed in this project:** +- Python API: `python/datafusion/functions.py` — each function wraps a call to `datafusion._internal.functions` +- Rust bindings: `crates/core/src/functions.rs` — `#[pyfunction]` definitions registered via `init_module()` + +**Evaluated and not requiring separate Python exposure:** +- `get_field_path` — already covered by `get_field(expr, *names)`, which takes a + variadic field path and dispatches to the same underlying + `functions::core::get_field` UDF as the upstream `get_field_path` helper. + +**How to check:** +1. Fetch the upstream scalar function documentation page +2. Compare against functions listed in `python/datafusion/functions.py` (check the `__all__` list and function definitions) +3. A function is covered if it exists in the Python API — it does NOT need a dedicated Rust `#[pyfunction]`. Many functions are aliases that reuse another function's Rust binding. +4. Check against the "evaluated and not requiring exposure" list before flagging as a gap +5. Only report functions that are missing from the Python `__all__` list / function definitions + +### 2. Aggregate Functions + +**Upstream source of truth:** +- Rust docs: https://docs.rs/datafusion/latest/datafusion/functions_aggregate/index.html +- User docs: https://datafusion.apache.org/user-guide/sql/aggregate_functions.html + +**Where they are exposed in this project:** +- Python API: `python/datafusion/functions.py` (aggregate functions are mixed in with scalar functions) +- Rust bindings: `crates/core/src/functions.rs` + +**Evaluated and not requiring separate Python exposure:** +- `count_distinct` — covered by `count(expr, distinct=True)`. Both forms call + `count_udaf` with `distinct: bool = true` and produce the same logical plan. +- `sum_distinct` — covered by `sum(expr, distinct=True)`. +- `avg_distinct` — covered by `avg(expr, distinct=True)`. + +**How to check:** +1. Fetch the upstream aggregate function documentation page +2. Compare against aggregate functions in `python/datafusion/functions.py` (check `__all__` list and function definitions) +3. A function is covered if it exists in the Python API, even if it aliases another function's Rust binding +4. Check against the "evaluated and not requiring exposure" list before flagging as a gap +5. Report only functions missing from the Python API + +### 3. Window Functions + +**Upstream source of truth:** +- Rust docs: https://docs.rs/datafusion/latest/datafusion/functions_window/index.html +- User docs: https://datafusion.apache.org/user-guide/sql/window_functions.html + +**Where they are exposed in this project:** +- Python API: `python/datafusion/functions.py` (window functions like `rank`, `dense_rank`, `lag`, `lead`, etc.) +- Rust bindings: `crates/core/src/functions.rs` + +**How to check:** +1. Fetch the upstream window function documentation page +2. Compare against window functions in `python/datafusion/functions.py` (check `__all__` list and function definitions) +3. A function is covered if it exists in the Python API, even if it aliases another function's Rust binding +4. Report only functions missing from the Python API + +### 4. Table Functions + +**Upstream source of truth:** +- Rust docs: https://docs.rs/datafusion/latest/datafusion/functions_table/index.html +- User docs: https://datafusion.apache.org/user-guide/sql/table_functions.html (if available) + +**Where they are exposed in this project:** +- Python API: `python/datafusion/functions.py` and `python/datafusion/user_defined.py` (TableFunction/udtf) +- Rust bindings: `crates/core/src/functions.rs` and `crates/core/src/udtf.rs` + +**How to check:** +1. Fetch the upstream table function documentation +2. Compare against what's available in the Python API +3. A function is covered if it exists in the Python API, even if it aliases another function's Rust binding +4. Report only functions missing from the Python API + +### 5. DataFrame Operations + +**Upstream source of truth:** +- Rust docs: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html + +**Where they are exposed in this project:** +- Python API: `python/datafusion/dataframe.py` — the `DataFrame` class +- Rust bindings: `crates/core/src/dataframe.rs` — `PyDataFrame` with `#[pymethods]` + +**Evaluated and not requiring separate Python exposure:** +- `show_limit` — already covered by `DataFrame.show()`, which provides the same functionality with a simpler API +- `with_param_values` — already covered by the `param_values` argument on `SessionContext.sql()`, which accomplishes the same thing more robustly +- `union_by_name_distinct` — already covered by `DataFrame.union_by_name(distinct=True)`, which provides a more Pythonic API + +**How to check:** +1. Fetch the upstream DataFrame documentation page listing all methods +2. Compare against methods in `python/datafusion/dataframe.py` — this is the source of truth for coverage +3. The Rust bindings (`crates/core/src/dataframe.rs`) may be consulted for context, but a method is covered if it exists in the Python API +4. Check against the "evaluated and not requiring exposure" list before flagging as a gap +5. Report only methods missing from the Python API + +### 6. SessionContext Methods + +**Upstream source of truth:** +- Rust docs: https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SessionContext.html + +**Where they are exposed in this project:** +- Python API: `python/datafusion/context.py` — the `SessionContext` class +- Rust bindings: `crates/core/src/context.rs` — `PySessionContext` with `#[pymethods]` + +**How to check:** +1. Fetch the upstream SessionContext documentation page listing all methods +2. Compare against methods in `python/datafusion/context.py` — this is the source of truth for coverage +3. The Rust bindings (`crates/core/src/context.rs`) may be consulted for context, but a method is covered if it exists in the Python API +4. Report only methods missing from the Python API + +### 7. FFI Types (datafusion-ffi) + +**Upstream source of truth:** +- Crate source: https://github.com/apache/datafusion/tree/main/datafusion/ffi/src +- Rust docs: https://docs.rs/datafusion-ffi/latest/datafusion_ffi/ + +**Where they are exposed in this project:** +- Rust bindings: various files under `crates/core/src/` and `crates/util/src/` +- FFI example: `examples/datafusion-ffi-example/src/` +- Dependency declared in root `Cargo.toml` and `crates/core/Cargo.toml` + +**Discovering currently supported FFI types:** +Grep for `use datafusion_ffi::` in `crates/core/src/` and `crates/util/src/` to find all FFI types currently imported and used. + +**Evaluated and not requiring direct Python exposure:** +These upstream FFI types have been reviewed and do not need to be independently exposed to end users: +- `FFI_ExecutionPlan` — already used indirectly through table providers; no need for direct exposure +- `FFI_PhysicalExpr` / `FFI_PhysicalSortExpr` — internal physical planning types not expected to be needed by end users +- `FFI_RecordBatchStream` — one level deeper than FFI_ExecutionPlan, used internally when execution plans stream results +- `FFI_SessionRef` / `ForeignSession` — session sharing across FFI; Python manages sessions natively via SessionContext +- `FFI_SessionConfig` — Python can configure sessions natively without FFI +- `FFI_ConfigOptions` / `FFI_TableOptions` — internal configuration plumbing +- `FFI_PlanProperties` / `FFI_Boundedness` / `FFI_EmissionType` — read from existing plans, not user-facing +- `FFI_Partitioning` — supporting type for physical planning +- Supporting/utility types (`FFI_Option`, `FFI_Result`, `WrappedSchema`, `WrappedArray`, `FFI_ColumnarValue`, `FFI_Volatility`, `FFI_InsertOp`, `FFI_AccumulatorArgs`, `FFI_Accumulator`, `FFI_GroupsAccumulator`, `FFI_EmitTo`, `FFI_AggregateOrderSensitivity`, `FFI_PartitionEvaluator`, `FFI_PartitionEvaluatorArgs`, `FFI_Range`, `FFI_SortOptions`, `FFI_Distribution`, `FFI_ExprProperties`, `FFI_SortProperties`, `FFI_Interval`, `FFI_TableProviderFilterPushDown`, `FFI_TableType`) — used as building blocks within the types above, not independently exposed + +**How to check:** +1. Discover currently supported types by grepping for `use datafusion_ffi::` in `crates/core/src/` and `crates/util/src/`, then compare against the upstream `datafusion-ffi` crate's `lib.rs` exports +2. If new FFI types appear upstream, evaluate whether they represent a user-facing capability +3. Check against the "evaluated and not requiring exposure" list before flagging as a gap +4. Report any genuinely new types that enable user-facing functionality +5. For each currently supported FFI type, verify the full pipeline is present using the checklist from "Adding a New FFI Type": + - Rust PyO3 wrapper with `from_pycapsule()` method + - Python Protocol type (e.g., `ScalarUDFExportable`) for FFI objects + - Python wrapper class with full type hints on all public methods + - ABC base class (if the type can be user-implemented) + - Registered in Rust `init_module()` and Python `__init__.py` + - FFI example in `examples/datafusion-ffi-example/` + - Type appears in union type hints where accepted + +### 8. Spark-Compatible Functions (`datafusion-spark` crate) + +**Upstream source of truth:** +- Crate source: https://github.com/apache/datafusion/tree/main/datafusion/spark/src +- Rust docs: https://docs.rs/datafusion-spark/latest/datafusion_spark/ + +**Where they are exposed in this project:** +- Python API: `python/datafusion/functions/spark.py` — each function wraps + a call to `datafusion._internal.functions.spark`; the public surface is + the module's `__all__` list. +- Rust bindings: `crates/core/src/spark_functions.rs` — `#[pyfunction]` + definitions registered via `init_module()` and re-exported under + `datafusion._internal.functions.spark`. + +**Coverage policy:** The spark namespace mirrors +`pyspark.sql.functions` parameter names and shapes exactly so pyspark +callers can paste code unchanged. Extras over pyspark are permitted as +long as positional pyspark calls still work — for example, the spark +`avg` / `try_sum` / `collect_list` / `collect_set` retain the +`distinct`/`filter`/`order_by`/`null_treatment` kwargs from the main +namespace while pyspark's single-positional form continues to work. + +**How to check:** +1. Fetch the upstream `datafusion-spark` function list from the crate + source under `datafusion/spark/src/function/` (each subdirectory is a + category: `string/`, `math/`, `datetime/`, etc.). The crate's + `function.rs` collects all `ScalarUDF` factories. +2. Cross-reference against `pyspark.sql.functions` for the public-facing + shape — pyspark is the contract this namespace is matching. +3. Compare against the functions listed in + `python/datafusion/functions/spark.py`'s `__all__`. A function is + covered if it exists in the Python `spark` namespace, even if it + aliases another function's Rust binding. +4. Report functions that are missing from the Python spark namespace. + +### 9. `__all__` Hygiene (functions.py and functions/spark.py) + +Independent of upstream parity, also flag public `def` symbols in +`python/datafusion/functions.py` **and** `python/datafusion/functions/spark.py` +that are missing from that file's `__all__`. These are functions a user +can call but that do not show up in +`from datafusion.functions import *`, in tab-completion against the +namespace, or in generated API docs — typically an oversight rather than +an intentional omission. + +**How to check:** +1. Grep for `^def ([a-z_][a-z0-9_]*)\(` in each file to enumerate every + public function definition. +2. Read the `__all__` list at the top of the same file. +3. Report any function in (1) that is not in (2). Skip private helpers + (names starting with `_`). + +A historical example: `instr` and `position` shipped as public `def`s but +were absent from `__all__` until the gap was caught here. + +For each finding, propose adding the name to `__all__` in alphabetical +position with the existing entries. + +## Checking for Existing GitHub Issues + +After identifying missing APIs, search the open issues at https://github.com/apache/datafusion-python/issues for each gap to see if an issue already exists requesting that API be exposed. Search using the function or method name as the query. + +- If an existing issue is found, include a link to it in the report. Do NOT create a new issue. +- If no existing issue is found, note that no issue exists yet. If the user asks to create issues for missing APIs, each issue should specify that Python test coverage is required as part of the implementation. + +## Output Format + +For each area checked, produce a report like: + +``` +## [Area Name] Coverage Report + +### Currently Exposed (X functions/methods) +- list of what's already available + +### Missing from Upstream (Y functions/methods) +- function_name — brief description of what it does (existing issue: #123) +- function_name — brief description of what it does (no existing issue) + +### Notes +- Any relevant observations about partial implementations, naming differences, etc. +``` + +## Implementation Pattern + +If the user asks you to implement missing features, follow these patterns: + +### Adding a New Function (Scalar/Aggregate/Window) + +**Step 1: Rust binding** in `crates/core/src/functions.rs`: +```rust +#[pyfunction] +#[pyo3(signature = (arg1, arg2))] +fn new_function_name(arg1: PyExpr, arg2: PyExpr) -> PyResult { + Ok(datafusion::functions::module::expr_fn::new_function_name(arg1.expr, arg2.expr).into()) +} +``` +Then register in `init_module()`: +```rust +m.add_wrapped(wrap_pyfunction!(new_function_name))?; +``` + +**Step 2: Python wrapper** in `python/datafusion/functions.py`: +```python +def new_function_name(arg1: Expr, arg2: Expr) -> Expr: + """Description of what the function does. + + Args: + arg1: Description of first argument. + arg2: Description of second argument. + + Returns: + Description of return value. + """ + return Expr(f.new_function_name(arg1.expr, arg2.expr)) +``` +Add to `__all__` list. + +### Adding a New DataFrame Method + +**Step 1: Rust binding** in `crates/core/src/dataframe.rs`: +```rust +#[pymethods] +impl PyDataFrame { + fn new_method(&self, py: Python, param: PyExpr) -> PyDataFusionResult { + let df = self.df.as_ref().clone().new_method(param.into())?; + Ok(Self::new(df)) + } +} +``` + +**Step 2: Python wrapper** in `python/datafusion/dataframe.py`: +```python +def new_method(self, param: Expr) -> DataFrame: + """Description of the method.""" + return DataFrame(self.df.new_method(param.expr)) +``` + +### Adding a New SessionContext Method + +**Step 1: Rust binding** in `crates/core/src/context.rs`: +```rust +#[pymethods] +impl PySessionContext { + pub fn new_method(&self, py: Python, param: String) -> PyDataFusionResult { + let df = wait_for_future(py, self.ctx.new_method(¶m))?; + Ok(PyDataFrame::new(df)) + } +} +``` + +**Step 2: Python wrapper** in `python/datafusion/context.py`: +```python +def new_method(self, param: str) -> DataFrame: + """Description of the method.""" + return DataFrame(self.ctx.new_method(param)) +``` + +### Adding a New FFI Type + +FFI types require a full pipeline from C struct through to a typed Python wrapper. Each layer must be present. + +**Step 1: Rust PyO3 wrapper class** in a new or existing file under `crates/core/src/`: +```rust +use datafusion_ffi::new_type::FFI_NewType; + +#[pyclass(from_py_object, frozen, name = "RawNewType", module = "datafusion.module_name", subclass)] +pub struct PyNewType { + pub inner: Arc, +} + +#[pymethods] +impl PyNewType { + #[staticmethod] + fn from_pycapsule(obj: &Bound<'_, PyAny>) -> PyDataFusionResult { + let capsule = obj + .getattr("__datafusion_new_type__")? + .call0()? + .downcast::()?; + let ffi_ptr = unsafe { capsule.reference::() }; + let provider: Arc = ffi_ptr.into(); + Ok(Self { inner: provider }) + } + + fn some_method(&self) -> PyResult<...> { + // wrap inner trait method + } +} +``` +Register in the appropriate `init_module()`: +```rust +m.add_class::()?; +``` + +**Step 2: Python Protocol type** in the appropriate Python module (e.g., `python/datafusion/catalog.py`): +```python +class NewTypeExportable(Protocol): + """Type hint for objects providing a __datafusion_new_type__ PyCapsule.""" + + def __datafusion_new_type__(self) -> object: ... +``` + +**Step 3: Python wrapper class** in the same module: +```python +class NewType: + """Description of the type. + + This class wraps a DataFusion NewType, which can be created from a native + Python implementation or imported from an FFI-compatible library. + """ + + def __init__( + self, + new_type: df_internal.module_name.RawNewType | NewTypeExportable, + ) -> None: + if isinstance(new_type, df_internal.module_name.RawNewType): + self._raw = new_type + else: + self._raw = df_internal.module_name.RawNewType.from_pycapsule(new_type) + + def some_method(self) -> ReturnType: + """Description of the method.""" + return self._raw.some_method() +``` + +**Step 4: ABC base class** (if users should be able to subclass and provide custom implementations in Python): +```python +from abc import ABC, abstractmethod + +class NewTypeProvider(ABC): + """Abstract base class for implementing a custom NewType in Python.""" + + @abstractmethod + def some_method(self) -> ReturnType: + """Description of the method.""" + ... +``` + +**Step 5: Module exports** — add to the appropriate `__init__.py`: +- Add the wrapper class (`NewType`) to `python/datafusion/__init__.py` +- Add the ABC (`NewTypeProvider`) if applicable +- Add the Protocol type (`NewTypeExportable`) if it should be public + +**Step 6: FFI example** — add an example implementation under `examples/datafusion-ffi-example/src/`: +```rust +// examples/datafusion-ffi-example/src/new_type.rs +use datafusion_ffi::new_type::FFI_NewType; +// ... example showing how an external Rust library exposes this type via PyCapsule +``` + +**Checklist for each FFI type:** +- [ ] Rust PyO3 wrapper with `from_pycapsule()` method +- [ ] Python Protocol type (e.g., `NewTypeExportable`) for FFI objects +- [ ] Python wrapper class with full type hints on all public methods +- [ ] ABC base class (if the type can be user-implemented) +- [ ] Registered in Rust `init_module()` and Python `__init__.py` +- [ ] FFI example in `examples/datafusion-ffi-example/` +- [ ] Type appears in union type hints where accepted (e.g., `Table | TableProviderExportable`) + +## Important Notes + +- The upstream DataFusion version used by this project is specified in `crates/core/Cargo.toml` — check the `datafusion` dependency version to ensure you're comparing against the right upstream version. +- Some upstream features may intentionally not be exposed (e.g., internal-only APIs). Use judgment about what's user-facing. +- When fetching upstream docs, prefer the published docs.rs documentation as it matches the crate version. +- Function aliases (e.g., `array_append` / `list_append`) should both be exposed if upstream supports them. +- Check the `__all__` list in `functions.py` to see what's publicly exported vs just defined. diff --git a/.ai/skills/make-pythonic/SKILL.md b/.ai/skills/make-pythonic/SKILL.md new file mode 100644 index 000000000..7d490ec03 --- /dev/null +++ b/.ai/skills/make-pythonic/SKILL.md @@ -0,0 +1,465 @@ + + +--- +name: make-pythonic +description: Audit and improve datafusion-python functions to accept native Python types (int, float, str, bool) instead of requiring explicit lit() or col() wrapping. Analyzes function signatures, checks upstream Rust implementations for type constraints, and applies the appropriate coercion pattern. +argument-hint: [scope] (e.g., "string functions", "datetime functions", "array functions", "math functions", "all", or a specific function name like "split_part") +--- + +# Make Python API Functions More Pythonic + +You are improving the datafusion-python API to feel more natural to Python users. The goal is to allow functions to accept native Python types (int, float, str, bool, etc.) for arguments that are contextually always or typically literal values, instead of requiring users to manually wrap them in `lit()`. + +**Core principle:** A Python user should be able to write `split_part(col("a"), ",", 2)` instead of `split_part(col("a"), lit(","), lit(2))` when the arguments are contextually obvious literals. + +## Scope: `functions` vs `functions.spark` + +Both `python/datafusion/functions/__init__.py` and +`python/datafusion/functions/spark.py` are in scope. We want both to feel +pythonic — accept native Python types where the argument is contextually +a literal — but `functions.spark` carries an additional constraint: +**every signature must remain compatible with `pyspark.sql.functions`**. + +Compatibility rules for the spark namespace: + +- **Parameter names must match pyspark exactly.** Pyspark callers pass by + keyword (`spark.shiftleft(col=..., numBits=...)`), so renames break + them. Do NOT rename a parameter just because it would be more pythonic + in the main namespace. +- **Positional order must match pyspark exactly.** Reordering breaks + positional pyspark calls. +- **Type unions may widen the input set, never narrow it.** Pyspark + accepts `Column` or `str` (column name) for most args; we accept + `Expr` already, and widening to `Expr | int` / `Expr | str` for + literal-friendly arguments is on-brand because the int/str case is + exactly what a pyspark caller would also try. Just verify the widened + set is a superset of what pyspark accepts for that arg. +- **Extra keyword arguments are allowed** as long as they default to + `None` and pyspark's positional/keyword form still works (e.g. the + spark `avg`/`try_sum`/`collect_list`/`collect_set` retain DataFusion's + `distinct`/`filter`/`order_by`/`null_treatment` kwargs). + +Practical effect: in `functions.spark`, apply Categories A and (where +pyspark exposes the same arg as a non-`Expr`) B normally, but cross-check +each proposed signature against `pyspark.sql.functions` before landing +it. When pyspark's own type hint is `Column | str` for a "column name" +arg, prefer leaving the spark wrapper at `Expr` — Category C +("`Expr | str` meaning column name") is unusual in `functions.py` and +should remain so in `functions.spark`. + +## How to Identify Candidates + +The user may specify a scope via `$ARGUMENTS`. If no scope is given or "all" is specified, audit all functions in `python/datafusion/functions/__init__.py` **and** `python/datafusion/functions/spark.py`. When updating a spark-namespace function, apply the compatibility rules from "Scope" above on top of the standard analysis. + +For each function, determine if any parameter can accept native Python types by evaluating **two complementary signals**: + +### Signal 1: Contextual Understanding + +Some arguments are contextually always or almost always literal values based on what the function does: + +| Context | Typical Arguments | Examples | +|---------|------------------|----------| +| **String position/count** | Character counts, indices, repetition counts | `left(str, n)`, `right(str, n)`, `repeat(str, n)`, `lpad(str, count, ...)` | +| **Delimiters/separators** | Fixed separator characters | `split_part(str, delim, idx)`, `concat_ws(sep, ...)` | +| **Search/replace patterns** | Literal search strings, replacements | `replace(str, from, to)`, `regexp_replace(str, pattern, replacement, flags)` | +| **Date/time parts** | Part names from a fixed set | `date_part(part, date)`, `date_trunc(part, date)` | +| **Rounding precision** | Decimal place counts | `round(val, places)`, `trunc(val, places)` | +| **Fill characters** | Padding characters | `lpad(str, count, fill)`, `rpad(str, count, fill)` | + +### Signal 2: Upstream Rust Implementation + +Check the Rust binding in `crates/core/src/functions.rs` and the upstream DataFusion function implementation to determine type constraints. The upstream source is cached locally at: + +``` +~/.cargo/registry/src/index.crates.io-*/datafusion-functions-/src/ +``` + +Check the DataFusion version in `crates/core/Cargo.toml` to find the right directory. Key subdirectories: `string/`, `datetime/`, `math/`, `regex/`. + +For **aggregate functions**, the upstream source is in a separate crate: + +``` +~/.cargo/registry/src/index.crates.io-*/datafusion-functions-aggregate-/src/ +``` + +There are five concrete techniques to check, in order of signal strength: + +#### Technique 1: Check `invoke_with_args()` for literal-only enforcement (strongest signal) + +Some functions pattern-match on `ColumnarValue::Scalar` in their `invoke_with_args()` method and **return an error** if the argument is a column/array. This means the argument **must** be a literal — passing a column expression will fail at runtime. + +Example from `date_trunc.rs`: +```rust +let granularity_str = if let ColumnarValue::Scalar(ScalarValue::Utf8(Some(v))) = granularity { + v.to_lowercase() +} else { + return exec_err!("Granularity of `date_trunc` must be non-null scalar Utf8"); +}; +``` + +**If you find this pattern:** The argument is **Category B** — accept only the corresponding native Python type (e.g., `str`), not `Expr`. The function will error at runtime with a column expression anyway. + +#### Technique 1a: Check `accumulator()` for literal-only enforcement (aggregate functions) + +Technique 1 applies to scalar UDFs. Aggregate functions do not have `invoke_with_args()` — instead, they enforce literal-only arguments in their `accumulator()` (or `create_accumulator()`) method, which runs at planning time before any data is processed. + +Look for these patterns inside `accumulator()`: + +- `get_scalar_value(expr)` — evaluates the expression against an empty batch and errors if it's not a scalar +- `validate_percentile_expr(expr)` — specific helper used by percentile functions +- `downcast_ref::()` — checks that the physical expression is a literal constant + +Example from `approx_percentile_cont.rs`: +```rust +fn accumulator(&self, args: AccumulatorArgs) -> Result { + let percentile = + validate_percentile_expr(&args.exprs[1], "APPROX_PERCENTILE_CONT")?; + // ... +} +``` + +Where `validate_percentile_expr` calls `get_scalar_value` and errors with `"must be a literal"`. + +Example from `string_agg.rs`: +```rust +fn accumulator(&self, acc_args: AccumulatorArgs) -> Result> { + let Some(lit) = acc_args.exprs[1].as_any().downcast_ref::() else { + return not_impl_err!( + "The second argument of the string_agg function must be a string literal" + ); + }; + // ... +} +``` + +**If you find this pattern:** The argument is **Category B** — accept only the corresponding native Python type, not `Expr`. The function will error at planning time with a non-literal expression. + +To discover which aggregate functions have literal-only arguments, search the upstream aggregate crate for `get_scalar_value`, `validate_percentile_expr`, and `downcast_ref::()` inside `accumulator()` methods. For example, you should expect to find `approx_percentile_cont` (percentile) and `string_agg` (delimiter) among the results. + +#### Technique 1b: Check `partition_evaluator()` for literal-only enforcement (window functions) + +Window functions do not have `invoke_with_args()` or `accumulator()`. Instead, they enforce literal-only arguments in their `partition_evaluator()` method, which constructs the evaluator that processes each partition. + +The upstream source is in a separate crate: + +``` +~/.cargo/registry/src/index.crates.io-*/datafusion-functions-window-/src/ +``` + +Look for `get_scalar_value_from_args()` calls inside `partition_evaluator()`. This helper (defined in the window crate's `utils.rs`) calls `downcast_ref::()` and errors with `"There is only support Literal types for field at idx: {index} in Window Function"`. + +Example from `ntile.rs`: +```rust +fn partition_evaluator( + &self, + partition_evaluator_args: PartitionEvaluatorArgs, +) -> Result> { + let scalar_n = + get_scalar_value_from_args(partition_evaluator_args.input_exprs(), 0)? + .ok_or_else(|| { + exec_datafusion_err!("NTILE requires a positive integer") + })?; + // ... +} +``` + +**If you find this pattern:** The argument is **Category B** — accept only the corresponding native Python type, not `Expr`. The function will error at planning time with a non-literal expression. + +To discover which window functions have literal-only arguments, search the upstream window crate for `get_scalar_value_from_args` inside `partition_evaluator()` methods. For example, you should expect to find `ntile` (n) and `lead`/`lag` (offset, default_value) among the results. + +#### Technique 2: Check the `Signature` for data type constraints + +Each function defines a `Signature::coercible(...)` that specifies what data types each argument accepts, using `Coercion` entries. This tells you the expected **data type** even if it doesn't enforce literal-only. + +Example from `repeat.rs`: +```rust +signature: Signature::coercible( + vec![ + Coercion::new_exact(TypeSignatureClass::Native(logical_string())), + Coercion::new_implicit( + TypeSignatureClass::Native(logical_int64()), + vec![TypeSignatureClass::Integer], + NativeType::Int64, + ), + ], + Volatility::Immutable, +), +``` + +This tells you arg 2 (`n`) must be an integer type coerced to Int64. Use this to choose the correct Python type (e.g., `int` not `str` or `float`). + +Common mappings: +| Rust Type Constraint | Python Type | +|---------------------|-------------| +| `logical_int64()` / `TypeSignatureClass::Integer` | `int` | +| `logical_float64()` / `TypeSignatureClass::Numeric` | `int \| float` | +| `logical_string()` / `TypeSignatureClass::String` | `str` | +| `LogicalType::Boolean` | `bool` | + +**Important:** In Python's type system (PEP 484), `float` already accepts `int` values, so `int | float` is redundant and will fail the `ruff` linter (rule PYI041). Use `float` alone when the Rust side accepts a float/numeric type — Python users can still pass integer literals like `log(10, col("a"))` or `power(col("a"), 3)` without issue. Only use `int` when the Rust side strictly requires an integer (e.g., `logical_int64()`). + +#### Technique 3: Check `return_field_from_args()` for `scalar_arguments` usage + +Functions that inspect literal values at query planning time use `args.scalar_arguments.get(n)` in their `return_field_from_args()` method. This indicates the argument is **expected to be a literal** for optimal behavior (e.g., to determine output type precision), but may still work as a column. + +Example from `round.rs`: +```rust +let decimal_places: Option = match args.scalar_arguments.get(1) { + None => Some(0), + Some(None) => None, // argument is not a literal (column) + Some(Some(scalar)) if scalar.is_null() => Some(0), + Some(Some(scalar)) => Some(decimal_places_from_scalar(scalar)?), +}; +``` + +**If you find this pattern:** The argument is **Category A** — accept native types AND `Expr`. It works as a column but is primarily used as a literal. + +#### Decision flow + +``` +What kind of function is this? + Scalar UDF: + Is argument rejected at runtime if not a literal? + (check invoke_with_args for ColumnarValue::Scalar-only match + exec_err!) + → YES: Category B — accept only native type, no Expr + → NO: continue below + Aggregate: + Is argument rejected at planning time if not a literal? + (check accumulator() for get_scalar_value / validate_percentile_expr / + downcast_ref::() + error) + → YES: Category B — accept only native type, no Expr + → NO: continue below + Window: + Is argument rejected at planning time if not a literal? + (check partition_evaluator() for get_scalar_value_from_args / + downcast_ref::() + error) + → YES: Category B — accept only native type, no Expr + → NO: continue below + +Does the Signature constrain it to a specific data type? + → YES: Category A — accept Expr | + → NO: Leave as Expr only +``` + +## Coercion Categories + +When making a function more pythonic, apply the correct coercion pattern based on **what the argument represents**: + +### Category A: Arguments That Should Accept Native Types AND Expr + +These are arguments that are *typically* literals but *could* be column references in advanced use cases. For these, accept a union type and coerce native types to `Expr.literal()`. + +**Type hint pattern:** `Expr | int`, `Expr | str`, `Expr | int | str`, etc. + +**When to use:** When the argument could plausibly come from a column in some use case (e.g., the repeat count might come from a column in a data-driven scenario). + +```python +def repeat(string: Expr, n: Expr | int) -> Expr: + """Repeats the ``string`` to ``n`` times. + + Examples: + >>> ctx = dfn.SessionContext() + >>> df = ctx.from_pydict({"a": ["ha"]}) + >>> result = df.select( + ... dfn.functions.repeat(dfn.col("a"), 3).alias("r")) + >>> result.collect_column("r")[0].as_py() + 'hahaha' + """ + if not isinstance(n, Expr): + n = Expr.literal(n) + return Expr(f.repeat(string.expr, n.expr)) +``` + +### Category B: Arguments That Should ONLY Accept Specific Native Types + +These are arguments where an `Expr` never makes sense because the value must be a fixed literal known at query-planning time (not a per-row value). For these, accept only the native type(s) and wrap internally. + +**Type hint pattern:** `str`, `int`, `list[str]`, etc. (no `Expr` in the union) + +**When to use:** When the argument is from a fixed enumeration or is always a compile-time constant, **AND** the parameter was not previously typed as `Expr`: +- Separator in `concat_ws` (already typed as `str` in the Rust binding) +- Index in `array_position` (already typed as `int` in the Rust binding) +- Values that the Rust implementation already accepts as native types + +**Backward compatibility rule:** If a parameter was previously typed as `Expr`, you **must** keep `Expr` in the union even if the Rust side requires a literal. Removing `Expr` would break existing user code like `date_part(lit("year"), col("a"))`. Use **Category A** instead — accept `Expr | str` — and let users who pass column expressions discover the runtime error from the Rust side. Never silently break backward compatibility. + +```python +def concat_ws(separator: str, *args: Expr) -> Expr: + """Concatenates the list ``args`` with the separator. + + ``separator`` is already typed as ``str`` in the Rust binding, so + there is no backward-compatibility concern. + + Examples: + >>> ctx = dfn.SessionContext() + >>> df = ctx.from_pydict({"a": ["hello"], "b": ["world"]}) + >>> result = df.select( + ... dfn.functions.concat_ws("-", dfn.col("a"), dfn.col("b")).alias("c")) + >>> result.collect_column("c")[0].as_py() + 'hello-world' + """ + args = [arg.expr for arg in args] + return Expr(f.concat_ws(separator, args)) +``` + +### Category C: Arguments That Should Accept str as Column Name + +In some contexts a string argument naturally refers to a column name rather than a literal. This is the pattern used by DataFrame methods. + +**Type hint pattern:** `Expr | str` + +**When to use:** Only when the string contextually means a column name (rare in `functions.py`, more common in DataFrame methods). + +```python +# Use _to_raw_expr() from expr.py for this pattern +from datafusion.expr import _to_raw_expr + +def some_function(column: Expr | str) -> Expr: + raw = _to_raw_expr(column) # str -> col(str) + return Expr(f.some_function(raw)) +``` + +**IMPORTANT:** In `functions.py`, string arguments almost never mean column names. Functions operate on expressions, and column references should use `col()`. Category C applies mainly to DataFrame methods and context APIs, not to scalar/aggregate/window functions. Do NOT convert string arguments to column expressions in `functions.py` unless there is a very clear reason to do so. + +## Implementation Steps + +For each function being updated: + +### Step 1: Analyze the Function + +1. Read the current Python function signature in `python/datafusion/functions/__init__.py` +2. Read the Rust binding in `crates/core/src/functions.rs` +3. Optionally check the upstream DataFusion docs for the function +4. Determine which category (A, B, or C) applies to each parameter + +### Step 2: Update the Python Function + +1. **Change the type hints** to accept native types (e.g., `Expr` -> `Expr | int`) +2. **Add coercion logic** at the top of the function body +3. **Update the docstring** examples to use the simpler calling convention +4. **Preserve backward compatibility** — existing code using `Expr` must still work + +### Step 3: Update Alias Type Hints + +After updating a primary function, find all alias functions that delegate to it (e.g., `instr` and `position` delegate to `strpos`). Update each alias's **parameter type hints** to match the primary function's new signature. Do not add coercion logic to aliases — the primary function handles that. + +### Step 4: Update Docstring Examples (primary functions only) + +Per the project's CLAUDE.md rules: +- Every function must have doctest-style examples +- Optional parameters need examples both without and with the optional args, using keyword argument syntax +- Reuse the same input data across examples where possible + +**Update examples to demonstrate the pythonic calling convention:** + +```python +# BEFORE (old style - still works but verbose) +dfn.functions.left(dfn.col("a"), dfn.lit(3)) + +# AFTER (new style - shown in examples) +dfn.functions.left(dfn.col("a"), 3) +``` + +### Step 5: Run Tests + +After making changes, run the doctests to verify: +```bash +python -m pytest --doctest-modules python/datafusion/functions/__init__.py -v +``` + +## Coercion Helper Pattern + +Use the coercion helpers from `datafusion.expr` to convert native Python values to `Expr`. These are the complement of `ensure_expr()` — where `ensure_expr` *rejects* non-`Expr` values, the coercion helpers *wrap* them via `Expr.literal()`. + +**For required parameters** use `coerce_to_expr`: + +```python +from datafusion.expr import coerce_to_expr + +def left(string: Expr, n: Expr | int) -> Expr: + n = coerce_to_expr(n) + return Expr(f.left(string.expr, n.expr)) +``` + +**For optional nullable parameters** use `coerce_to_expr_or_none`: + +```python +from datafusion.expr import coerce_to_expr, coerce_to_expr_or_none + +def regexp_count( + string: Expr, + pattern: Expr | str, + start: Expr | int | None = None, + flags: Expr | str | None = None, +) -> Expr: + pattern = coerce_to_expr(pattern) + start = coerce_to_expr_or_none(start) + flags = coerce_to_expr_or_none(flags) + return Expr( + f.regexp_count( + string.expr, + pattern.expr, + start.expr if start is not None else None, + flags.expr if flags is not None else None, + ) + ) +``` + +Both helpers are defined in `python/datafusion/expr.py` alongside `ensure_expr`. Import them in `functions.py` via: + +```python +from datafusion.expr import coerce_to_expr, coerce_to_expr_or_none +``` + +## What NOT to Change + +- **Do not change arguments that represent data columns.** If an argument is the primary data being operated on (e.g., the `string` in `left(string, n)` or the `array` in `array_sort(array)`), it should remain `Expr` only. Users should use `col()` for column references. +- **Do not change variadic `*args: Expr` parameters.** These represent multiple expressions and should stay as `Expr`. +- **Do not change arguments where the coercion is ambiguous.** If it is unclear whether a string should be a column name or a literal, leave it as `Expr` and let the user be explicit. +- **Do not add coercion logic to simple aliases.** If a function is just `return other_function(...)`, the primary function handles coercion. However, you **must update the alias's type hints** to match the primary function's signature so that type checkers and documentation accurately reflect what the alias accepts. +- **Do not change the Rust bindings.** All coercion happens in the Python layer. The Rust functions continue to accept `PyExpr`. + +## Priority Order + +When auditing functions, process them in this order: + +1. **Date/time functions** — `date_part`, `date_trunc`, `date_bin` — these have the clearest literal arguments +2. **String functions** — `left`, `right`, `repeat`, `lpad`, `rpad`, `split_part`, `substring`, `replace`, `regexp_replace`, `regexp_match`, `regexp_count` — common and verbose without coercion +3. **Math functions** — `round`, `trunc`, `power` — numeric literal arguments +4. **Array functions** — `array_slice`, `array_position`, `array_remove_n`, `array_replace_n`, `array_resize`, `array_element` — index and count arguments +5. **Other functions** — any remaining functions with literal arguments + +## Output Format + +For each function analyzed, report: + +``` +## [Function Name] + +**Current signature:** `function(arg1: Expr, arg2: Expr) -> Expr` +**Proposed signature:** `function(arg1: Expr, arg2: Expr | int) -> Expr` +**Category:** A (accepts native + Expr) +**Arguments changed:** +- `arg2`: Expr -> Expr | int (always a literal count) +**Rust binding:** Takes PyExpr, wraps to literal internally +**Status:** [Changed / Skipped / Needs Discussion] +``` + +If asked to implement (not just audit), make the changes directly and show a summary of what was updated. diff --git a/.cargo/config.toml b/.cargo/config.toml deleted file mode 100644 index af951327f..000000000 --- a/.cargo/config.toml +++ /dev/null @@ -1,5 +0,0 @@ -[target.x86_64-apple-darwin] -rustflags = ["-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup"] - -[target.aarch64-apple-darwin] -rustflags = ["-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup"] diff --git a/.claude/skills b/.claude/skills new file mode 120000 index 000000000..6838a1160 --- /dev/null +++ b/.claude/skills @@ -0,0 +1 @@ +../.ai/skills \ No newline at end of file diff --git a/.github/actions/build-wheel/action.yml b/.github/actions/build-wheel/action.yml new file mode 100644 index 000000000..25f75d1f8 --- /dev/null +++ b/.github/actions/build-wheel/action.yml @@ -0,0 +1,110 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Composite action that builds a datafusion-python wheel with maturin. +# Centralises the abi3-vs-free-threaded argument logic so platform jobs +# stay short and changes to wheel-build flags happen in one place. + +name: "Build wheel" +description: "Build datafusion-python wheel with maturin (abi3 or free-threaded)" + +inputs: + target: + description: "Rust target triple (e.g. x86_64-unknown-linux-gnu). Required when manylinux is set; ignored for native builds." + required: false + default: "" + python-tag: + description: "abi3 (covers 3.10..3.14 GIL builds) or a free-threaded interpreter such as 3.13t / 3.14t" + required: true + build-mode: + description: "release or debug" + required: true + features: + description: "Comma-separated extra features (in addition to those implied by the python-tag)" + required: false + default: "substrait" + manylinux: + description: "manylinux tag for maturin-action (e.g. 2_28). Leave empty to use uv-run maturin natively." + required: false + default: "" + out-dir: + description: "Output directory for built wheels" + required: false + default: "dist" + +outputs: + args: + description: "Computed maturin args (for debugging)" + value: ${{ steps.args.outputs.args }} + +runs: + using: "composite" + steps: + - name: Compute maturin args + id: args + shell: bash + run: | + set -euo pipefail + FEATURES="${{ inputs.features }}" + TAG="${{ inputs.python-tag }}" + if [ "$TAG" = "abi3" ]; then + # Default features include the `abi3` cargo feature. + # One wheel covers Python 3.10..3.14 (GIL builds only). + BUILD_ARGS="--features ${FEATURES}" + else + # Free-threaded build: disable abi3, force mimalloc back in, pin interpreter. + if [ "${RUNNER_OS:-}" = "Windows" ]; then + # Windows free-threaded builds ship as `python.exe` (no `tN` + # suffix). Resolve sys.executable so the path is independent of + # PATH ordering, and assert the interpreter is actually + # free-threaded before we hand the wheel off. + INTERP=$(python -c 'import sys; print(sys.executable)') + python -c "import sysconfig, sys; \ + v = sysconfig.get_config_var('Py_GIL_DISABLED'); \ + sys.exit(0 if v == 1 else f'expected free-threaded interpreter, got Py_GIL_DISABLED={v!r} at {sys.executable}')" + # Backslashes in BUILD_ARGS would be parsed as escapes when the + # output is re-expanded in the next step; use forward slashes + # (maturin/Rust accept them on Windows). + INTERP="${INTERP//\\//}" + else + INTERP="python${TAG}" + fi + BUILD_ARGS="--no-default-features --features mimalloc,${FEATURES} --interpreter ${INTERP}" + fi + if [ "${{ inputs.build-mode }}" = "release" ]; then + BUILD_ARGS="--release --strip ${BUILD_ARGS}" + fi + BUILD_ARGS="${BUILD_ARGS} --out ${{ inputs.out-dir }}" + echo "args=${BUILD_ARGS}" >> "$GITHUB_OUTPUT" + echo "maturin args: ${BUILD_ARGS}" + + - name: Build via maturin-action (manylinux container) + if: inputs.manylinux != '' + uses: PyO3/maturin-action@v1 + with: + target: ${{ inputs.target }} + manylinux: ${{ inputs.manylinux }} + maturin-version: "1.13.3" + args: ${{ steps.args.outputs.args }} + rustup-components: rust-std + + - name: Build via native maturin + if: inputs.manylinux == '' + shell: bash + # Use `uvx` so maturin is available even when `uv sync` was skipped + # (free-threaded matrix entries don't pre-populate the project venv). + run: uvx maturin@1.13.3 build ${{ steps.args.outputs.args }} diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 455a0dc1a..c35801b11 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -36,6 +36,7 @@ on: env: CARGO_TERM_COLOR: always RUST_BACKTRACE: 1 + UV_LOCKED: true jobs: # ============================================ @@ -47,7 +48,7 @@ jobs: - uses: actions/checkout@v6 - name: Setup Rust - uses: dtolnay/rust-toolchain@stable + uses: dtolnay/rust-toolchain@4cda84d5c5c54efe2404f9d843567869ab1699d4 with: toolchain: "nightly" components: rustfmt @@ -68,7 +69,7 @@ jobs: with: python-version: "3.12" - - uses: astral-sh/setup-uv@v6 + - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true @@ -99,7 +100,7 @@ jobs: run: taplo format --check check-crates-patch: - if: inputs.build_mode == 'release' + if: inputs.build_mode == 'release' && startsWith(github.ref, 'refs/tags/') runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 @@ -112,7 +113,7 @@ jobs: steps: - uses: actions/checkout@v6 - - uses: astral-sh/setup-uv@v6 + - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true @@ -124,7 +125,7 @@ jobs: - name: Generate license file run: uv run --no-project python ./dev/create_license.py - - uses: actions/upload-artifact@v6 + - uses: actions/upload-artifact@v7 with: name: python-wheel-license path: LICENSE.txt @@ -134,49 +135,59 @@ jobs: # ============================================ build-manylinux-x86_64: needs: [generate-license, lint-rust, lint-python] - name: ManyLinux x86_64 + name: Linux x86_64 (${{ matrix.python-tag }}) runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-tag: ["abi3", "3.14t"] steps: - uses: actions/checkout@v6 - run: rm LICENSE.txt - name: Download LICENSE.txt - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: name: python-wheel-license path: . - name: Setup Rust - uses: dtolnay/rust-toolchain@stable + uses: dtolnay/rust-toolchain@4cda84d5c5c54efe2404f9d843567869ab1699d4 - name: Cache Cargo uses: Swatinem/rust-cache@v2 with: - key: ${{ inputs.build_mode }} + key: ${{ inputs.build_mode }}-${{ matrix.python-tag }} - - uses: astral-sh/setup-uv@v6 + - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true - - name: Build (release mode) - uses: PyO3/maturin-action@v1 + - name: Add extra swap for release build if: inputs.build_mode == 'release' + run: | + set -euxo pipefail + sudo swapoff -a || true + sudo rm -f /swapfile + sudo fallocate -l 8G /swapfile || sudo dd if=/dev/zero of=/swapfile bs=1M count=8192 + sudo chmod 600 /swapfile + sudo mkswap /swapfile + sudo swapon /swapfile + free -h + swapon --show + + - name: Build wheel + uses: ./.github/actions/build-wheel with: target: x86_64-unknown-linux-gnu + python-tag: ${{ matrix.python-tag }} + build-mode: ${{ inputs.build_mode }} + features: "protoc,substrait" manylinux: "2_28" - args: --release --strip --features protoc,substrait --out dist - rustup-components: rust-std - - - name: Build (debug mode) - uses: PyO3/maturin-action@v1 - if: inputs.build_mode == 'debug' - with: - target: x86_64-unknown-linux-gnu - manylinux: "2_28" - args: --features protoc,substrait --out dist - rustup-components: rust-std + # FFI test wheel only needs to be built once per platform; gate to abi3. - name: Build FFI test library + if: matrix.python-tag == 'abi3' uses: PyO3/maturin-action@v1 with: target: x86_64-unknown-linux-gnu @@ -186,13 +197,14 @@ jobs: rustup-components: rust-std - name: Archive wheels - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 with: - name: dist-manylinux-x86_64 + name: dist-manylinux-x86_64-${{ matrix.python-tag }} path: dist/* - name: Archive FFI test wheel - uses: actions/upload-artifact@v6 + if: matrix.python-tag == 'abi3' + uses: actions/upload-artifact@v7 with: name: test-ffi-manylinux-x86_64 path: examples/datafusion-ffi-example/dist/* @@ -202,53 +214,61 @@ jobs: # ============================================ build-manylinux-aarch64: needs: [generate-license, lint-rust, lint-python] - name: ManyLinux arm64 + name: Linux arm64 (${{ matrix.python-tag }}) runs-on: ubuntu-24.04-arm + strategy: + fail-fast: false + matrix: + python-tag: ["abi3", "3.14t"] steps: - uses: actions/checkout@v6 - run: rm LICENSE.txt - name: Download LICENSE.txt - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: name: python-wheel-license path: . - name: Setup Rust - uses: dtolnay/rust-toolchain@stable + uses: dtolnay/rust-toolchain@4cda84d5c5c54efe2404f9d843567869ab1699d4 - name: Cache Cargo uses: Swatinem/rust-cache@v2 with: - key: ${{ inputs.build_mode }} + key: ${{ inputs.build_mode }}-${{ matrix.python-tag }} - - uses: astral-sh/setup-uv@v6 + - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true - - name: Build (release mode) - uses: PyO3/maturin-action@v1 + - name: Add extra swap for release build if: inputs.build_mode == 'release' + run: | + set -euxo pipefail + sudo swapoff -a || true + sudo rm -f /swapfile + sudo fallocate -l 8G /swapfile || sudo dd if=/dev/zero of=/swapfile bs=1M count=8192 + sudo chmod 600 /swapfile + sudo mkswap /swapfile + sudo swapon /swapfile + free -h + swapon --show + + - name: Build wheel + uses: ./.github/actions/build-wheel with: target: aarch64-unknown-linux-gnu + python-tag: ${{ matrix.python-tag }} + build-mode: ${{ inputs.build_mode }} + features: "protoc,substrait" manylinux: "2_28" - args: --release --strip --features protoc,substrait --out dist - rustup-components: rust-std - - - name: Build (debug mode) - uses: PyO3/maturin-action@v1 - if: inputs.build_mode == 'debug' - with: - target: aarch64-unknown-linux-gnu - manylinux: "2_28" - args: --features protoc,substrait --out dist - rustup-components: rust-std - name: Archive wheels - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 if: inputs.build_mode == 'release' with: - name: dist-manylinux-aarch64 + name: dist-manylinux-aarch64-${{ matrix.python-tag }} path: dist/* # ============================================ @@ -256,21 +276,22 @@ jobs: # ============================================ build-python-mac-win: needs: [generate-license, lint-rust, lint-python] - name: macOS arm64 & Windows + name: ${{ matrix.os == 'macos-latest' && 'macOS arm64' || 'Windows x86_64' }} (${{ matrix.python-tag }}) runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: - python-version: ["3.10"] os: [macos-latest, windows-latest] + python-tag: ["abi3", "3.14t"] steps: - uses: actions/checkout@v6 - - uses: dtolnay/rust-toolchain@stable + - name: Setup Rust + uses: dtolnay/rust-toolchain@4cda84d5c5c54efe2404f9d843567869ab1699d4 - run: rm LICENSE.txt - name: Download LICENSE.txt - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: name: python-wheel-license path: . @@ -278,9 +299,16 @@ jobs: - name: Cache Cargo uses: Swatinem/rust-cache@v2 with: - key: ${{ inputs.build_mode }} + key: ${{ inputs.build_mode }}-${{ matrix.python-tag }} - - uses: astral-sh/setup-uv@v7 + - name: Setup Python (free-threaded) + if: matrix.python-tag != 'abi3' + uses: actions/setup-python@v6 + with: + python-version: ${{ matrix.python-tag }} + freethreaded: true + + - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true @@ -291,22 +319,22 @@ jobs: repo-token: ${{ secrets.GITHUB_TOKEN }} - name: Install dependencies + if: matrix.python-tag == 'abi3' run: uv sync --dev --no-install-package datafusion - # Run clippy BEFORE maturin so we can avoid rebuilding. The features must match - # exactly the features used by maturin. Linux maturin builds need to happen in a - # container so only run this for our mac runner. + # Clippy is interpreter-agnostic; run once per OS (against the abi3 entry) + # so the matrix doesn't pay the cost three times. - name: Run Clippy - if: matrix.os != 'windows-latest' + if: matrix.os != 'windows-latest' && matrix.python-tag == 'abi3' run: cargo clippy --no-deps --all-targets --features substrait -- -D warnings - - name: Build Python package (release mode) - if: inputs.build_mode == 'release' - run: uv run --no-project maturin build --release --strip --features substrait - - - name: Build Python package (debug mode) - if: inputs.build_mode != 'release' - run: uv run --no-project maturin build --features substrait + - name: Build wheel + uses: ./.github/actions/build-wheel + with: + python-tag: ${{ matrix.python-tag }} + build-mode: ${{ inputs.build_mode }} + features: "substrait" + out-dir: "target/wheels" - name: List Windows wheels if: matrix.os == 'windows-latest' @@ -320,10 +348,10 @@ jobs: run: find target/wheels/ - name: Archive wheels - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 if: inputs.build_mode == 'release' with: - name: dist-${{ matrix.os }} + name: dist-${{ matrix.os }}-${{ matrix.python-tag }} path: target/wheels/* # ============================================ @@ -332,19 +360,21 @@ jobs: build-macos-x86_64: if: inputs.build_mode == 'release' needs: [generate-license, lint-rust, lint-python] + name: macOS x86_64 (${{ matrix.python-tag }}) runs-on: macos-15-intel strategy: fail-fast: false matrix: - python-version: ["3.10"] + python-tag: ["abi3", "3.14t"] steps: - uses: actions/checkout@v6 - - uses: dtolnay/rust-toolchain@stable + - name: Setup Rust + uses: dtolnay/rust-toolchain@4cda84d5c5c54efe2404f9d843567869ab1699d4 - run: rm LICENSE.txt - name: Download LICENSE.txt - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: name: python-wheel-license path: . @@ -352,9 +382,16 @@ jobs: - name: Cache Cargo uses: Swatinem/rust-cache@v2 with: - key: ${{ inputs.build_mode }} + key: ${{ inputs.build_mode }}-${{ matrix.python-tag }} - - uses: astral-sh/setup-uv@v7 + - name: Setup Python (free-threaded) + if: matrix.python-tag != 'abi3' + uses: actions/setup-python@v6 + with: + python-version: ${{ matrix.python-tag }} + freethreaded: true + + - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true @@ -365,19 +402,24 @@ jobs: repo-token: ${{ secrets.GITHUB_TOKEN }} - name: Install dependencies + if: matrix.python-tag == 'abi3' run: uv sync --dev --no-install-package datafusion - - name: Build (release mode) - run: | - uv run --no-project maturin build --release --strip --features substrait + - name: Build wheel + uses: ./.github/actions/build-wheel + with: + python-tag: ${{ matrix.python-tag }} + build-mode: ${{ inputs.build_mode }} + features: "substrait" + out-dir: "target/wheels" - name: List Mac wheels run: find target/wheels/ - name: Archive wheels - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7 with: - name: dist-macos-aarch64 + name: dist-macos-aarch64-${{ matrix.python-tag }} path: target/wheels/* # ============================================ @@ -393,7 +435,7 @@ jobs: - uses: actions/checkout@v6 - run: rm LICENSE.txt - name: Download LICENSE.txt - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: name: python-wheel-license path: . @@ -431,7 +473,7 @@ jobs: - build-sdist steps: - name: Merge Build Artifacts - uses: actions/upload-artifact/merge@v6 + uses: actions/upload-artifact/merge@v7 with: name: dist pattern: dist-* @@ -478,15 +520,16 @@ jobs: python-version: "3.10" - name: Install dependencies - uses: astral-sh/setup-uv@v7 + uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true - # Download the Linux wheel built in the previous job + # Download the Linux wheel built in the previous job. + # Docs only need the abi3 wheel — interpreter doesn't matter for sphinx. - name: Download pre-built Linux wheel - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: - name: dist-manylinux-x86_64 + name: dist-manylinux-x86_64-abi3 path: wheels/ # Install from the pre-built wheels @@ -511,9 +554,10 @@ jobs: run: | set -x cd docs - curl -O https://gist.githubusercontent.com/ritchie46/cac6b337ea52281aa23c049250a4ff03/raw/89a957ff3919d90e6ef2d34235e6bf22304f3366/pokemon.csv - curl -O https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-01.parquet - uv run --no-project make html + # build.sh downloads the example data, registers the Jupyter kernel + # myst-nb needs, symlinks the data next to each executed page, and + # runs sphinx. Using it here keeps CI identical to a local build. + uv run --no-project bash ./build.sh - name: Copy & push the generated HTML if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref_type == 'tag') diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 000000000..2d0f166ba --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,54 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +name: "CodeQL" + +on: + push: + branches: [ "main" ] + pull_request: + branches: [ "main" ] + schedule: + - cron: '16 4 * * 1' + +permissions: + contents: read + +jobs: + analyze: + name: Analyze Actions + runs-on: ubuntu-latest + permissions: + contents: read + security-events: write + + steps: + - name: Checkout repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + + - name: Initialize CodeQL + uses: github/codeql-action/init@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4 + with: + languages: actions + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4 + with: + category: "/language:actions" diff --git a/.github/workflows/dev.yml b/.github/workflows/dev.yml index 2c8ecbc5e..841bf205a 100644 --- a/.github/workflows/dev.yml +++ b/.github/workflows/dev.yml @@ -16,7 +16,12 @@ # under the License. name: Dev -on: [push, pull_request] +on: + push: + branches: + - main + - branch-* + pull_request: jobs: diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 692563019..558e751c8 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -15,87 +15,89 @@ # specific language governing permissions and limitations # under the License. -# Reusable workflow for running tests -# This ensures the same tests run for both debug (PRs) and release (main/tags) builds +# Reusable workflow for running tests. +# Single matrix covers both GIL (abi3 wheel) and free-threaded +# (per-interpreter wheels) builds. name: Test on: workflow_call: +env: + UV_LOCKED: true + jobs: test-matrix: runs-on: ubuntu-latest + # Backstop: a hung multiprocessing worker (e.g. during a pickle regression) + # should not block CI longer than this. + timeout-minutes: 30 strategy: fail-fast: false matrix: - python-version: - - "3.10" - - "3.11" - - "3.12" - - "3.13" - - "3.14" - toolchain: - - "stable" - + include: + # GIL builds — all share the same abi3 wheel. + - { python-version: "3.10", wheel-tag: "abi3", freethreaded: false } + - { python-version: "3.11", wheel-tag: "abi3", freethreaded: false } + - { python-version: "3.12", wheel-tag: "abi3", freethreaded: false } + - { python-version: "3.13", wheel-tag: "abi3", freethreaded: false } + - { python-version: "3.14", wheel-tag: "abi3", freethreaded: false } + # Free-threaded builds — one wheel per interpreter. + - { python-version: "3.14t", wheel-tag: "3.14t", freethreaded: true } steps: - uses: actions/checkout@v6 - - name: Verify example datafusion version - run: | - MAIN_VERSION=$(grep -A 1 "name = \"datafusion-common\"" Cargo.lock | grep "version = " | head -1 | sed 's/.*version = "\(.*\)"/\1/') - EXAMPLE_VERSION=$(grep -A 1 "name = \"datafusion-common\"" examples/datafusion-ffi-example/Cargo.lock | grep "version = " | head -1 | sed 's/.*version = "\(.*\)"/\1/') - echo "Main crate datafusion version: $MAIN_VERSION" - echo "FFI example datafusion version: $EXAMPLE_VERSION" - - if [ "$MAIN_VERSION" != "$EXAMPLE_VERSION" ]; then - echo "❌ Error: FFI example datafusion versions don't match!" - exit 1 - fi - - name: Setup Python + id: setup-python uses: actions/setup-python@v6 with: python-version: ${{ matrix.python-version }} + freethreaded: ${{ matrix.freethreaded }} - name: Cache Cargo uses: actions/cache@v5 with: path: ~/.cargo - key: cargo-cache-${{ matrix.toolchain }}-${{ hashFiles('Cargo.lock') }} + key: cargo-cache-stable-${{ hashFiles('Cargo.lock') }} - name: Install dependencies - uses: astral-sh/setup-uv@v7 + uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true - # Download the Linux wheel built in the build workflow - name: Download pre-built Linux wheel - uses: actions/download-artifact@v7 + uses: actions/download-artifact@v8 with: - name: dist-manylinux-x86_64 + name: dist-manylinux-x86_64-${{ matrix.wheel-tag }} path: wheels/ - # Download the FFI test wheel + # FFI test wheel only built once (under the abi3 matrix entry in build.yml). - name: Download pre-built FFI test wheel - uses: actions/download-artifact@v7 + if: matrix.wheel-tag == 'abi3' + uses: actions/download-artifact@v8 with: name: test-ffi-manylinux-x86_64 path: wheels/ - # Install from the pre-built wheels - name: Install from pre-built wheels run: | set -x - uv venv - # Install development dependencies - uv sync --dev --no-install-package datafusion - # Install all pre-built wheels + # Create the venv with the setup-python interpreter, then point + # every uv command explicitly at the venv's own interpreter. + # uv's interpreter discovery skips free-threaded builds unless + # asked by exact path, so plain `uv sync` (even with an activated + # venv or --active) re-picks the system 3.12 and recreates .venv. + # Targeting .venv/bin/python keeps sync and pip install in the + # same 3.14t environment as the cp314t wheel. + uv venv --python "${{ steps.setup-python.outputs.python-path }}" + VENV_PY="$PWD/.venv/bin/python" + uv sync --python "$VENV_PY" --dev --no-install-package datafusion WHEELS=$(find wheels/ -name "*.whl") if [ -n "$WHEELS" ]; then echo "Installing wheels:" echo "$WHEELS" - uv pip install wheels/*.whl + uv pip install --python "$VENV_PY" wheels/*.whl else echo "ERROR: No wheels found!" exit 1 @@ -104,30 +106,32 @@ jobs: - name: Run tests env: RUST_BACKTRACE: 1 + # On free-threaded interpreters, fail loud if any C extension + # re-enables the GIL implicitly. + PYTHON_GIL: ${{ matrix.freethreaded && '0' || '' }} run: | git submodule update --init - uv run --no-project pytest -v --import-mode=importlib + # Use the .venv interpreter directly; uv discovery would skip the + # free-threaded build and re-pick the system 3.12 (see install step). + uv run --python "$PWD/.venv/bin/python" --no-project pytest -v --import-mode=importlib + # FFI + TPC-H examples only need to run once; gate to abi3 entries. - name: FFI unit tests + if: matrix.wheel-tag == 'abi3' run: | cd examples/datafusion-ffi-example uv run --no-project pytest python/tests/_test*.py - - name: Cache the generated dataset - id: cache-tpch-dataset - uses: actions/cache@v5 - with: - path: benchmarks/tpch/data - key: tpch-data-2.18.0 - - - name: Run dbgen to create 1 Gb dataset - if: ${{ steps.cache-tpch-dataset.outputs.cache-hit != 'true' }} + - name: Run tpchgen-cli to create 1 Gb dataset + if: matrix.wheel-tag == 'abi3' run: | - cd benchmarks/tpch - RUN_IN_CI=TRUE ./tpch-gen.sh 1 + mkdir examples/tpch/data + cd examples/tpch/data + uv pip install tpchgen-cli + uv run --no-project tpchgen-cli -s 1 --format=parquet - name: Run TPC-H examples + if: matrix.wheel-tag == 'abi3' run: | cd examples/tpch - uv run --no-project python convert_data_to_parquet.py uv run --no-project pytest _tests.py diff --git a/.github/workflows/verify-release-candidate.yml b/.github/workflows/verify-release-candidate.yml index a10a4faa9..6ecb547b5 100644 --- a/.github/workflows/verify-release-candidate.yml +++ b/.github/workflows/verify-release-candidate.yml @@ -27,7 +27,7 @@ on: required: true type: string rc_number: - description: Release candidate number (e.g., 0) + description: Release candidate number (e.g., 1) required: true type: string @@ -73,6 +73,11 @@ jobs: version: "27.4" repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Set RUSTFLAGS for Windows GNU linker + if: matrix.os == 'windows' + shell: bash + run: echo "RUSTFLAGS=-C link-arg=-Wl,--exclude-libs=ALL" >> "$GITHUB_ENV" + - name: Run release candidate verification shell: bash run: ./dev/release/verify-release-candidate.sh "${{ inputs.version }}" "${{ inputs.rc_number }}" diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 8ae6a4e32..0a212480b 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -17,7 +17,7 @@ repos: - repo: https://github.com/rhysd/actionlint - rev: v1.7.6 + rev: v1.7.12 hooks: - id: actionlint-docker - repo: https://github.com/astral-sh/ruff-pre-commit @@ -53,5 +53,12 @@ repos: additional_dependencies: - tomli + - repo: https://github.com/astral-sh/uv-pre-commit + # uv version. + rev: 0.10.7 + hooks: + # Update the uv lockfile + - id: uv-lock + default_language_version: python: python3 diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..fda08b23c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,92 @@ + + +# Agent Instructions for Contributors + +This file is for agents working **on** the datafusion-python project (developing, +testing, reviewing). If you need to **use** the DataFusion DataFrame API (write +queries, build expressions, understand available functions), see the user-facing +skill at [`SKILL.md`](skills/datafusion_python/SKILL.md). + +## Skills + +This project uses AI agent skills stored in `.ai/skills/`. Each skill is a directory containing a `SKILL.md` file with instructions for performing a specific task. + +Skills follow the [Agent Skills](https://agentskills.io) open standard. Each skill directory contains: + +- `SKILL.md` — The skill definition with YAML frontmatter (name, description, argument-hint) and detailed instructions. +- Additional supporting files as needed. + +To discover what skills are available, list `.ai/skills/` and read each +`SKILL.md`. The frontmatter `name` and `description` fields summarize the +skill's purpose. + +## Pull Requests + +Every pull request must follow the template in +`.github/pull_request_template.md`. The description must include these sections: + +1. **Which issue does this PR close?** — Link the issue with `Closes #NNN`. +2. **Rationale for this change** — Why the change is needed (skip if the issue + already explains it clearly). +3. **What changes are included in this PR?** — Summarize the individual changes. +4. **Are there any user-facing changes?** — Note any changes visible to users + (new APIs, changed behavior, new files shipped in the package, etc.). If + there are breaking changes to public APIs, add the `api change` label. + +## Pre-commit Checks + +Always run pre-commit checks **before** committing. The hooks are defined in +`.pre-commit-config.yaml` and run automatically on `git commit` if pre-commit +is installed as a git hook. To run all hooks manually: + +```bash +pre-commit run --all-files +``` + +Fix any failures before committing. + +## Python Function Docstrings + +Every Python function must include a docstring with usage examples. + +- **Examples are required**: Each function needs at least one doctest-style example + demonstrating basic usage. +- **Optional parameters**: If a function has optional parameters, include separate + examples that show usage both without and with the optional arguments. Pass + optional arguments using their keyword name (e.g., `step=dfn.lit(3)`) so readers + can immediately see which parameter is being demonstrated. +- **Reuse input data**: Use the same input data across examples wherever possible. + The examples should demonstrate how different optional arguments change the output + for the same input, making the effect of each option easy to understand. +- **Alias functions**: Functions that are simple aliases (e.g., `list_sort` aliasing + `array_sort`) only need a one-line description and a `See Also` reference to the + primary function. They do not need their own examples. + +## Aggregate and Window Function Documentation + +When adding or updating an aggregate or window function, ensure the corresponding +site documentation is kept in sync: + +- **Aggregations**: `docs/source/user-guide/common-operations/aggregations.md` — + add new aggregate functions to the "Aggregate Functions" list and include usage + examples if appropriate. +- **Window functions**: `docs/source/user-guide/common-operations/windows.md` — + add new window functions to the "Available Functions" list and include usage + examples if appropriate. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 120000 index 000000000..47dc3e3d8 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/Cargo.lock b/Cargo.lock index 40b1ba7f1..ab222b177 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2,54 +2,6 @@ # It is not intended for manual editing. version = 4 -[[package]] -name = "abi_stable" -version = "0.11.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "69d6512d3eb05ffe5004c59c206de7f99c34951504056ce23fc953842f12c445" -dependencies = [ - "abi_stable_derive", - "abi_stable_shared", - "const_panic", - "core_extensions", - "crossbeam-channel", - "generational-arena", - "libloading", - "lock_api", - "parking_lot", - "paste", - "repr_offset", - "rustc_version", - "serde", - "serde_derive", - "serde_json", -] - -[[package]] -name = "abi_stable_derive" -version = "0.11.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7178468b407a4ee10e881bc7a328a65e739f0863615cca4429d43916b05e898" -dependencies = [ - "abi_stable_shared", - "as_derive_utils", - "core_extensions", - "proc-macro2", - "quote", - "rustc_version", - "syn 1.0.109", - "typed-arena", -] - -[[package]] -name = "abi_stable_shared" -version = "0.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b2b5df7688c123e63f4d4d649cba63f2967ba7f7861b1664fca3f77d3dad2b63" -dependencies = [ - "core_extensions", -] - [[package]] name = "adler2" version = "2.0.1" @@ -87,9 +39,9 @@ checksum = "cc7bb162ec39d46ab1ca8c77bf72e890535becd1751bb45f64c597edb4c8c6b3" [[package]] name = "alloc-stdlib" -version = "0.2.2" +version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94fb8275041c72129eb51b7d0322c29b8387a0386127718b096429201a5d6ece" +checksum = "0e76a019e91224d279006ff972f1e984179a6e9feb050adba6ce8274aef23195" dependencies = [ "alloc-no-stdlib", ] @@ -111,53 +63,24 @@ dependencies = [ [[package]] name = "anyhow" -version = "1.0.101" +version = "1.0.102" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f0e0fee31ef5ed1ba1316088939cea399010ed7731dba877ed44aeb407a75ea" - -[[package]] -name = "apache-avro" -version = "0.21.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "36fa98bc79671c7981272d91a8753a928ff6a1cd8e4f20a44c45bd5d313840bf" -dependencies = [ - "bigdecimal", - "bon", - "bzip2", - "crc32fast", - "digest", - "liblzma", - "log", - "miniz_oxide", - "num-bigint", - "quad-rand", - "rand", - "regex-lite", - "serde", - "serde_bytes", - "serde_json", - "snap", - "strum", - "strum_macros", - "thiserror", - "uuid", - "zstd", -] +checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" [[package]] name = "ar_archive_writer" -version = "0.5.1" +version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7eb93bbb63b9c227414f6eb3a0adfddca591a8ce1e9b60661bb08969b87e340b" +checksum = "4087686b4b0a3427190bae57a1d9a478dbb2d40c5dc1bd6e2b6d797913bdd348" dependencies = [ "object", ] [[package]] name = "arc-swap" -version = "1.8.2" +version = "1.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9f3647c145568cec02c42054e07bdf9a5a698e15b466fb2341bfc393cd24aa5" +checksum = "6a3a1fd6f75306b68087b831f025c712524bcb19aad54e557b1129cfa0a2b207" dependencies = [ "rustversion", ] @@ -176,9 +99,9 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" [[package]] name = "arrow" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "602268ce9f569f282cedb9a9f6bac569b680af47b9b077d515900c03c5d190da" +checksum = "61d285d16bce7d0be61912f7928342b673067b6b7d7ef6cc179258ba7de1fecf" dependencies = [ "arrow-arith", "arrow-array", @@ -198,9 +121,9 @@ dependencies = [ [[package]] name = "arrow-arith" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cd53c6bf277dea91f136ae8e3a5d7041b44b5e489e244e637d00ae302051f56f" +checksum = "757ef1836251e88222542a7da2623bc1c9cb9e20afefa6db2c41e79991cd91d4" dependencies = [ "arrow-array", "arrow-buffer", @@ -212,9 +135,9 @@ dependencies = [ [[package]] name = "arrow-array" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e53796e07a6525edaf7dc28b540d477a934aff14af97967ad1d5550878969b9e" +checksum = "bc9a4a4b2b5ecd0e04df03471661cb61f28bed3c7fd50994715129b01b2edb97" dependencies = [ "ahash", "arrow-buffer", @@ -223,29 +146,54 @@ dependencies = [ "chrono", "chrono-tz", "half", - "hashbrown 0.16.1", + "hashbrown 0.17.1", + "libc", "num-complex", "num-integer", "num-traits", ] +[[package]] +name = "arrow-avro" +version = "59.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9fb45cd6bd2b25c0965793b83200eaca82214273a8030fbbc2d783e4c7c65a61" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-schema", + "bytes", + "bzip2", + "crc", + "flate2", + "indexmap", + "liblzma", + "rand 0.9.4", + "serde", + "serde_json", + "snap", + "strum_macros", + "uuid", + "zstd", +] + [[package]] name = "arrow-buffer" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2c1a85bb2e94ee10b76531d8bc3ce9b7b4c0d508cabfb17d477f63f2617bd20" +checksum = "c12b576ef18c1deb80925a248b25ad84f419198d791b8e293fc6aaa60441fe90" dependencies = [ "bytes", "half", - "num-bigint", + "num-bigint 0.5.1", "num-traits", ] [[package]] name = "arrow-cast" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89fb245db6b0e234ed8e15b644edb8664673fefe630575e94e62cd9d489a8a26" +checksum = "68338a9096a5dc9bc11927c58c43a8526d96bf6abd2012ef6c0c9f505991cc79" dependencies = [ "arrow-array", "arrow-buffer", @@ -254,7 +202,7 @@ dependencies = [ "arrow-schema", "arrow-select", "atoi", - "base64", + "base64 0.23.1", "chrono", "comfy-table", "half", @@ -265,9 +213,9 @@ dependencies = [ [[package]] name = "arrow-csv" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d374882fb465a194462527c0c15a93aa19a554cf690a6b77a26b2a02539937a7" +checksum = "25011b52b346407d497ef0030e12b45e4f2d0cc279efc09c4f3d09106db30e36" dependencies = [ "arrow-array", "arrow-cast", @@ -280,9 +228,9 @@ dependencies = [ [[package]] name = "arrow-data" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "189d210bc4244c715fa3ed9e6e22864673cccb73d5da28c2723fb2e527329b33" +checksum = "723fe4aeed7604e00b9883a465af4ff0a0e6c44c03e41a68c3d1cbc403e0e44d" dependencies = [ "arrow-buffer", "arrow-schema", @@ -293,9 +241,9 @@ dependencies = [ [[package]] name = "arrow-ipc" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7968c2e5210c41f4909b2ef76f6e05e172b99021c2def5edf3cc48fdd39d1d6c" +checksum = "149437b14371f5b9ec60f5ddc751483ae99d7a7072653c0075e5e469156eea7b" dependencies = [ "arrow-array", "arrow-buffer", @@ -309,15 +257,16 @@ dependencies = [ [[package]] name = "arrow-json" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92111dba5bf900f443488e01f00d8c4ddc2f47f5c50039d18120287b580baa22" +checksum = "f18b9123ccfec418a663f821c9a034af339711678c11ffe00d3ec07da5ff9f7e" dependencies = [ "arrow-array", "arrow-buffer", "arrow-cast", - "arrow-data", + "arrow-ord", "arrow-schema", + "arrow-select", "chrono", "half", "indexmap", @@ -333,9 +282,9 @@ dependencies = [ [[package]] name = "arrow-ord" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "211136cb253577ee1a6665f741a13136d4e563f64f5093ffd6fb837af90b9495" +checksum = "e6c08dff0686cf23ca4f562803f191ccbeb726dbae6309cd4b4aaf65e0f2c979" dependencies = [ "arrow-array", "arrow-buffer", @@ -346,9 +295,9 @@ dependencies = [ [[package]] name = "arrow-pyarrow" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "205437da4c0877c756c81bfe847a621d0a740cd00a155109d65510a1a62ebcd9" +checksum = "c196ecc25b3a8dcbc1d842f2619cee653dcfa2fb8b56a291bc0481c3cf5c3821" dependencies = [ "arrow-array", "arrow-data", @@ -358,9 +307,9 @@ dependencies = [ [[package]] name = "arrow-row" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e0f20145f9f5ea3fe383e2ba7a7487bf19be36aa9dbf5dd6a1f92f657179663" +checksum = "bbec439386df71ad570e6758a946111322b9e9dc8db83b5527321f0b4c9119c2" dependencies = [ "arrow-array", "arrow-buffer", @@ -371,9 +320,9 @@ dependencies = [ [[package]] name = "arrow-schema" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1b47e0ca91cc438d2c7879fe95e0bca5329fff28649e30a88c6f760b1faeddcb" +checksum = "e6fed2ca0d1eade57e811cbe73b98ad50cc08a1183e13b2d2aa43a7df593f40e" dependencies = [ "bitflags", "serde_core", @@ -382,9 +331,9 @@ dependencies = [ [[package]] name = "arrow-select" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "750a7d1dda177735f5e82a314485b6915c7cccdbb278262ac44090f4aba4a325" +checksum = "466b19cf75130b891dc1b23a84b343c714c62c64c9c62e365c76aa0ff90a53fb" dependencies = [ "ahash", "arrow-array", @@ -396,9 +345,9 @@ dependencies = [ [[package]] name = "arrow-string" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e1eab1208bc4fe55d768cdc9b9f3d9df5a794cdb3ee2586bf89f9b30dc31ad8c" +checksum = "c838a25bb3691e919e0f617616ac51a4ff8517a952e29ca133cf0c22b2ce65b1" dependencies = [ "arrow-array", "arrow-buffer", @@ -411,23 +360,11 @@ dependencies = [ "regex-syntax", ] -[[package]] -name = "as_derive_utils" -version = "0.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff3c96645900a44cf11941c111bd08a6573b0e2f9f69bc9264b179d8fae753c4" -dependencies = [ - "core_extensions", - "proc-macro2", - "quote", - "syn 1.0.109", -] - [[package]] name = "async-compression" -version = "0.4.40" +version = "0.4.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d67d43201f4d20c78bcda740c142ca52482d81da80681533d33bf3f0596c8e2" +checksum = "e79b3f8a79cccc2898f31920fc69f304859b3bd567490f75ebf51ae1c792a9ac" dependencies = [ "compression-codecs", "compression-core", @@ -440,9 +377,6 @@ name = "async-ffi" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f4de21c0feef7e5a556e51af767c953f0501f7f300ba785cc99c47bdc8081a50" -dependencies = [ - "abi_stable", -] [[package]] name = "async-recursion" @@ -452,7 +386,7 @@ checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -463,7 +397,7 @@ checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -483,9 +417,9 @@ checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" [[package]] name = "autocfg" -version = "1.5.0" +version = "1.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" [[package]] name = "base64" @@ -493,6 +427,12 @@ version = "0.22.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" +[[package]] +name = "base64" +version = "0.23.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5" + [[package]] name = "bigdecimal" version = "0.4.10" @@ -501,17 +441,16 @@ checksum = "4d6867f1565b3aad85681f1015055b087fcfd840d6aeee6eee7f2da317603695" dependencies = [ "autocfg", "libm", - "num-bigint", + "num-bigint 0.4.6", "num-integer", "num-traits", - "serde", ] [[package]] name = "bitflags" -version = "2.11.0" +version = "2.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" [[package]] name = "blake2" @@ -519,14 +458,14 @@ version = "0.10.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe" dependencies = [ - "digest", + "digest 0.10.7", ] [[package]] name = "blake3" -version = "1.8.3" +version = "1.8.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2468ef7d57b3fb7e16b576e8377cdbde2320c60e1491e961d11da40fc4f02a2d" +checksum = "0aa83c34e62843d924f905e0f5c866eb1dd6545fc4d719e803d9ba6030371fce" dependencies = [ "arrayref", "arrayvec", @@ -546,35 +485,19 @@ dependencies = [ ] [[package]] -name = "bon" -version = "3.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2d13a61f2963b88eef9c1be03df65d42f6996dfeac1054870d950fcf66686f83" -dependencies = [ - "bon-macros", - "rustversion", -] - -[[package]] -name = "bon-macros" -version = "3.9.0" +name = "block-buffer" +version = "0.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d314cc62af2b6b0c65780555abb4d02a03dd3b799cd42419044f0c38d99738c0" +checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa" dependencies = [ - "darling", - "ident_case", - "prettyplease", - "proc-macro2", - "quote", - "rustversion", - "syn 2.0.117", + "hybrid-array", ] [[package]] name = "brotli" -version = "8.0.2" +version = "8.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4bd8b9603c7aa97359dbd97ecf258968c95f3adddd6db2f7e7a5bef101c84560" +checksum = "5cc91aac060a7a1e25823bdccbfb6af1875b88f17c6daac97894eed8207166b3" dependencies = [ "alloc-no-stdlib", "alloc-stdlib", @@ -583,9 +506,9 @@ dependencies = [ [[package]] name = "brotli-decompressor" -version = "5.0.0" +version = "5.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "874bb8112abecc98cbd6d81ea4fa7e94fb9449648c93cc89aa40c81c24d7de03" +checksum = "3a32acac15fe1967bc3986b2a6347dffc965602354ea6f450ad07e8bfd253583" dependencies = [ "alloc-no-stdlib", "alloc-stdlib", @@ -593,15 +516,9 @@ dependencies = [ [[package]] name = "bumpalo" -version = "3.20.1" +version = "3.20.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c6f81257d10a0f602a294ae4182251151ff97dbb504ef9afcdda4a64b24d9b4" - -[[package]] -name = "byteorder" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" [[package]] name = "bytes" @@ -620,9 +537,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.56" +version = "1.2.64" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aebf35691d1bfb0ac386a69bac2fde4dd276fb618cf8bf4f5318fe285e821bb2" +checksum = "dad887fd958be91b5098c0248def011f4523ab786cd411be668777e55063501f" dependencies = [ "find-msvc-tools", "jobserver", @@ -642,11 +559,22 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" +[[package]] +name = "chacha20" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601" +dependencies = [ + "cfg-if", + "cpufeatures", + "rand_core 0.10.1", +] + [[package]] name = "chrono" -version = "0.4.44" +version = "0.4.45" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c673075a2e0e5f4a1dde27ce9dee1ea4558c7ffe648f576438a20ca1d2acc4b0" +checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" dependencies = [ "iana-time-zone", "num-traits", @@ -666,9 +594,9 @@ dependencies = [ [[package]] name = "cmake" -version = "0.1.57" +version = "0.1.58" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75443c44cd6b379beb8c5b45d85d0773baf31cce901fe7bb252f4eff3008ef7d" +checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678" dependencies = [ "cc", ] @@ -685,9 +613,9 @@ dependencies = [ [[package]] name = "compression-codecs" -version = "0.4.37" +version = "0.4.38" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eb7b51a7d9c967fc26773061ba86150f19c50c0d65c887cb1fbe295fd16619b7" +checksum = "ce2548391e9c1929c21bf6aa2680af86fe4c1b33e6cea9ac1cfeec0bd11218cf" dependencies = [ "bzip2", "compression-core", @@ -700,9 +628,15 @@ dependencies = [ [[package]] name = "compression-core" -version = "0.4.31" +version = "0.4.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc14f565cf027a105f7a44ccf9e5b424348421a1d8952a8fc9d499d313107789" + +[[package]] +name = "const-oid" +version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75984efb6ed102a0d42db99afb6c1948f0380d1d91808d5529916e6c08b49d8d" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" [[package]] name = "const-random" @@ -724,15 +658,6 @@ dependencies = [ "tiny-keccak", ] -[[package]] -name = "const_panic" -version = "0.2.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e262cdaac42494e3ae34c43969f9cdeb7da178bdb4b66fa6a1ea2edb4c8ae652" -dependencies = [ - "typewit", -] - [[package]] name = "constant_time_eq" version = "0.4.2" @@ -756,28 +681,28 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" [[package]] -name = "core_extensions" -version = "1.5.4" +name = "cpufeatures" +version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42bb5e5d0269fd4f739ea6cedaf29c16d81c27a7ce7582008e90eb50dcd57003" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" dependencies = [ - "core_extensions_proc_macros", + "libc", ] [[package]] -name = "core_extensions_proc_macros" -version = "1.5.4" +name = "crc" +version = "3.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "533d38ecd2709b7608fb8e18e4504deb99e9a72879e6aa66373a76d8dc4259ea" +checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d" +dependencies = [ + "crc-catalog", +] [[package]] -name = "cpufeatures" -version = "0.2.17" +name = "crc-catalog" +version = "2.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" -dependencies = [ - "libc", -] +checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853" [[package]] name = "crc32fast" @@ -788,15 +713,6 @@ dependencies = [ "cfg-if", ] -[[package]] -name = "crossbeam-channel" -version = "0.5.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "82b8f8f868b36967f9606790d1903570de9ceaf870a7bf9fbbd3016d636a2cb2" -dependencies = [ - "crossbeam-utils", -] - [[package]] name = "crossbeam-utils" version = "0.8.21" @@ -819,6 +735,15 @@ dependencies = [ "typenum", ] +[[package]] +name = "crypto-common" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" +dependencies = [ + "hybrid-array", +] + [[package]] name = "cstr" version = "0.2.12" @@ -850,45 +775,11 @@ dependencies = [ "memchr", ] -[[package]] -name = "darling" -version = "0.23.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "25ae13da2f202d56bd7f91c25fba009e7717a1e4a1cc98a76d844b65ae912e9d" -dependencies = [ - "darling_core", - "darling_macro", -] - -[[package]] -name = "darling_core" -version = "0.23.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9865a50f7c335f53564bb694ef660825eb8610e0a53d3e11bf1b0d3df31e03b0" -dependencies = [ - "ident_case", - "proc-macro2", - "quote", - "strsim", - "syn 2.0.117", -] - -[[package]] -name = "darling_macro" -version = "0.23.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac3984ec7bd6cfa798e62b4a642426a5be0e68f9401cfc2a01e3fa9ea2fcdb8d" -dependencies = [ - "darling_core", - "quote", - "syn 2.0.117", -] - [[package]] name = "dashmap" -version = "6.1.0" +version = "6.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5041cc499144891f3790297212f32a74fb938e5136a14943f338ef9e0ae276cf" +checksum = "e6361d5c062261c78a176addb82d4c821ae42bed6089de0e12603cd25de2059c" dependencies = [ "cfg-if", "crossbeam-utils", @@ -900,13 +791,12 @@ dependencies = [ [[package]] name = "datafusion" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "arrow-schema", "async-trait", - "bytes", "bzip2", "chrono", "datafusion-catalog", @@ -937,14 +827,13 @@ dependencies = [ "datafusion-sql", "flate2", "futures", - "itertools", + "indexmap", + "itertools 0.15.0", "liblzma", "log", "object_store", "parking_lot", "parquet", - "rand", - "regex", "sqlparser", "tempfile", "tokio", @@ -955,8 +844,8 @@ dependencies = [ [[package]] name = "datafusion-catalog" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "async-trait", @@ -970,7 +859,7 @@ dependencies = [ "datafusion-physical-plan", "datafusion-session", "futures", - "itertools", + "itertools 0.15.0", "log", "object_store", "parking_lot", @@ -979,8 +868,8 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "async-trait", @@ -994,40 +883,42 @@ dependencies = [ "datafusion-physical-expr-common", "datafusion-physical-plan", "futures", - "itertools", + "itertools 0.15.0", "log", "object_store", + "percent-encoding", ] [[package]] name = "datafusion-common" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ - "ahash", - "apache-avro", "arrow", "arrow-ipc", + "arrow-schema", "chrono", + "foldhash 0.2.0", "half", - "hashbrown 0.16.1", + "hashbrown 0.17.1", "indexmap", - "itertools", + "itertools 0.15.0", "libc", "log", + "num-traits", "object_store", "parquet", - "paste", "recursive", "sqlparser", "tokio", + "uuid", "web-time", ] [[package]] name = "datafusion-common-runtime" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "futures", "log", @@ -1036,8 +927,8 @@ dependencies = [ [[package]] name = "datafusion-datasource" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "async-compression", @@ -1053,15 +944,17 @@ dependencies = [ "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-plan", + "datafusion-proto-models", "datafusion-session", "flate2", "futures", "glob", - "itertools", + "itertools 0.15.0", "liblzma", "log", "object_store", - "rand", + "parking_lot", + "rand 0.9.4", "tokio", "tokio-util", "url", @@ -1070,8 +963,8 @@ dependencies = [ [[package]] name = "datafusion-datasource-arrow" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "arrow-ipc", @@ -1084,36 +977,36 @@ dependencies = [ "datafusion-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", + "datafusion-proto-models", "datafusion-session", "futures", - "itertools", + "itertools 0.15.0", "object_store", "tokio", ] [[package]] name = "datafusion-datasource-avro" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ - "apache-avro", "arrow", + "arrow-avro", "async-trait", "bytes", "datafusion-common", "datafusion-datasource", - "datafusion-physical-expr-common", + "datafusion-physical-expr-adapter", "datafusion-physical-plan", "datafusion-session", "futures", - "num-traits", "object_store", ] [[package]] name = "datafusion-datasource-csv" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "async-trait", @@ -1125,6 +1018,7 @@ dependencies = [ "datafusion-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", + "datafusion-proto-models", "datafusion-session", "futures", "object_store", @@ -1134,8 +1028,8 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "async-trait", @@ -1147,20 +1041,21 @@ dependencies = [ "datafusion-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", + "datafusion-proto-models", "datafusion-session", "futures", "object_store", - "serde_json", "tokio", "tokio-stream", ] [[package]] name = "datafusion-datasource-parquet" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", + "arrow-schema", "async-trait", "bytes", "datafusion-common", @@ -1168,15 +1063,17 @@ dependencies = [ "datafusion-datasource", "datafusion-execution", "datafusion-expr", + "datafusion-functions", "datafusion-functions-aggregate-common", "datafusion-physical-expr", "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-plan", + "datafusion-proto-models", "datafusion-pruning", "datafusion-session", "futures", - "itertools", + "itertools 0.15.0", "log", "object_store", "parking_lot", @@ -1186,18 +1083,18 @@ dependencies = [ [[package]] name = "datafusion-doc" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" [[package]] name = "datafusion-execution" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "arrow-buffer", "async-trait", - "chrono", + "bytes", "dashmap", "datafusion-common", "datafusion-expr", @@ -1206,17 +1103,21 @@ dependencies = [ "log", "object_store", "parking_lot", - "rand", + "pin-project-lite", + "rand 0.9.4", "tempfile", + "tokio", + "tokio-util", "url", ] [[package]] name = "datafusion-expr" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", + "arrow-schema", "async-trait", "chrono", "datafusion-common", @@ -1225,9 +1126,10 @@ dependencies = [ "datafusion-functions-aggregate-common", "datafusion-functions-window-common", "datafusion-physical-expr-common", + "datafusion-proto-common", + "datafusion-proto-models", "indexmap", - "itertools", - "paste", + "itertools 0.15.0", "recursive", "serde_json", "sqlparser", @@ -1235,26 +1137,25 @@ dependencies = [ [[package]] name = "datafusion-expr-common" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "datafusion-common", "indexmap", - "itertools", - "paste", + "itertools 0.15.0", ] [[package]] name = "datafusion-ffi" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ - "abi_stable", "arrow", "arrow-schema", "async-ffi", "async-trait", + "chrono", "datafusion-catalog", "datafusion-common", "datafusion-datasource", @@ -1263,25 +1164,50 @@ dependencies = [ "datafusion-functions-aggregate-common", "datafusion-physical-expr", "datafusion-physical-expr-common", + "datafusion-physical-optimizer", "datafusion-physical-plan", "datafusion-proto", "datafusion-proto-common", "datafusion-session", "futures", + "libloading", "log", "prost", "semver", + "stabby", "tokio", ] +[[package]] +name = "datafusion-ffi-example" +version = "54.0.0" +dependencies = [ + "arrow", + "arrow-array", + "arrow-schema", + "async-trait", + "datafusion", + "datafusion-catalog", + "datafusion-common", + "datafusion-expr", + "datafusion-ffi", + "datafusion-functions-aggregate", + "datafusion-functions-window", + "datafusion-proto", + "datafusion-python-util", + "pyo3", + "pyo3-build-config", + "pyo3-log", +] + [[package]] name = "datafusion-functions" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "arrow-buffer", - "base64", + "base64 0.23.1", "blake2", "blake3", "chrono", @@ -1292,25 +1218,24 @@ dependencies = [ "datafusion-expr", "datafusion-expr-common", "datafusion-macros", + "datafusion-physical-expr-common", "hex", - "itertools", + "itertools 0.15.0", "log", - "md-5", + "md-5 0.11.0", "memchr", "num-traits", - "rand", + "rand 0.9.4", "regex", "sha2", - "unicode-segmentation", "uuid", ] [[package]] name = "datafusion-functions-aggregate" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ - "ahash", "arrow", "datafusion-common", "datafusion-doc", @@ -1321,17 +1246,16 @@ dependencies = [ "datafusion-physical-expr", "datafusion-physical-expr-common", "half", + "hashbrown 0.17.1", "log", "num-traits", - "paste", ] [[package]] name = "datafusion-functions-aggregate-common" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ - "ahash", "arrow", "datafusion-common", "datafusion-expr-common", @@ -1340,8 +1264,8 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "arrow-ord", @@ -1355,32 +1279,32 @@ dependencies = [ "datafusion-functions-aggregate-common", "datafusion-macros", "datafusion-physical-expr-common", - "hashbrown 0.16.1", - "itertools", + "hashbrown 0.17.1", + "itertools 0.15.0", "itoa", "log", - "paste", + "memchr", ] [[package]] name = "datafusion-functions-table" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "async-trait", "datafusion-catalog", "datafusion-common", "datafusion-expr", + "datafusion-physical-expr", "datafusion-physical-plan", "parking_lot", - "paste", ] [[package]] name = "datafusion-functions-window" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "datafusion-common", @@ -1391,13 +1315,12 @@ dependencies = [ "datafusion-physical-expr", "datafusion-physical-expr-common", "log", - "paste", ] [[package]] name = "datafusion-functions-window-common" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -1405,18 +1328,18 @@ dependencies = [ [[package]] name = "datafusion-macros" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "datafusion-doc", "quote", - "syn 2.0.117", + "syn 3.0.3", ] [[package]] name = "datafusion-optimizer" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "chrono", @@ -1425,7 +1348,7 @@ dependencies = [ "datafusion-expr-common", "datafusion-physical-expr", "indexmap", - "itertools", + "itertools 0.15.0", "log", "recursive", "regex", @@ -1434,22 +1357,21 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ - "ahash", "arrow", "datafusion-common", "datafusion-expr", "datafusion-expr-common", "datafusion-functions-aggregate-common", "datafusion-physical-expr-common", + "datafusion-proto-models", "half", - "hashbrown 0.16.1", + "hashbrown 0.17.1", "indexmap", - "itertools", + "itertools 0.15.0", "parking_lot", - "paste", "petgraph", "recursive", "tokio", @@ -1457,8 +1379,8 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-adapter" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "datafusion-common", @@ -1466,29 +1388,30 @@ dependencies = [ "datafusion-functions", "datafusion-physical-expr", "datafusion-physical-expr-common", - "itertools", + "itertools 0.15.0", ] [[package]] name = "datafusion-physical-expr-common" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ - "ahash", "arrow", "chrono", "datafusion-common", "datafusion-expr-common", - "hashbrown 0.16.1", + "datafusion-proto-models", + "hashbrown 0.17.1", "indexmap", - "itertools", + "itertools 0.15.0", "parking_lot", + "pin-project", ] [[package]] name = "datafusion-physical-optimizer" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "datafusion-common", @@ -1499,20 +1422,23 @@ dependencies = [ "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-pruning", - "itertools", + "datafusion-session", + "itertools 0.15.0", "recursive", ] [[package]] name = "datafusion-physical-plan" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ - "ahash", "arrow", + "arrow-data", + "arrow-ipc", "arrow-ord", "arrow-schema", "async-trait", + "bytes", "datafusion-common", "datafusion-common-runtime", "datafusion-execution", @@ -1522,25 +1448,27 @@ dependencies = [ "datafusion-functions-window-common", "datafusion-physical-expr", "datafusion-physical-expr-common", + "datafusion-proto-common", + "datafusion-proto-models", "futures", "half", - "hashbrown 0.16.1", + "hashbrown 0.17.1", "indexmap", - "itertools", + "itertools 0.15.0", "log", "num-traits", "parking_lot", "pin-project-lite", + "serde_json", "tokio", ] [[package]] name = "datafusion-proto" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", - "chrono", "datafusion-catalog", "datafusion-catalog-listing", "datafusion-common", @@ -1556,25 +1484,35 @@ dependencies = [ "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-proto-common", + "datafusion-proto-models", "object_store", "prost", - "rand", ] [[package]] name = "datafusion-proto-common" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "datafusion-common", "prost", ] +[[package]] +name = "datafusion-proto-models" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" +dependencies = [ + "datafusion-common", + "datafusion-proto-common", + "prost", +] + [[package]] name = "datafusion-pruning" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "datafusion-common", @@ -1583,21 +1521,23 @@ dependencies = [ "datafusion-physical-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", - "itertools", "log", ] [[package]] name = "datafusion-python" -version = "52.0.0" +version = "54.0.0" dependencies = [ "arrow", "arrow-select", "async-trait", + "chrono", "cstr", "datafusion", "datafusion-ffi", "datafusion-proto", + "datafusion-python-util", + "datafusion-spark", "datafusion-substrait", "futures", "log", @@ -1616,11 +1556,25 @@ dependencies = [ "uuid", ] +[[package]] +name = "datafusion-python-util" +version = "54.0.0" +dependencies = [ + "arrow", + "datafusion", + "datafusion-ffi", + "datafusion-proto", + "prost", + "pyo3", + "tokio", +] + [[package]] name = "datafusion-session" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ + "arrow-schema", "async-trait", "datafusion-common", "datafusion-execution", @@ -1629,10 +1583,39 @@ dependencies = [ "parking_lot", ] +[[package]] +name = "datafusion-spark" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" +dependencies = [ + "arrow", + "bigdecimal", + "chrono", + "crc32fast", + "datafusion", + "datafusion-catalog", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions", + "datafusion-functions-aggregate", + "datafusion-functions-aggregate-common", + "datafusion-functions-nested", + "log", + "num-traits", + "percent-encoding", + "rand 0.9.4", + "serde_json", + "sha1", + "sha2", + "twox-hash", + "url", +] + [[package]] name = "datafusion-sql" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "arrow", "bigdecimal", @@ -1645,19 +1628,20 @@ dependencies = [ "recursive", "regex", "sqlparser", + "stacker", ] [[package]] name = "datafusion-substrait" -version = "53.0.0" -source = "git+https://github.com/apache/datafusion.git?rev=35749607f585b3bf25b66b7d2289c56c18d03e4f#35749607f585b3bf25b66b7d2289c56c18d03e4f" +version = "55.0.0" +source = "git+https://github.com/apache/datafusion?rev=55.0.0-rc3#d5552342012888b7d1a3ab88d92e3d292fc0cde0" dependencies = [ "async-recursion", "async-trait", "chrono", "datafusion", "half", - "itertools", + "itertools 0.15.0", "object_store", "pbjson-types", "prost", @@ -1672,20 +1656,31 @@ version = "0.10.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" dependencies = [ - "block-buffer", - "crypto-common", + "block-buffer 0.10.4", + "crypto-common 0.1.7", "subtle", ] +[[package]] +name = "digest" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" +dependencies = [ + "block-buffer 0.12.1", + "const-oid", + "crypto-common 0.2.2", +] + [[package]] name = "displaydoc" -version = "0.2.5" +version = "0.2.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" +checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -1696,9 +1691,9 @@ checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" [[package]] name = "either" -version = "1.15.0" +version = "1.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719" +checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" [[package]] name = "equivalent" @@ -1718,9 +1713,9 @@ dependencies = [ [[package]] name = "fastrand" -version = "2.3.0" +version = "2.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be" +checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6" [[package]] name = "find-msvc-tools" @@ -1838,7 +1833,7 @@ checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -1870,15 +1865,6 @@ dependencies = [ "slab", ] -[[package]] -name = "generational-arena" -version = "0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877e94aff08e743b651baaea359664321055749b398adff8740a7399af7796e7" -dependencies = [ - "cfg-if", -] - [[package]] name = "generic-array" version = "0.14.7" @@ -1911,22 +1897,21 @@ dependencies = [ "cfg-if", "js-sys", "libc", - "r-efi", + "r-efi 5.3.0", "wasip2", "wasm-bindgen", ] [[package]] name = "getrandom" -version = "0.4.1" +version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "139ef39800118c7683f2fd3c98c1b23c09ae076556b435f8e9064ae108aaeeec" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" dependencies = [ "cfg-if", "libc", - "r-efi", - "wasip2", - "wasip3", + "r-efi 6.0.0", + "rand_core 0.10.1", ] [[package]] @@ -1937,9 +1922,9 @@ checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" [[package]] name = "h2" -version = "0.4.13" +version = "0.4.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2f44da3a8150a6703ed5d34e164b875fd14c2cdab9af1252a9a1020bde2bdc54" +checksum = "6cb093c84e8bd9b188d4c4a8cb6579fc016968d14c99882163cd3ff402a4f155" dependencies = [ "atomic-waker", "bytes", @@ -1992,6 +1977,17 @@ dependencies = [ "foldhash 0.2.0", ] +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" +dependencies = [ + "allocator-api2", + "equivalent", + "foldhash 0.2.0", +] + [[package]] name = "heck" version = "0.5.0" @@ -2006,9 +2002,9 @@ checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" [[package]] name = "http" -version = "1.4.0" +version = "1.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3ba2a386d7f85a81f119ad7498ebe444d2e22c2af0b86b069416ace48b3311a" +checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425" dependencies = [ "bytes", "itoa", @@ -2049,11 +2045,20 @@ version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424" +[[package]] +name = "hybrid-array" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9155a582abd142abc056962c29e3ce5ff2ad5469f4246b537ed42c5deba857da" +dependencies = [ + "typenum", +] + [[package]] name = "hyper" -version = "1.8.1" +version = "1.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2ab2d4f250c3d7b1c9fcdff1cece94ea4e2dfbec68614f7b87cb205f24ca9d11" +checksum = "55281c53a1894c864990125767da440a4e630446785086f52523b20033b74498" dependencies = [ "atomic-waker", "bytes", @@ -2065,7 +2070,6 @@ dependencies = [ "httparse", "itoa", "pin-project-lite", - "pin-utils", "smallvec", "tokio", "want", @@ -2073,16 +2077,15 @@ dependencies = [ [[package]] name = "hyper-rustls" -version = "0.27.7" +version = "0.27.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3c93eb611681b207e1fe55d5a71ecf91572ec8a6705cdb6857f7d8d5242cf58" +checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f" dependencies = [ "http", "hyper", "hyper-util", "rustls", "rustls-native-certs", - "rustls-pki-types", "tokio", "tokio-rustls", "tower-service", @@ -2094,7 +2097,7 @@ version = "0.1.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" dependencies = [ - "base64", + "base64 0.22.1", "bytes", "futures-channel", "futures-util", @@ -2137,12 +2140,13 @@ dependencies = [ [[package]] name = "icu_collections" -version = "2.1.1" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c6b649701667bbe825c3b7e6388cb521c23d88644678e83c0c4d0a621a34b43" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" dependencies = [ "displaydoc", "potential_utf", + "utf8_iter", "yoke", "zerofrom", "zerovec", @@ -2150,9 +2154,9 @@ dependencies = [ [[package]] name = "icu_locale_core" -version = "2.1.1" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "edba7861004dd3714265b4db54a3c390e880ab658fec5f7db895fae2046b5bb6" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" dependencies = [ "displaydoc", "litemap", @@ -2163,9 +2167,9 @@ dependencies = [ [[package]] name = "icu_normalizer" -version = "2.1.1" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f6c8828b67bf8908d82127b2054ea1b4427ff0230ee9141c54251934ab1b599" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" dependencies = [ "icu_collections", "icu_normalizer_data", @@ -2177,15 +2181,15 @@ dependencies = [ [[package]] name = "icu_normalizer_data" -version = "2.1.1" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7aedcccd01fc5fe81e6b489c15b247b8b0690feb23304303a9e560f37efc560a" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" [[package]] name = "icu_properties" -version = "2.1.2" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "020bfc02fe870ec3a66d93e677ccca0562506e5872c650f893269e08615d74ec" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" dependencies = [ "icu_collections", "icu_locale_core", @@ -2197,15 +2201,15 @@ dependencies = [ [[package]] name = "icu_properties_data" -version = "2.1.2" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "616c294cf8d725c6afcd8f55abc17c56464ef6211f9ed59cccffe534129c77af" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" [[package]] name = "icu_provider" -version = "2.1.1" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85962cf0ce02e1e0a629cc34e7ca3e373ce20dda4c4d7294bbd0bf1fdb59e614" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" dependencies = [ "displaydoc", "icu_locale_core", @@ -2216,18 +2220,6 @@ dependencies = [ "zerovec", ] -[[package]] -name = "id-arena" -version = "2.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" - -[[package]] -name = "ident_case" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39" - [[package]] name = "idna" version = "1.1.0" @@ -2241,9 +2233,9 @@ dependencies = [ [[package]] name = "idna_adapter" -version = "1.2.1" +version = "1.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3acae9609540aa318d1bc588455225fb2085b9ed0c4f6bd0d9d5bcd86f1a0344" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" dependencies = [ "icu_normalizer", "icu_properties", @@ -2251,52 +2243,45 @@ dependencies = [ [[package]] name = "indexmap" -version = "2.13.0" +version = "2.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7714e70437a7dc3ac8eb7e6f8df75fd8eb422675fc7678aff7364301092b1017" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" dependencies = [ "equivalent", - "hashbrown 0.16.1", + "hashbrown 0.17.1", "serde", "serde_core", ] -[[package]] -name = "integer-encoding" -version = "3.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02" - [[package]] name = "ipnet" -version = "2.11.0" +version = "2.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "469fb0b9cefa57e3ef31275ee7cacb78f2fdca44e4765491884a2b119d4eb130" +checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" [[package]] -name = "iri-string" -version = "0.7.10" +name = "itertools" +version = "0.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c91338f0783edbd6195decb37bae672fd3b165faffb89bf7b9e6942f8b1a731a" +checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" dependencies = [ - "memchr", - "serde", + "either", ] [[package]] name = "itertools" -version = "0.14.0" +version = "0.15.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" +checksum = "8b4baf93f58d4425749ca49a51c50ebab072c5df6994d08fed93541c331481dc" dependencies = [ "either", ] [[package]] name = "itoa" -version = "1.0.17" +version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92ecc6618181def0457392ccd0ee51198e065e016d1d527a7ac1b6dc7c1f09d2" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" [[package]] name = "jobserver" @@ -2310,20 +2295,15 @@ dependencies = [ [[package]] name = "js-sys" -version = "0.3.85" +version = "0.3.102" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c942ebf8e95485ca0d52d97da7c5a2c387d0e7f0ba4c35e93bfcaee045955b3" +checksum = "03d04c30968dffe80775bd4d7fb676131cd04a1fb46d2686dbffbaec2d9dfd31" dependencies = [ - "once_cell", + "cfg-if", + "futures-util", "wasm-bindgen", ] -[[package]] -name = "leb128fmt" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" - [[package]] name = "lexical-core" version = "1.0.6" @@ -2383,24 +2363,24 @@ dependencies = [ [[package]] name = "libbz2-rs-sys" -version = "0.2.2" +version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2c4a545a15244c7d945065b5d392b2d2d7f21526fba56ce51467b06ed445e8f7" +checksum = "34b357333733e8260735ba5894eb928c02ecc69c78715f01a8019e7fa7f2db4c" [[package]] name = "libc" -version = "0.2.182" +version = "0.2.186" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6800badb6cb2082ffd7b6a67e6125bb39f18782f793520caee8cb8846be06112" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" [[package]] name = "libloading" -version = "0.7.4" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b67380fd3b2fbe7527a606e18729d21c6f3951633d0500574c4dc22d2d638b9f" +checksum = "754ca22de805bb5744484a5b151a9e1a8e837d5dc232c2d7d8c2e3492edc8b60" dependencies = [ "cfg-if", - "winapi", + "windows-link", ] [[package]] @@ -2414,9 +2394,9 @@ dependencies = [ [[package]] name = "liblzma-sys" -version = "0.4.5" +version = "0.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f2db66f3268487b5033077f266da6777d057949b8f93c8ad82e441df25e6186" +checksum = "1a60851d15cd8c5346eca4ab8babff585be2ae4bc8097c067291d3ffe2add3b6" dependencies = [ "cc", "libc", @@ -2431,25 +2411,24 @@ checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" [[package]] name = "libmimalloc-sys" -version = "0.1.44" +version = "0.1.49" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "667f4fec20f29dfc6bc7357c582d91796c169ad7e2fce709468aefeb2c099870" +checksum = "6a45a52f43e1c16f667ccfe4dd8c85b7f7c204fd5e3bf46c5b0db9a5c3c0b8e9" dependencies = [ "cc", - "libc", ] [[package]] name = "linux-raw-sys" -version = "0.11.0" +version = "0.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df1d3c3b53da64cf5760482273a98e575c651a67eec7f77df96b5b642de8f039" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" [[package]] name = "litemap" -version = "0.8.1" +version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6373607a59f0be73a39b6fe456b8192fcc3585f602af20751600e974dd455e77" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" [[package]] name = "lock_api" @@ -2462,9 +2441,9 @@ dependencies = [ [[package]] name = "log" -version = "0.4.29" +version = "0.4.32" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897" +checksum = "953f07c43838f8e6f9758cab68bf5bed85465e7587ebe0b823f1bcd81978ad3a" [[package]] name = "lru-slab" @@ -2474,9 +2453,9 @@ checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154" [[package]] name = "lz4_flex" -version = "0.12.0" +version = "0.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ab6473172471198271ff72e9379150e9dfd70d8e533e0752a27e515b48dd375e" +checksum = "ecbdfe44b1bd960b68170b417450a628c43f7cf56bb3c5317e61cb230ee7f226" dependencies = [ "twox-hash", ] @@ -2488,20 +2467,30 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf" dependencies = [ "cfg-if", - "digest", + "digest 0.10.7", +] + +[[package]] +name = "md-5" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" +dependencies = [ + "cfg-if", + "digest 0.11.3", ] [[package]] name = "memchr" -version = "2.8.0" +version = "2.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" [[package]] name = "mimalloc" -version = "0.1.48" +version = "0.1.52" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e1ee66a4b64c74f4ef288bcbb9192ad9c3feaad75193129ac8509af543894fd8" +checksum = "2d4139bb28d14ad1facf21d5eb8825051b326e172d216b39f6d31df53cc97862" dependencies = [ "libmimalloc-sys", ] @@ -2518,9 +2507,9 @@ dependencies = [ [[package]] name = "mio" -version = "1.1.1" +version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a69bcab0ad47271a0234d9422b131806bf3968021e5dc9328caf2d4cd58557fc" +checksum = "02bd0af71c67b473010cbbc60715ee815645a4dc942899111f494b4b737d6fda" dependencies = [ "libc", "wasi", @@ -2541,7 +2530,16 @@ checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" dependencies = [ "num-integer", "num-traits", - "serde", +] + +[[package]] +name = "num-bigint" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93e7820bc0a80a0238e650327316f929ba18d5be054b647490a3a6a339f3e7c0" +dependencies = [ + "num-integer", + "num-traits", ] [[package]] @@ -2583,27 +2581,29 @@ dependencies = [ [[package]] name = "object_store" -version = "0.13.1" +version = "0.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2858065e55c148d294a9f3aae3b0fa9458edadb41a108397094566f4e3c0dfb" +checksum = "622acbc9100d3c10e2ee15804b0caa40e55c933d5aa53814cd520805b7958a49" dependencies = [ "async-trait", - "base64", + "base64 0.22.1", "bytes", "chrono", "form_urlencoded", - "futures", + "futures-channel", + "futures-core", + "futures-util", "http", "http-body-util", "httparse", "humantime", "hyper", - "itertools", - "md-5", + "itertools 0.14.0", + "md-5 0.10.6", "parking_lot", "percent-encoding", "quick-xml", - "rand", + "rand 0.10.1", "reqwest", "ring", "rustls-pki-types", @@ -2621,9 +2621,9 @@ dependencies = [ [[package]] name = "once_cell" -version = "1.21.3" +version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" [[package]] name = "openssl-probe" @@ -2631,15 +2631,6 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" -[[package]] -name = "ordered-float" -version = "2.10.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "68f19d67e5a2795c94e73e0bb1cc1a7edeb2e28efd39e2e1c9b7a40c1108b11c" -dependencies = [ - "num-traits", -] - [[package]] name = "parking_lot" version = "0.12.5" @@ -2665,9 +2656,9 @@ dependencies = [ [[package]] name = "parquet" -version = "58.0.0" +version = "59.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f491d0ef1b510194426ee67ddc18a9b747ef3c42050c19322a2cd2e1666c29b" +checksum = "7065842956a20c2a536924ce8e4d9955f7422451511b9eb7500d7bfe5077e59c" dependencies = [ "ahash", "arrow-array", @@ -2676,42 +2667,34 @@ dependencies = [ "arrow-ipc", "arrow-schema", "arrow-select", - "base64", + "base64 0.23.1", "brotli", "bytes", "chrono", "flate2", "futures", "half", - "hashbrown 0.16.1", + "hashbrown 0.17.1", "lz4_flex", - "num-bigint", + "num-bigint 0.5.1", "num-integer", "num-traits", "object_store", - "paste", "seq-macro", "simdutf8", "snap", - "thrift", "tokio", "twox-hash", "zstd", ] -[[package]] -name = "paste" -version = "1.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" - [[package]] name = "pbjson" version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "898bac3fa00d0ba57a4e8289837e965baa2dee8c3749f3b11d45a64b4223d9c3" dependencies = [ - "base64", + "base64 0.22.1", "serde", ] @@ -2722,7 +2705,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "af22d08a625a2213a78dbb0ffa253318c5c79ce3133d32d296655a7bdfb02095" dependencies = [ "heck", - "itertools", + "itertools 0.14.0", "prost", "prost-types", ] @@ -2779,22 +2762,36 @@ dependencies = [ ] [[package]] -name = "pin-project-lite" -version = "0.2.16" +name = "pin-project" +version = "1.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3b3cff922bd51709b605d9ead9aa71031d81447142d828eb4a6eba76fe619f9b" +checksum = "2466b2336ed02bcdca6b294417127b90ec92038d1d5c4fbeac971a922e0e0924" +dependencies = [ + "pin-project-internal", +] [[package]] -name = "pin-utils" -version = "0.1.0" +name = "pin-project-internal" +version = "1.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" [[package]] name = "pkg-config" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" [[package]] name = "portable-atomic" @@ -2804,9 +2801,9 @@ checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" [[package]] name = "potential_utf" -version = "0.1.4" +version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b73949432f5e2a09657003c25bca5e19a0e9c84f8058ca374f49e0ebe605af77" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" dependencies = [ "zerovec", ] @@ -2827,7 +2824,16 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" dependencies = [ "proc-macro2", - "syn 2.0.117", + "syn 2.0.118", +] + +[[package]] +name = "proc-macro-crate" +version = "3.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f" +dependencies = [ + "toml_edit", ] [[package]] @@ -2841,9 +2847,9 @@ dependencies = [ [[package]] name = "prost" -version = "0.14.3" +version = "0.14.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2ea70524a2f82d518bce41317d0fae74151505651af45faf1ffbd6fd33f0568" +checksum = "528ac67416ff8646872a3c02cad9cc4ee5dc9f9540c9b10771855c95cb2e5ae1" dependencies = [ "bytes", "prost-derive", @@ -2851,12 +2857,12 @@ dependencies = [ [[package]] name = "prost-build" -version = "0.14.3" +version = "0.14.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "343d3bd7056eda839b03204e68deff7d1b13aba7af2b2fd16890697274262ee7" +checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042" dependencies = [ "heck", - "itertools", + "itertools 0.14.0", "log", "multimap", "petgraph", @@ -2864,28 +2870,28 @@ dependencies = [ "prost", "prost-types", "regex", - "syn 2.0.117", + "syn 2.0.118", "tempfile", ] [[package]] name = "prost-derive" -version = "0.14.3" +version = "0.14.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "27c6023962132f4b30eb4c172c91ce92d933da334c59c23cddee82358ddafb0b" +checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf" dependencies = [ "anyhow", - "itertools", + "itertools 0.14.0", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] name = "prost-types" -version = "0.14.3" +version = "0.14.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8991c4cbdb8bc5b11f0b074ffe286c30e523de90fee5ba8132f1399f23cb3dd7" +checksum = "f94967dc7688f3054c7fac87473ffae4cc4c3904800e2d9f5b857246d8963b0a" dependencies = [ "prost", ] @@ -2901,9 +2907,9 @@ dependencies = [ [[package]] name = "psm" -version = "0.1.30" +version = "0.1.31" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3852766467df634d74f0b2d7819bf8dc483a0eb2e3b0f50f756f9cfe8b0d18d8" +checksum = "645dbe486e346d9b5de3ef16ede18c26e6c70ad97418f4874b8b1889d6e761ea" dependencies = [ "ar_archive_writer", "cc", @@ -2911,9 +2917,9 @@ dependencies = [ [[package]] name = "pyo3" -version = "0.28.2" +version = "0.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf85e27e86080aafd5a22eae58a162e133a589551542b3e5cee4beb27e54f8e1" +checksum = "cd274650b21d4bfc26a0a47587962c1edb425f69287324355cd040c3ea66071c" dependencies = [ "libc", "once_cell", @@ -2925,9 +2931,9 @@ dependencies = [ [[package]] name = "pyo3-async-runtimes" -version = "0.28.0" +version = "0.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e7364a95bf00e8377bbf9b0f09d7ff9715a29d8fcf93b47d1a967363b973178" +checksum = "b3ef68daa7316a3fac65e5e18b2203f010346de1c1c53456811a2624673ab046" dependencies = [ "futures-channel", "futures-util", @@ -2939,18 +2945,18 @@ dependencies = [ [[package]] name = "pyo3-build-config" -version = "0.28.2" +version = "0.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8bf94ee265674bf76c09fa430b0e99c26e319c945d96ca0d5a8215f31bf81cf7" +checksum = "c5e2a7d2f0d013342f295c048ad19237add5154a55b1c5a254c0ec93d4109078" dependencies = [ "target-lexicon", ] [[package]] name = "pyo3-ffi" -version = "0.28.2" +version = "0.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "491aa5fc66d8059dd44a75f4580a2962c1862a1c2945359db36f6c2818b748dc" +checksum = "ca85c467da1bbc8d866eea5deff9cf29ea5f7785054a17da36e65bda9c05845b" dependencies = [ "libc", "pyo3-build-config", @@ -2958,9 +2964,9 @@ dependencies = [ [[package]] name = "pyo3-log" -version = "0.13.3" +version = "0.13.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "26c2ec80932c5c3b2d4fbc578c9b56b2d4502098587edb8bef5b6bfcad43682e" +checksum = "f64083bd3a16a353d9d62335808e8e13d0552d2a2b83fdb084496192dcfa9fcd" dependencies = [ "arc-swap", "log", @@ -2969,40 +2975,33 @@ dependencies = [ [[package]] name = "pyo3-macros" -version = "0.28.2" +version = "0.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f5d671734e9d7a43449f8480f8b38115df67bef8d21f76837fa75ee7aaa5e52e" +checksum = "9ac53762fd065daa3194dd09337a38bd793a188100fd1a9304c4ab312d901771" dependencies = [ "proc-macro2", "pyo3-macros-backend", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] name = "pyo3-macros-backend" -version = "0.28.2" +version = "0.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22faaa1ce6c430a1f71658760497291065e6450d7b5dc2bcf254d49f66ee700a" +checksum = "4ca3a1557399783172dc5bf39cfca835157732532cba56b71d2292161e53b362" dependencies = [ "heck", "proc-macro2", - "pyo3-build-config", "quote", - "syn 2.0.117", + "syn 2.0.118", ] -[[package]] -name = "quad-rand" -version = "0.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a651516ddc9168ebd67b24afd085a718be02f8858fe406591b013d101ce2f40" - [[package]] name = "quick-xml" -version = "0.38.4" +version = "0.39.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b66c2058c55a409d601666cffe35f04333cf1013010882cec174a7467cd4e21c" +checksum = "cdcc8dd4e2f670d309a5f0e83fe36dfdc05af317008fea29144da1a2ac858e5e" dependencies = [ "memchr", "serde", @@ -3030,14 +3029,14 @@ dependencies = [ [[package]] name = "quinn-proto" -version = "0.11.13" +version = "0.11.14" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f1906b49b0c3bc04b5fe5d86a77925ae6524a19b816ae38ce1e426255f1d8a31" +checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098" dependencies = [ "bytes", "getrandom 0.3.4", "lru-slab", - "rand", + "rand 0.9.4", "ring", "rustc-hash", "rustls", @@ -3065,9 +3064,9 @@ dependencies = [ [[package]] name = "quote" -version = "1.0.44" +version = "1.0.45" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "21b2ebcf727b7760c461f091f9f0f539b77b8e87f2fd88131e7f1b433b3cece4" +checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" dependencies = [ "proc-macro2", ] @@ -3078,14 +3077,31 @@ version = "5.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + [[package]] name = "rand" -version = "0.9.2" +version = "0.9.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6db2770f06117d490610c7488547d543617b21bfa07796d7a12f6f1bd53850d1" +checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea" dependencies = [ "rand_chacha", - "rand_core", + "rand_core 0.9.5", +] + +[[package]] +name = "rand" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2e8e8bcc7961af1fdac401278c6a831614941f6164ee3bf4ce61b7edb162207" +dependencies = [ + "chacha20", + "getrandom 0.4.3", + "rand_core 0.10.1", ] [[package]] @@ -3095,7 +3111,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" dependencies = [ "ppv-lite86", - "rand_core", + "rand_core 0.9.5", ] [[package]] @@ -3107,6 +3123,12 @@ dependencies = [ "getrandom 0.3.4", ] +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + [[package]] name = "recursive" version = "0.1.1" @@ -3124,7 +3146,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" dependencies = [ "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -3138,9 +3160,9 @@ dependencies = [ [[package]] name = "regex" -version = "1.12.3" +version = "1.12.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e10754a14b9137dd7b1e3e5b0493cc9171fdd105e0ab477f51b72e7f3ac0e276" +checksum = "f1292b7759ae1cb9ec195452d1390a074f0cd8541ab7a5a8c31cd6db45d4a6ba" dependencies = [ "aho-corasick", "memchr", @@ -3159,17 +3181,11 @@ dependencies = [ "regex-syntax", ] -[[package]] -name = "regex-lite" -version = "0.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cab834c73d247e67f4fae452806d17d3c7501756d98c8808d7c9c7aa7d18f973" - [[package]] name = "regex-syntax" -version = "0.8.9" +version = "0.8.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a96887878f22d7bad8a3b6dc5b7440e0ada9a245242924394987b21cf2210a4c" +checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" [[package]] name = "regress" @@ -3181,22 +3197,13 @@ dependencies = [ "memchr", ] -[[package]] -name = "repr_offset" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fb1070755bd29dffc19d0971cab794e607839ba2ef4b69a9e6fbc8733c1b72ea" -dependencies = [ - "tstr", -] - [[package]] name = "reqwest" version = "0.12.28" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147" dependencies = [ - "base64", + "base64 0.22.1", "bytes", "futures-core", "futures-util", @@ -3248,9 +3255,9 @@ dependencies = [ [[package]] name = "rustc-hash" -version = "2.1.1" +version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "357703d41365b4b27c590e3ed91eabb1b663f07c4c084095e60cbed4362dff0d" +checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" [[package]] name = "rustc_version" @@ -3263,9 +3270,9 @@ dependencies = [ [[package]] name = "rustix" -version = "1.1.3" +version = "1.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "146c9e247ccc180c1f61615433868c99f3de3ae256a30a43b49f67c2d9171f34" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" dependencies = [ "bitflags", "errno", @@ -3276,9 +3283,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.36" +version = "0.23.40" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c665f33d38cea657d9614f766881e4d510e0eda4239891eea56b4cadcf01801b" +checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b" dependencies = [ "once_cell", "ring", @@ -3290,9 +3297,9 @@ dependencies = [ [[package]] name = "rustls-native-certs" -version = "0.8.3" +version = "0.8.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "612460d5f7bea540c490b2b6395d8e34a953e52b491accd6c86c8164c5932a63" +checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d" dependencies = [ "openssl-probe", "rustls-pki-types", @@ -3302,9 +3309,9 @@ dependencies = [ [[package]] name = "rustls-pki-types" -version = "1.14.0" +version = "1.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "be040f8b0a225e40375822a563fa9524378b9d63112f53e19ffff34df5d33fdd" +checksum = "30a7197ae7eb376e574fe940d068c30fe0462554a3ddbe4eca7838e049c937a9" dependencies = [ "web-time", "zeroize", @@ -3312,9 +3319,9 @@ dependencies = [ [[package]] name = "rustls-webpki" -version = "0.103.9" +version = "0.103.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7df23109aa6c1567d1c575b9952556388da57401e4ace1d15f79eedad0d8f53" +checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" dependencies = [ "ring", "rustls-pki-types", @@ -3344,9 +3351,9 @@ dependencies = [ [[package]] name = "schannel" -version = "0.1.28" +version = "0.1.29" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "891d81b926048e76efe18581bf793546b4c0eaf8448d72be8de2bbee5fd166e1" +checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939" dependencies = [ "windows-sys 0.61.2", ] @@ -3372,7 +3379,7 @@ dependencies = [ "proc-macro2", "quote", "serde_derive_internals", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -3383,9 +3390,9 @@ checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" [[package]] name = "security-framework" -version = "3.6.0" +version = "3.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d17b898a6d6948c3a8ee4372c17cb384f90d2e6e912ef00895b14fd7ab54ec38" +checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d" dependencies = [ "bitflags", "core-foundation", @@ -3396,9 +3403,9 @@ dependencies = [ [[package]] name = "security-framework-sys" -version = "2.16.0" +version = "2.17.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "321c8673b092a9a42605034a9879d73cb79101ed5fd117bc9a597b89b4e9e61a" +checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3" dependencies = [ "core-foundation-sys", "libc", @@ -3406,9 +3413,9 @@ dependencies = [ [[package]] name = "semver" -version = "1.0.27" +version = "1.0.28" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d767eb0aabc880b29956c35734170f26ed551a859dbd361d140cdbeca61ab1e2" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" dependencies = [ "serde", "serde_core", @@ -3430,16 +3437,6 @@ dependencies = [ "serde_derive", ] -[[package]] -name = "serde_bytes" -version = "0.11.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8" -dependencies = [ - "serde", - "serde_core", -] - [[package]] name = "serde_core" version = "1.0.228" @@ -3457,7 +3454,7 @@ checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -3468,15 +3465,16 @@ checksum = "18d26a20a969b9e3fdf2fc2d9f21eda6c40e2de84c9408bb5d3b05d499aae711" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] name = "serde_json" -version = "1.0.149" +version = "1.0.150" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" dependencies = [ + "indexmap", "itoa", "memchr", "serde", @@ -3486,14 +3484,14 @@ dependencies = [ [[package]] name = "serde_tokenstream" -version = "0.2.2" +version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "64060d864397305347a78851c51588fd283767e7e7589829e8121d65512340f1" +checksum = "d7c49585c52c01f13c5c2ebb333f14f6885d76daa768d8a037d28017ec538c69" dependencies = [ "proc-macro2", "quote", "serde", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -3521,28 +3519,45 @@ dependencies = [ "unsafe-libyaml", ] +[[package]] +name = "sha1" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest 0.11.3", +] + [[package]] name = "sha2" -version = "0.10.9" +version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" +checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" dependencies = [ "cfg-if", "cpufeatures", - "digest", + "digest 0.11.3", ] +[[package]] +name = "sha2-const-stable" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5f179d4e11094a893b82fff208f74d448a7512f99f5a0acbd5c679b705f83ed9" + [[package]] name = "shlex" -version = "1.3.0" +version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" [[package]] name = "simd-adler32" -version = "0.3.8" +version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e320a6c5ad31d271ad523dcf3ad13e2767ad8b1cb8f047f75a8aeaf8da139da2" +checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" [[package]] name = "simdutf8" @@ -3552,9 +3567,9 @@ checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" [[package]] name = "siphasher" -version = "1.0.2" +version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b2aa850e253778c88a04c3d7323b043aeda9d3e30d5971937c1855769763678e" +checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649" [[package]] name = "slab" @@ -3564,9 +3579,9 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" [[package]] name = "smallvec" -version = "1.15.1" +version = "1.15.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" [[package]] name = "snap" @@ -3576,19 +3591,19 @@ checksum = "1b6b67fb9a61334225b5b790716f609cd58395f895b3fe8b328786812a40bc3b" [[package]] name = "socket2" -version = "0.6.2" +version = "0.6.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "86f4aa3ad99f2088c990dfa82d367e19cb29268ed67c574d10d0a4bfe71f07e0" +checksum = "52d1cfed4120b4d927bf7c0f86d2087a4a7d6027c906d9f9d525a80573b9be51" dependencies = [ "libc", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] name = "sqlparser" -version = "0.61.0" +version = "0.62.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dbf5ea8d4d7c808e1af1cbabebca9a2abe603bcefc22294c5b95018d53200cb7" +checksum = "13c6d1b651dc4edf07eead2a0c6c78016ce971bc2c10da5266861b13f25e7cec" dependencies = [ "log", "recursive", @@ -3603,7 +3618,41 @@ checksum = "a6dd45d8fc1c79299bfbb7190e42ccbbdf6a5f52e4a6ad98d92357ea965bd289" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", +] + +[[package]] +name = "stabby" +version = "72.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7b834ec7ced12095fea1e4b07dcb7e8cf2b59b18afa3eac52494d835965a5ec" +dependencies = [ + "rustversion", + "stabby-abi", +] + +[[package]] +name = "stabby-abi" +version = "72.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff1a4f477858a5bdf927c9fab7f579899de9b13e39f8b3b3b300c89fbab632f4" +dependencies = [ + "rustc_version", + "rustversion", + "sha2-const-stable", + "stabby-macros", +] + +[[package]] +name = "stabby-macros" +version = "72.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b31c4b2434980b67ad83f300a58088ba14d59454dcd79ba3d87419bbd924d31e" +dependencies = [ + "proc-macro-crate", + "proc-macro2", + "quote", + "syn 2.0.118", ] [[package]] @@ -3614,48 +3663,37 @@ checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" [[package]] name = "stacker" -version = "0.1.23" +version = "0.1.24" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08d74a23609d509411d10e2176dc2a4346e3b4aea2e7b1869f19fdedbc71c013" +checksum = "640c8cdd92b6b12f5bcb1803ca3bbf5ab96e5e6b6b96b9ab77dabe9e880b3190" dependencies = [ "cc", "cfg-if", "libc", "psm", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] -[[package]] -name = "strsim" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" - -[[package]] -name = "strum" -version = "0.27.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf" - [[package]] name = "strum_macros" -version = "0.27.2" +version = "0.28.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7695ce3845ea4b33927c055a39dc438a45b059f7c1b3d91d38d10355fb8cbca7" +checksum = "ab85eea0270ee17587ed4156089e10b9e6880ee688791d45a905f5b1ca36f664" dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] name = "substrait" -version = "0.62.2" +version = "0.63.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62fc4b483a129b9772ccb9c3f7945a472112fdd9140da87f8a4e7f1d44e045d0" +checksum = "e620ff4d5c02fd6f7752931aa74b16a26af66a63022cc1ad412c77edbe0bab47" dependencies = [ "heck", + "indexmap", "pbjson", "pbjson-build", "pbjson-types", @@ -3670,7 +3708,7 @@ dependencies = [ "serde", "serde_json", "serde_yaml", - "syn 2.0.117", + "syn 2.0.118", "typify", "walkdir", ] @@ -3683,9 +3721,9 @@ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" [[package]] name = "syn" -version = "1.0.109" +version = "2.0.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" dependencies = [ "proc-macro2", "quote", @@ -3694,9 +3732,9 @@ dependencies = [ [[package]] name = "syn" -version = "2.0.117" +version = "3.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" +checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" dependencies = [ "proc-macro2", "quote", @@ -3720,7 +3758,7 @@ checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -3731,12 +3769,12 @@ checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca" [[package]] name = "tempfile" -version = "3.25.0" +version = "3.27.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0136791f7c95b1f6dd99f9cc786b91bb81c3800b639b3478e561ddb7be95e5f1" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" dependencies = [ "fastrand", - "getrandom 0.4.1", + "getrandom 0.4.3", "once_cell", "rustix", "windows-sys 0.61.2", @@ -3759,18 +3797,7 @@ checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", -] - -[[package]] -name = "thrift" -version = "0.17.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e54bc85fc7faa8bc175c4bab5b92ba8d9a3ce893d0e9f42cc455c8ab16a9e09" -dependencies = [ - "byteorder", - "integer-encoding", - "ordered-float", + "syn 2.0.118", ] [[package]] @@ -3784,9 +3811,9 @@ dependencies = [ [[package]] name = "tinystr" -version = "0.8.2" +version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42d3e9c45c09de15d06dd8acf5f4e0e399e85927b7f00711024eb7ae10fa4869" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" dependencies = [ "displaydoc", "zerovec", @@ -3794,9 +3821,9 @@ dependencies = [ [[package]] name = "tinyvec" -version = "1.10.0" +version = "1.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bfa5fdc3bce6191a1dbc8c02d5c8bffcf557bafa17c124c5264a458f1b0613fa" +checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3" dependencies = [ "tinyvec_macros", ] @@ -3809,9 +3836,9 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" [[package]] name = "tokio" -version = "1.49.0" +version = "1.52.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72a2903cd7736441aac9df9d7688bd0ce48edccaadf181c3b90be801e81d3d86" +checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe" dependencies = [ "bytes", "libc", @@ -3824,13 +3851,13 @@ dependencies = [ [[package]] name = "tokio-macros" -version = "2.6.0" +version = "2.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "af407857209536a95c8e56f8231ef2c2e2aff839b22e07a1ffcbc617e9db9fa5" +checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -3868,6 +3895,36 @@ dependencies = [ "tokio", ] +[[package]] +name = "toml_datetime" +version = "1.1.1+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3165f65f62e28e0115a00b2ebdd37eb6f3b641855f9d636d3cd4103767159ad7" +dependencies = [ + "serde_core", +] + +[[package]] +name = "toml_edit" +version = "0.25.12+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2153edc6955a6c354fad8f5efd38b6a8769bdccf9fe50f8e1329f81b0baa5d7" +dependencies = [ + "indexmap", + "toml_datetime", + "toml_parser", + "winnow", +] + +[[package]] +name = "toml_parser" +version = "1.1.2+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2abe9b86193656635d2411dc43050282ca48aa31c2451210f4202550afb7526" +dependencies = [ + "winnow", +] + [[package]] name = "tower" version = "0.5.3" @@ -3885,20 +3942,20 @@ dependencies = [ [[package]] name = "tower-http" -version = "0.6.8" +version = "0.6.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4e6559d53cc268e5031cd8429d05415bc4cb4aefc4aa5d6cc35fbf5b924a1f8" +checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840" dependencies = [ "bitflags", "bytes", "futures-util", "http", "http-body", - "iri-string", "pin-project-lite", "tower", "tower-layer", "tower-service", + "url", ] [[package]] @@ -3932,7 +3989,7 @@ checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -3950,44 +4007,20 @@ version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" -[[package]] -name = "tstr" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f8e0294f14baae476d0dd0a2d780b2e24d66e349a9de876f5126777a37bdba7" -dependencies = [ - "tstr_proc_macros", -] - -[[package]] -name = "tstr_proc_macros" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e78122066b0cb818b8afd08f7ed22f7fdbc3e90815035726f0840d0d26c0747a" - [[package]] name = "twox-hash" version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c" - -[[package]] -name = "typed-arena" -version = "2.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6af6ae20167a9ece4bcb41af5b80f8a1f1df981f6391189ce00fd257af04126a" +dependencies = [ + "rand 0.9.4", +] [[package]] name = "typenum" -version = "1.19.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb" - -[[package]] -name = "typewit" -version = "1.14.2" +version = "1.20.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8c1ae7cc0fdb8b842d65d127cb981574b0d2b249b74d1c7a2986863dc134f71" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" [[package]] name = "typify" @@ -4014,7 +4047,7 @@ dependencies = [ "semver", "serde", "serde_json", - "syn 2.0.117", + "syn 2.0.118", "thiserror", "unicode-ident", ] @@ -4032,7 +4065,7 @@ dependencies = [ "serde", "serde_json", "serde_tokenstream", - "syn 2.0.117", + "syn 2.0.118", "typify-impl", ] @@ -4044,9 +4077,9 @@ checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" [[package]] name = "unicode-segmentation" -version = "1.12.0" +version = "1.13.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f6ccf251212114b54433ec949fd6a7841275f9ada20dddd2f29e9ceea4501493" +checksum = "c6f5d3c3b1bf09027a88a6bc961fc00497d651009560b5463668dc81b0fa87a8" [[package]] name = "unicode-width" @@ -4054,12 +4087,6 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" -[[package]] -name = "unicode-xid" -version = "0.2.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" - [[package]] name = "unsafe-libyaml" version = "0.2.11" @@ -4092,13 +4119,12 @@ checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" [[package]] name = "uuid" -version = "1.21.0" +version = "1.23.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b672338555252d43fd2240c714dc444b8c6fb0a5c5335e65a07bba7742735ddb" +checksum = "144d6b123cef80b301b8f72a9e2ca4370ddec21950d0a103dd22c437006d2db7" dependencies = [ - "getrandom 0.4.1", + "getrandom 0.4.3", "js-sys", - "serde_core", "wasm-bindgen", ] @@ -4135,27 +4161,18 @@ checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" [[package]] name = "wasip2" -version = "1.0.2+wasi-0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9517f9239f02c069db75e65f174b3da828fe5f5b945c4dd26bd25d89c03ebcf5" -dependencies = [ - "wit-bindgen", -] - -[[package]] -name = "wasip3" -version = "0.4.0+wasi-0.3.0-rc-2026-01-06" +version = "1.0.4+wasi-0.2.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" +checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" dependencies = [ "wit-bindgen", ] [[package]] name = "wasm-bindgen" -version = "0.2.108" +version = "0.2.125" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "64024a30ec1e37399cf85a7ffefebdb72205ca1c972291c51512360d90bd8566" +checksum = "8ddb3f79143bced6de84270411622a2699cee572fc0875aeaf1e7867cf9fca1a" dependencies = [ "cfg-if", "once_cell", @@ -4166,23 +4183,19 @@ dependencies = [ [[package]] name = "wasm-bindgen-futures" -version = "0.4.58" +version = "0.4.75" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70a6e77fd0ae8029c9ea0063f87c46fde723e7d887703d74ad2616d792e51e6f" +checksum = "503b14d284f2c8dac03b819967e155ea753f573586193b2b2c95990cb5d69280" dependencies = [ - "cfg-if", - "futures-util", "js-sys", - "once_cell", "wasm-bindgen", - "web-sys", ] [[package]] name = "wasm-bindgen-macro" -version = "0.2.108" +version = "0.2.125" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "008b239d9c740232e71bd39e8ef6429d27097518b6b30bdf9086833bd5b6d608" +checksum = "4e21a184b13fb19e157296e2c46056aec9092264fab83e4ba59e68c61b323c3d" dependencies = [ "quote", "wasm-bindgen-macro-support", @@ -4190,48 +4203,26 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro-support" -version = "0.2.108" +version = "0.2.125" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5256bae2d58f54820e6490f9839c49780dff84c65aeab9e772f15d5f0e913a55" +checksum = "fecefd9c35bd935a20fc3fc344b5f29138961e4f47fb03297d88f2587afb5ebd" dependencies = [ "bumpalo", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-shared" -version = "0.2.108" +version = "0.2.125" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1f01b580c9ac74c8d8f0c0e4afb04eeef2acf145458e52c03845ee9cd23e3d12" +checksum = "23939e44bb9a5d7576fa2b563dc2e136628f1224e88a8deed09e04858b77871f" dependencies = [ "unicode-ident", ] -[[package]] -name = "wasm-encoder" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" -dependencies = [ - "leb128fmt", - "wasmparser", -] - -[[package]] -name = "wasm-metadata" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" -dependencies = [ - "anyhow", - "indexmap", - "wasm-encoder", - "wasmparser", -] - [[package]] name = "wasm-streams" version = "0.4.2" @@ -4245,23 +4236,11 @@ dependencies = [ "web-sys", ] -[[package]] -name = "wasmparser" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" -dependencies = [ - "bitflags", - "hashbrown 0.15.5", - "indexmap", - "semver", -] - [[package]] name = "web-sys" -version = "0.3.85" +version = "0.3.102" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "312e32e551d92129218ea9a2452120f4aabc03529ef03e4d0d82fb2780608598" +checksum = "a6430a72df5eb332242960fe84b3002a241163998241eb596d4f739b9757061d" dependencies = [ "js-sys", "wasm-bindgen", @@ -4277,22 +4256,6 @@ dependencies = [ "wasm-bindgen", ] -[[package]] -name = "winapi" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" -dependencies = [ - "winapi-i686-pc-windows-gnu", - "winapi-x86_64-pc-windows-gnu", -] - -[[package]] -name = "winapi-i686-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" - [[package]] name = "winapi-util" version = "0.1.11" @@ -4302,12 +4265,6 @@ dependencies = [ "windows-sys 0.61.2", ] -[[package]] -name = "winapi-x86_64-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" - [[package]] name = "windows-core" version = "0.62.2" @@ -4329,7 +4286,7 @@ checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -4340,7 +4297,7 @@ checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -4376,15 +4333,6 @@ dependencies = [ "windows-targets 0.52.6", ] -[[package]] -name = "windows-sys" -version = "0.59.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b" -dependencies = [ - "windows-targets 0.52.6", -] - [[package]] name = "windows-sys" version = "0.60.2" @@ -4533,104 +4481,31 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" [[package]] -name = "wit-bindgen" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" -dependencies = [ - "wit-bindgen-rust-macro", -] - -[[package]] -name = "wit-bindgen-core" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" -dependencies = [ - "anyhow", - "heck", - "wit-parser", -] - -[[package]] -name = "wit-bindgen-rust" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" -dependencies = [ - "anyhow", - "heck", - "indexmap", - "prettyplease", - "syn 2.0.117", - "wasm-metadata", - "wit-bindgen-core", - "wit-component", -] - -[[package]] -name = "wit-bindgen-rust-macro" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" -dependencies = [ - "anyhow", - "prettyplease", - "proc-macro2", - "quote", - "syn 2.0.117", - "wit-bindgen-core", - "wit-bindgen-rust", -] - -[[package]] -name = "wit-component" -version = "0.244.0" +name = "winnow" +version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" +checksum = "0592e1c9d151f854e6fd382574c3a0855250e1d9b2f99d9281c6e6391af352f1" dependencies = [ - "anyhow", - "bitflags", - "indexmap", - "log", - "serde", - "serde_derive", - "serde_json", - "wasm-encoder", - "wasm-metadata", - "wasmparser", - "wit-parser", + "memchr", ] [[package]] -name = "wit-parser" -version = "0.244.0" +name = "wit-bindgen" +version = "0.57.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" -dependencies = [ - "anyhow", - "id-arena", - "indexmap", - "log", - "semver", - "serde", - "serde_derive", - "serde_json", - "unicode-xid", - "wasmparser", -] +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" [[package]] name = "writeable" -version = "0.6.2" +version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9edde0db4769d2dc68579893f2306b26c6ecfbe0ef499b013d731b7b9247e0b9" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" [[package]] name = "yoke" -version = "0.8.1" +version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72d6e5c6afb84d73944e5cedb052c4680d5657337201555f9f2a16b7406d4954" +checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5" dependencies = [ "stable_deref_trait", "yoke-derive", @@ -4639,68 +4514,68 @@ dependencies = [ [[package]] name = "yoke-derive" -version = "0.8.1" +version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b659052874eb698efe5b9e8cf382204678a0086ebf46982b79d6ca3182927e5d" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", "synstructure", ] [[package]] name = "zerocopy" -version = "0.8.39" +version = "0.8.52" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db6d35d663eadb6c932438e763b262fe1a70987f9ae936e60158176d710cae4a" +checksum = "ce1022995ff5ff5d841ad7d994facc23098cd40152f2c1d11cd607c6f530653f" dependencies = [ "zerocopy-derive", ] [[package]] name = "zerocopy-derive" -version = "0.8.39" +version = "0.8.52" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4122cd3169e94605190e77839c9a40d40ed048d305bfdc146e7df40ab0f3e517" +checksum = "1ae7f38b72ec2a254e2b87ef277cf2cd4fb97cbebf944faa6f33354da0867930" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] name = "zerofrom" -version = "0.1.6" +version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "50cc42e0333e05660c3587f3bf9d0478688e15d870fab3346451ce7f8c9fbea5" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" dependencies = [ "zerofrom-derive", ] [[package]] name = "zerofrom-derive" -version = "0.1.6" +version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d71e5d6e06ab090c67b5e44993ec16b72dcbaabc526db883a360057678b48502" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", "synstructure", ] [[package]] name = "zeroize" -version = "1.8.2" +version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0" +checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" [[package]] name = "zerotrie" -version = "0.2.3" +version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2a59c17a5562d507e4b54960e8569ebee33bee890c70aa3fe7b97e85a9fd7851" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" dependencies = [ "displaydoc", "yoke", @@ -4709,9 +4584,9 @@ dependencies = [ [[package]] name = "zerovec" -version = "0.11.5" +version = "0.11.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6c28719294829477f525be0186d13efa9a3c602f7ec202ca9e353d310fb9a002" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" dependencies = [ "yoke", "zerofrom", @@ -4720,20 +4595,20 @@ dependencies = [ [[package]] name = "zerovec-derive" -version = "0.11.2" +version = "0.11.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eadce39539ca5cb3985590102671f2567e659fca9666581ad3411d59207951f3" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] name = "zlib-rs" -version = "0.6.2" +version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c745c48e1007337ed136dc99df34128b9faa6ed542d80a1c673cf55a6d7236c8" +checksum = "3be3d40e40a133f9c916ee3f9f4fa2d9d63435b5fbe1bfc6d9dae0aa0ada1513" [[package]] name = "zmij" diff --git a/Cargo.toml b/Cargo.toml index b584470d6..a9e15d7e9 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -15,9 +15,8 @@ # specific language governing permissions and limitations # under the License. -[package] -name = "datafusion-python" -version = "52.0.0" +[workspace.package] +version = "54.0.0" homepage = "https://datafusion.apache.org/python" repository = "https://github.com/apache/datafusion-python" authors = ["Apache DataFusion "] @@ -26,76 +25,60 @@ readme = "README.md" license = "Apache-2.0" edition = "2024" rust-version = "1.88" -include = [ - "/src", - "/datafusion", - "/LICENSE.txt", - "build.rs", - "pyproject.toml", - "Cargo.toml", - "Cargo.lock", -] -[features] -default = ["mimalloc"] -protoc = ["datafusion-substrait/protoc"] -substrait = ["dep:datafusion-substrait"] +[workspace] +members = ["crates/core", "crates/util", "examples/datafusion-ffi-example"] +resolver = "3" -[dependencies] -tokio = { version = "1.49", features = [ - "macros", - "rt", - "rt-multi-thread", - "sync", -] } -pyo3 = { version = "0.28", features = [ - "extension-module", - "abi3", - "abi3-py310", -] } -pyo3-async-runtimes = { version = "0.28", features = ["tokio-runtime"] } +[workspace.dependencies] +tokio = { version = "1.52" } +pyo3 = { version = "0.29" } +pyo3-async-runtimes = { version = "0.29" } pyo3-log = "0.13.3" -arrow = { version = "58", features = ["pyarrow"] } -arrow-select = { version = "58" } -datafusion = { version = "53", features = ["avro", "unicode_expressions"] } -datafusion-substrait = { version = "53", optional = true } -datafusion-proto = { version = "53" } -datafusion-ffi = { version = "53" } -prost = "0.14.3" # keep in line with `datafusion-substrait` +chrono = { version = "0.4", default-features = false } +arrow = { version = "59" } +arrow-array = { version = "59" } +arrow-schema = { version = "59" } +arrow-select = { version = "59" } +datafusion = { version = "55.0.0" } +datafusion-substrait = { version = "55.0.0" } +datafusion-proto = { version = "55.0.0" } +datafusion-ffi = { version = "55.0.0" } +datafusion-catalog = { version = "55.0.0", default-features = false } +datafusion-common = { version = "55.0.0", default-features = false } +datafusion-functions-aggregate = { version = "55.0.0" } +datafusion-functions-window = { version = "55.0.0" } +datafusion-spark = { version = "55.0.0" } +datafusion-expr = { version = "55.0.0" } +prost = "0.14.3" serde_json = "1" -uuid = { version = "1.21", features = ["v4"] } -mimalloc = { version = "0.1", optional = true, default-features = false, features = [ - "local_dynamic_tls", -] } +uuid = { version = "1.23" } +mimalloc = { version = "0.1", default-features = false } async-trait = "0.1.89" futures = "0.3" cstr = "0.2" -object_store = { version = "0.13.1", features = [ - "aws", - "gcp", - "azure", - "http", -] } +object_store = { version = "0.13.1" } url = "2" log = "0.4.29" parking_lot = "0.12" - -[build-dependencies] -prost-types = "0.14.3" # keep in line with `datafusion-substrait` -pyo3-build-config = "0.28" - -[lib] -name = "datafusion_python" -crate-type = ["cdylib", "rlib"] +prost-types = "0.14.3" # keep in line with `datafusion-substrait` +pyo3-build-config = "0.29" +datafusion-python-util = { path = "crates/util", version = "54.0.0" } [profile.release] -lto = true -codegen-units = 1 +lto = "thin" +codegen-units = 2 # We cannot publish to crates.io with any patches in the below section. Developers # must remove any entries in this section before creating a release candidate. [patch.crates-io] -datafusion = { git = "https://github.com/apache/datafusion.git", rev = "35749607f585b3bf25b66b7d2289c56c18d03e4f" } -datafusion-substrait = { git = "https://github.com/apache/datafusion.git", rev = "35749607f585b3bf25b66b7d2289c56c18d03e4f" } -datafusion-proto = { git = "https://github.com/apache/datafusion.git", rev = "35749607f585b3bf25b66b7d2289c56c18d03e4f" } -datafusion-ffi = { git = "https://github.com/apache/datafusion.git", rev = "35749607f585b3bf25b66b7d2289c56c18d03e4f" } +datafusion = { git = "https://github.com/apache/datafusion", rev = "55.0.0-rc3" } +datafusion-substrait = { git = "https://github.com/apache/datafusion", rev = "55.0.0-rc3" } +datafusion-proto = { git = "https://github.com/apache/datafusion", rev = "55.0.0-rc3" } +datafusion-ffi = { git = "https://github.com/apache/datafusion", rev = "55.0.0-rc3" } +datafusion-catalog = { git = "https://github.com/apache/datafusion", rev = "55.0.0-rc3" } +datafusion-common = { git = "https://github.com/apache/datafusion", rev = "55.0.0-rc3" } +datafusion-functions-aggregate = { git = "https://github.com/apache/datafusion", rev = "55.0.0-rc3" } +datafusion-functions-window = { git = "https://github.com/apache/datafusion", rev = "55.0.0-rc3" } +datafusion-spark = { git = "https://github.com/apache/datafusion", rev = "55.0.0-rc3" } +datafusion-expr = { git = "https://github.com/apache/datafusion", rev = "55.0.0-rc3" } diff --git a/README.md b/README.md index 810ac8710..f6ee662d0 100644 --- a/README.md +++ b/README.md @@ -215,6 +215,22 @@ You can verify the installation by running: '0.6.0' ``` +## Using DataFusion with AI coding assistants + +This project ships a [`SKILL.md`](skills/datafusion_python/SKILL.md) that +teaches AI coding assistants how to write idiomatic DataFusion Python. It follows the +[Agent Skills](https://agentskills.io) open standard. + +**Preferred:** `npx skills add apache/datafusion-python` — installs the skill in +Claude Code, Cursor, Windsurf, Cline, Codex, Copilot, Gemini CLI, and other +supported agents. + +**Manual:** paste this line into your project's `AGENTS.md` / `CLAUDE.md`: + +``` +For DataFusion Python code, see https://github.com/apache/datafusion-python/blob/main/skills/datafusion_python/SKILL.md +``` + ## How to develop This assumes that you have rust and cargo installed. We use the workflow recommended by [pyo3](https://github.com/PyO3/pyo3) and [maturin](https://github.com/PyO3/maturin). The Maturin tools used in this workflow can be installed either via `uv` or `pip`. Both approaches should offer the same experience. It is recommended to use `uv` since it has significant performance improvements @@ -275,7 +291,16 @@ needing to activate the virtual environment: ```bash uv run --no-project maturin develop --uv -uv run --no-project pytest . +uv run --no-project pytest +``` + +To run the FFI tests within the examples folder, after you have built +`datafusion-python` with the previous commands: + +```bash +cd examples/datafusion-ffi-example +uv run --no-project maturin develop --uv +uv run --no-project pytest python/tests/_test_*py ``` ### Running & Installing pre-commit hooks @@ -303,6 +328,33 @@ There are scripts in `ci/scripts` for running Rust and Python linters. ./ci/scripts/rust_toml_fmt.sh ``` +## Checking Upstream DataFusion Coverage + +This project includes an [AI agent skill](.ai/skills/check-upstream/SKILL.md) for auditing which +features from the upstream Apache DataFusion Rust library are not yet exposed in these Python +bindings. This is useful when adding missing functions, auditing API coverage, or ensuring parity +with upstream. + +The skill accepts an optional area argument: + +``` +scalar functions +aggregate functions +window functions +dataframe +session context +ffi types +all +``` + +If no argument is provided, it defaults to checking all areas. The skill will fetch the upstream +DataFusion documentation, compare it against the functions and methods exposed in this project, and +produce a coverage report listing what is currently exposed and what is missing. + +The skill definition lives in `.ai/skills/check-upstream/SKILL.md` and follows the +[Agent Skills](https://agentskills.io) open standard. It can be used by any AI coding agent that +supports skill discovery, or followed manually. + ## How to update dependencies To change test dependencies, change the `pyproject.toml` and run diff --git a/conftest.py b/conftest.py index 1c89f92bc..0c9410636 100644 --- a/conftest.py +++ b/conftest.py @@ -19,7 +19,10 @@ import datafusion as dfn import numpy as np +import pyarrow as pa import pytest +from datafusion import col, lit +from datafusion import functions as F @pytest.fixture(autouse=True) @@ -27,3 +30,7 @@ def _doctest_namespace(doctest_namespace: dict) -> None: """Add common imports to the doctest namespace.""" doctest_namespace["dfn"] = dfn doctest_namespace["np"] = np + doctest_namespace["pa"] = pa + doctest_namespace["col"] = col + doctest_namespace["lit"] = lit + doctest_namespace["F"] = F diff --git a/crates/core/Cargo.toml b/crates/core/Cargo.toml new file mode 100644 index 000000000..c5f1e0167 --- /dev/null +++ b/crates/core/Cargo.toml @@ -0,0 +1,89 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[package] +name = "datafusion-python" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +description.workspace = true +homepage.workspace = true +repository.workspace = true +include = [ + "src", + "../LICENSE.txt", + "build.rs", + "../pyproject.toml", + "Cargo.toml", + "../Cargo.lock", +] + +[dependencies] +tokio = { workspace = true, features = [ + "macros", + "rt", + "rt-multi-thread", + "sync", +] } +pyo3 = { workspace = true, features = [ + "extension-module", + "generate-import-lib", +] } +pyo3-async-runtimes = { workspace = true, features = ["tokio-runtime"] } +pyo3-log = { workspace = true } +chrono = { workspace = true } +arrow = { workspace = true, features = ["pyarrow"] } +arrow-select = { workspace = true } +datafusion = { workspace = true, features = ["avro", "unicode_expressions"] } +datafusion-substrait = { workspace = true, optional = true } +datafusion-proto = { workspace = true } +datafusion-ffi = { workspace = true } +datafusion-spark = { workspace = true, features = ["core"] } +prost = { workspace = true } # keep in line with `datafusion-substrait` +serde_json = { workspace = true } +uuid = { workspace = true, features = ["v4"] } +mimalloc = { workspace = true, optional = true, features = [ + "local_dynamic_tls", + # Pin to mimalloc v2 until apache/datafusion-python#1607 resolves. + "v2", +] } +async-trait = { workspace = true } +futures = { workspace = true } +cstr = { workspace = true } +object_store = { workspace = true, features = ["aws", "gcp", "azure", "http"] } +url = { workspace = true } +log = { workspace = true } +parking_lot = { workspace = true } +datafusion-python-util = { workspace = true } + +[build-dependencies] +prost-types = { workspace = true } +pyo3-build-config = { workspace = true } + +[features] +default = ["mimalloc", "abi3"] +# Stable ABI build — single wheel covers Python 3.10..3.14 (GIL builds only). +# Mutually exclusive with free-threaded interpreters (cp313t / cp314t); the +# free-threaded wheel build must pass --no-default-features. +abi3 = ["pyo3/abi3", "pyo3/abi3-py310"] +protoc = ["datafusion-substrait/protoc"] +substrait = ["dep:datafusion-substrait"] + +[lib] +name = "datafusion_python" +crate-type = ["cdylib", "rlib"] diff --git a/build.rs b/crates/core/build.rs similarity index 100% rename from build.rs rename to crates/core/build.rs diff --git a/crates/core/src/analyzer.rs b/crates/core/src/analyzer.rs new file mode 100644 index 000000000..3a77e08a3 --- /dev/null +++ b/crates/core/src/analyzer.rs @@ -0,0 +1,59 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Analyzer rules layered on top of DataFusion's defaults. + +use datafusion::common::Result; +use datafusion::common::config::ConfigOptions; +use datafusion::logical_expr::LogicalPlan; +use datafusion::optimizer::AnalyzerRule; + +/// Resolve [`LambdaVariable`] references into bound lambda parameters. +/// +/// DataFusion's SQL planner resolves lambda variables inline as it plans a +/// higher-order function call, so SQL-built plans never carry unresolved +/// variables. Plans assembled programmatically through the Python expression +/// builder (e.g. `array_transform(col("xs"), lambda_(["v"], lambda_var("v")))`) +/// do carry them, and nothing in the default analyzer resolves them. This rule +/// runs [`LogicalPlan::resolve_lambda_variables`] so both construction paths +/// reach the optimizer with bound lambdas. +/// +/// [`LambdaVariable`]: datafusion::logical_expr::expr::LambdaVariable +#[derive(Debug)] +pub struct ResolveLambdaVariables {} + +impl ResolveLambdaVariables { + pub fn new() -> Self { + Self {} + } +} + +impl Default for ResolveLambdaVariables { + fn default() -> Self { + Self::new() + } +} + +impl AnalyzerRule for ResolveLambdaVariables { + fn analyze(&self, plan: LogicalPlan, _config: &ConfigOptions) -> Result { + plan.resolve_lambda_variables().map(|t| t.data) + } + + fn name(&self) -> &str { + "resolve_lambda_variables" + } +} diff --git a/src/array.rs b/crates/core/src/array.rs similarity index 88% rename from src/array.rs rename to crates/core/src/array.rs index 1ff08dfb2..dfe963183 100644 --- a/src/array.rs +++ b/crates/core/src/array.rs @@ -22,13 +22,11 @@ use arrow::array::{Array, ArrayRef}; use arrow::datatypes::{Field, FieldRef}; use arrow::ffi::{FFI_ArrowArray, FFI_ArrowSchema}; use arrow::pyarrow::ToPyArrow; -use pyo3::ffi::c_str; use pyo3::prelude::{PyAnyMethods, PyCapsuleMethods}; use pyo3::types::PyCapsule; use pyo3::{Bound, PyAny, PyResult, Python, pyclass, pymethods}; use crate::errors::PyDataFusionResult; -use crate::utils::validate_pycapsule; /// A Python object which implements the Arrow PyCapsule for importing /// into other libraries. @@ -53,10 +51,8 @@ impl PyArrowArrayExportable { requested_schema: Option>, ) -> PyDataFusionResult<(Bound<'py, PyCapsule>, Bound<'py, PyCapsule>)> { let field = if let Some(schema_capsule) = requested_schema { - validate_pycapsule(&schema_capsule, "arrow_schema")?; - let data: NonNull = schema_capsule - .pointer_checked(Some(c_str!("arrow_schema")))? + .pointer_checked(Some(c"arrow_schema"))? .cast(); let schema_ptr = unsafe { data.as_ref() }; let desired_field = Field::try_from(schema_ptr)?; @@ -67,10 +63,10 @@ impl PyArrowArrayExportable { }; let ffi_schema = FFI_ArrowSchema::try_from(&field)?; - let schema_capsule = PyCapsule::new(py, ffi_schema, Some(cr"arrow_schema".into()))?; + let schema_capsule = PyCapsule::new_with_value(py, ffi_schema, cr"arrow_schema")?; let ffi_array = FFI_ArrowArray::new(&self.array.to_data()); - let array_capsule = PyCapsule::new(py, ffi_array, Some(cr"arrow_array".into()))?; + let array_capsule = PyCapsule::new_with_value(py, ffi_array, cr"arrow_array")?; Ok((schema_capsule, array_capsule)) } diff --git a/src/catalog.rs b/crates/core/src/catalog.rs similarity index 93% rename from src/catalog.rs rename to crates/core/src/catalog.rs index 43325c30d..8ad49b098 100644 --- a/src/catalog.rs +++ b/crates/core/src/catalog.rs @@ -15,7 +15,6 @@ // specific language governing permissions and limitations // under the License. -use std::any::Any; use std::collections::HashSet; use std::ptr::NonNull; use std::sync::Arc; @@ -30,19 +29,18 @@ use datafusion::datasource::TableProvider; use datafusion_ffi::catalog_provider::FFI_CatalogProvider; use datafusion_ffi::proto::logical_extension_codec::FFI_LogicalExtensionCodec; use datafusion_ffi::schema_provider::FFI_SchemaProvider; +use datafusion_python_util::{ + create_logical_extension_capsule, ffi_logical_codec_from_pycapsule, wait_for_future, +}; use pyo3::IntoPyObjectExt; use pyo3::exceptions::PyKeyError; -use pyo3::ffi::c_str; use pyo3::prelude::*; use pyo3::types::PyCapsule; +use crate::context::PySessionContext; use crate::dataset::Dataset; use crate::errors::{PyDataFusionError, PyDataFusionResult, py_datafusion_err, to_datafusion_err}; use crate::table::PyTable; -use crate::utils::{ - create_logical_extension_capsule, extract_logical_extension_codec, validate_pycapsule, - wait_for_future, -}; #[pyclass( from_py_object, @@ -144,15 +142,12 @@ impl PyCatalogList { "Schema with name {name} doesn't exist." )))?; - Python::attach(|py| { - match catalog - .as_any() - .downcast_ref::() - { + Python::attach( + |py| match catalog.downcast_ref::() { Some(wrapped_catalog) => Ok(wrapped_catalog.catalog_provider.clone_ref(py)), None => PyCatalog::new_from_parts(catalog, self.codec.clone()).into_py_any(py), - } - }) + }, + ) } pub fn register_catalog(&self, name: &str, catalog_provider: Bound<'_, PyAny>) -> PyResult<()> { @@ -202,15 +197,12 @@ impl PyCatalog { "Schema with name {name} doesn't exist." )))?; - Python::attach(|py| { - match schema - .as_any() - .downcast_ref::() - { + Python::attach( + |py| match schema.downcast_ref::() { Some(wrapped_schema) => Ok(wrapped_schema.schema_provider.clone_ref(py)), None => PySchema::new_from_parts(schema, self.codec.clone()).into_py_any(py), - } - }) + }, + ) } pub fn register_schema(&self, name: &str, schema_provider: Bound<'_, PyAny>) -> PyResult<()> { @@ -357,10 +349,6 @@ impl SchemaProvider for RustWrappedPySchemaProvider { self.owner_name.as_deref() } - fn as_any(&self) -> &dyn Any { - self - } - fn table_names(&self) -> Vec { Python::attach(|py| { let provider = self.schema_provider.bind(py); @@ -466,10 +454,6 @@ impl RustWrappedPyCatalogProvider { #[async_trait] impl CatalogProvider for RustWrappedPyCatalogProvider { - fn as_any(&self) -> &dyn Any { - self - } - fn schema_names(&self) -> Vec { Python::attach(|py| { let provider = self.catalog_provider.bind(py); @@ -497,10 +481,7 @@ impl CatalogProvider for RustWrappedPyCatalogProvider { schema: Arc, ) -> datafusion::common::Result>> { Python::attach(|py| { - let py_schema = match schema - .as_any() - .downcast_ref::() - { + let py_schema = match schema.downcast_ref::() { Some(wrapped_schema) => wrapped_schema.schema_provider.as_any(), None => &PySchema::new_from_parts(schema, self.codec.clone()) .into_py_any(py) @@ -574,10 +555,6 @@ impl RustWrappedPyCatalogProviderList { #[async_trait] impl CatalogProviderList for RustWrappedPyCatalogProviderList { - fn as_any(&self) -> &dyn Any { - self - } - fn catalog_names(&self) -> Vec { Python::attach(|py| { let provider = self.catalog_provider_list.bind(py); @@ -605,10 +582,7 @@ impl CatalogProviderList for RustWrappedPyCatalogProviderList { catalog: Arc, ) -> Option> { Python::attach(|py| { - let py_catalog = match catalog - .as_any() - .downcast_ref::() - { + let py_catalog = match catalog.downcast_ref::() { Some(wrapped_schema) => wrapped_schema.catalog_provider.as_any().clone_ref(py), None => { match PyCatalog::new_from_parts(catalog, self.codec.clone()).into_py_any(py) { @@ -658,13 +632,12 @@ fn extract_catalog_provider_from_pyobj( } let provider = if let Ok(capsule) = catalog_provider.cast::() { - validate_pycapsule(capsule, "datafusion_catalog_provider")?; let data: NonNull = capsule - .pointer_checked(Some(c_str!("datafusion_catalog_provider")))? + .pointer_checked(Some(c"datafusion_catalog_provider"))? .cast(); let provider = unsafe { data.as_ref() }; - let provider: Arc = provider.into(); - provider as Arc + let provider: Arc = provider.into(); + provider } else { match catalog_provider.extract::() { Ok(py_catalog) => py_catalog.catalog, @@ -691,14 +664,12 @@ fn extract_schema_provider_from_pyobj( } let provider = if let Ok(capsule) = schema_provider.cast::() { - validate_pycapsule(capsule, "datafusion_schema_provider")?; - let data: NonNull = capsule - .pointer_checked(Some(c_str!("datafusion_schema_provider")))? + .pointer_checked(Some(c"datafusion_schema_provider"))? .cast(); let provider = unsafe { data.as_ref() }; - let provider: Arc = provider.into(); - provider as Arc + let provider: Arc = provider.into(); + provider } else { match schema_provider.extract::() { Ok(py_schema) => py_schema.schema, @@ -710,6 +681,17 @@ fn extract_schema_provider_from_pyobj( Ok(provider) } +fn extract_logical_extension_codec( + py: Python, + obj: Option>, +) -> PyResult> { + let obj = match obj { + Some(obj) => obj, + None => PySessionContext::global_ctx()?.into_bound_py_any(py)?, + }; + ffi_logical_codec_from_pycapsule(obj).map(Arc::new) +} + pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; diff --git a/crates/core/src/codec.rs b/crates/core/src/codec.rs new file mode 100644 index 000000000..26853e69f --- /dev/null +++ b/crates/core/src/codec.rs @@ -0,0 +1,1164 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Python-aware extension codecs. +//! +//! Datafusion-python plans can carry references to Python-defined +//! objects that the upstream protobuf codecs do not know how to +//! serialize: pure-Python scalar / aggregate / window UDFs, Python +//! query-planning extensions, and so on. Their state lives inside +//! `Py` callables and closures rather than being recoverable +//! from a name in the receiver's function registry. To ship a plan +//! across a process boundary (pickle, `multiprocessing`, Ray actor, +//! `datafusion-distributed`, etc.) those payloads have to be encoded +//! into the proto wire format itself. +//! +//! [`PythonLogicalCodec`] is the [`LogicalExtensionCodec`] that +//! datafusion-python parks on every `SessionContext`. It wraps a +//! user-supplied (or default) inner codec and adds Python-aware +//! in-band encoding on top: when the encoder sees a Python-defined +//! UDF, the codec cloudpickles the callable + signature into the +//! `fun_definition` proto field; when the decoder sees a payload it +//! produced, it reconstructs the UDF from the bytes alone — no +//! pre-registration on the receiver. UDFs the codec does not +//! recognise are delegated to `inner`, which is typically +//! `DefaultLogicalExtensionCodec` but may be a downstream-supplied +//! FFI codec installed via +//! `SessionContext.with_logical_extension_codec(...)`. +//! +//! [`PythonPhysicalCodec`] is the symmetric wrapper around +//! [`PhysicalExtensionCodec`]. Logical and physical layers each have +//! a `try_encode_udf` / `try_decode_udf` pair, so a `ScalarUDF` +//! referenced inside a `LogicalPlan`, an `ExecutionPlan`, or a +//! `PhysicalExpr` must encode identically through either layer for +//! plans to survive a serialization round-trip. Both codecs share +//! the same payload framing for that reason. +//! +//! Payloads emitted by these codecs are framed as +//! ` `. +//! The family magic identifies the UDF flavor; the version byte lets +//! the decoder reject too-new or too-old payloads with a clean error +//! instead of falling into an opaque `cloudpickle` tuple-unpack +//! failure when the tuple shape changes; the Python `(major, minor)` +//! bytes catch the cloudpickle-cross-minor-version case and raise an +//! actionable error instead of an opaque `marshal` failure on load +//! (cloudpickle payloads are not portable across Python minor +//! versions). Dispatch precedence on decode: **family match + +//! supported version + matching Python version → `inner` codec → +//! caller's `FunctionRegistry` fallback.** +//! +//! ## Wire-format family registry +//! +//! | Layer + kind | Family prefix | +//! | ----------------------------- | ------------- | +//! | `PythonLogicalCodec` scalar | `DFPYUDF` | +//! | `PythonLogicalCodec` agg | `DFPYUDA` | +//! | `PythonLogicalCodec` window | `DFPYUDW` | +//! | `PythonPhysicalCodec` scalar | `DFPYUDF` | +//! | `PythonPhysicalCodec` agg | `DFPYUDA` | +//! | `PythonPhysicalCodec` window | `DFPYUDW` | +//! | User FFI extension codec | user-chosen | +//! | Default codec | (none) | +//! +//! Current wire-format version is [`WIRE_VERSION_CURRENT`]; supported +//! receive range is `WIRE_VERSION_MIN_SUPPORTED..=WIRE_VERSION_CURRENT`. +//! Bump [`WIRE_VERSION_CURRENT`] whenever the cloudpickle tuple shape +//! changes; raise [`WIRE_VERSION_MIN_SUPPORTED`] when dropping support +//! for an older shape. +//! +//! Downstream FFI codecs should pick non-colliding family prefixes +//! (use a `DF` namespace plus a crate-specific suffix). The codec +//! implementations in this module currently delegate every method to +//! `inner`; the encoder/decoder hooks for each kind are added as the +//! corresponding Python-side type becomes serializable. + +use std::sync::Arc; + +use arrow::datatypes::{DataType, Field, Schema, SchemaRef}; +use arrow::ipc::reader::StreamReader; +use arrow::ipc::writer::StreamWriter; +use datafusion::common::{Result, TableReference}; +use datafusion::datasource::TableProvider; +use datafusion::datasource::file_format::FileFormatFactory; +use datafusion::execution::TaskContext; +use datafusion::logical_expr::{ + AggregateUDF, AggregateUDFImpl, Extension, LogicalPlan, ScalarUDF, ScalarUDFImpl, Signature, + TypeSignature, Volatility, WindowUDF, WindowUDFImpl, +}; +use datafusion::physical_expr::PhysicalExpr; +use datafusion::physical_expr_common::physical_expr::proto_decode::PhysicalExprDecodeCtx; +use datafusion::physical_expr_common::physical_expr::proto_encode::PhysicalExprEncodeCtx; +use datafusion::physical_plan::ExecutionPlan; +use datafusion_proto::logical_plan::{DefaultLogicalExtensionCodec, LogicalExtensionCodec}; +use datafusion_proto::physical_plan::{ + DefaultPhysicalExtensionCodec, PhysicalExtensionCodec, PhysicalProtoConverterExtension, +}; +use pyo3::prelude::*; +use pyo3::sync::PyOnceLock; +use pyo3::types::{PyBytes, PyTuple}; + +use crate::errors::to_datafusion_err; +use crate::udaf::PythonFunctionAggregateUDF; +use crate::udf::PythonFunctionScalarUDF; +use crate::udwf::PythonFunctionWindowUDF; + +// Wire-format framing for inlined Python UDF payloads. +// +// Layout: ` `. +// The family magic identifies the UDF flavor; the version byte lets +// the decoder reject too-new or too-old payloads with a clean error +// instead of falling into an opaque `cloudpickle` tuple-unpack failure +// when the tuple shape changes; the Python `(major, minor)` bytes +// catch the cloudpickle-cross-minor-version case (cloudpickle is not +// portable across Python minor versions) and raise an actionable +// error instead of an opaque `marshal` failure on load. Bump +// [`WIRE_VERSION_CURRENT`] whenever the tuple shape changes; raise +// [`WIRE_VERSION_MIN_SUPPORTED`] when dropping support for an older +// shape. + +/// Family prefix for an inlined Python scalar UDF +/// (cloudpickled tuple of name, callable, input schema, return field, +/// volatility). +pub(crate) const PY_SCALAR_UDF_FAMILY: &[u8] = b"DFPYUDF"; + +/// Family prefix for an inlined Python aggregate UDF +/// (cloudpickled tuple of name, accumulator factory, input schema bytes, +/// return schema bytes (single-field IPC schema), state schema bytes, +/// volatility). +pub(crate) const PY_AGG_UDF_FAMILY: &[u8] = b"DFPYUDA"; + +/// Family prefix for an inlined Python window UDF +/// (cloudpickled tuple of name, evaluator factory, input schema bytes, +/// return schema bytes (single-field IPC schema), volatility). +pub(crate) const PY_WINDOW_UDF_FAMILY: &[u8] = b"DFPYUDW"; + +/// Wire-format version this build emits. +pub(crate) const WIRE_VERSION_CURRENT: u8 = 1; + +/// Oldest wire-format version this build still decodes. Bump when +/// retiring support for an older payload shape. +pub(crate) const WIRE_VERSION_MIN_SUPPORTED: u8 = 1; + +/// Tag `buf` with the framing header for `family` at the current +/// wire-format version, stamping `py_version` as `(major, minor)` +/// bytes. Append-only — the caller writes the cloudpickle payload +/// after. +fn write_wire_header(buf: &mut Vec, family: &[u8], py_version: (u8, u8)) { + buf.extend_from_slice(family); + buf.push(WIRE_VERSION_CURRENT); + buf.push(py_version.0); + buf.push(py_version.1); +} + +/// Inspect the framing on `buf`. +/// +/// * `Ok(None)` — `buf` does not carry `family`. The caller should +/// delegate to its `inner` codec. +/// * `Ok(Some(payload))` — `buf` carries `family` at a version this +/// build accepts and a Python `(major, minor)` matching +/// `expected_py`; `payload` is the cloudpickle blob. +/// * `Err(_)` — `buf` carries `family` but the wire-format version +/// is outside `WIRE_VERSION_MIN_SUPPORTED..=WIRE_VERSION_CURRENT`, +/// or the stamped Python `(major, minor)` does not match +/// `expected_py`. The error names the offending values so an +/// operator can diagnose sender/receiver drift instead of seeing +/// an opaque cloudpickle tuple-unpack or `marshal` failure. +fn strip_wire_header<'a>( + buf: &'a [u8], + family: &[u8], + kind: &str, + expected_py: (u8, u8), +) -> Result> { + if !buf.starts_with(family) { + return Ok(None); + } + let version_idx = family.len(); + let Some(&version) = buf.get(version_idx) else { + return Err(datafusion::error::DataFusionError::Execution(format!( + "Truncated inline Python {kind} payload: missing wire-format version byte" + ))); + }; + if !(WIRE_VERSION_MIN_SUPPORTED..=WIRE_VERSION_CURRENT).contains(&version) { + return Err(datafusion::error::DataFusionError::Execution(format!( + "Inline Python {kind} payload wire-format version v{version}; \ + this build supports v{WIRE_VERSION_MIN_SUPPORTED}..=v{WIRE_VERSION_CURRENT}. \ + Align datafusion-python versions on sender and receiver." + ))); + } + let py_major_idx = version_idx + 1; + let Some(&encoded_major) = buf.get(py_major_idx) else { + return Err(datafusion::error::DataFusionError::Execution(format!( + "Truncated inline Python {kind} payload: missing Python major version byte" + ))); + }; + let py_minor_idx = version_idx + 2; + let Some(&encoded_minor) = buf.get(py_minor_idx) else { + return Err(datafusion::error::DataFusionError::Execution(format!( + "Truncated inline Python {kind} payload: missing Python minor version byte" + ))); + }; + let (current_major, current_minor) = expected_py; + if encoded_major != current_major || encoded_minor != current_minor { + return Err(datafusion::error::DataFusionError::Execution(format!( + "Inline Python {kind} payload was serialized on Python \ + {encoded_major}.{encoded_minor} but this process is running Python \ + {current_major}.{current_minor}. cloudpickle payloads are not portable \ + across Python minor versions. Align Python versions on sender and receiver." + ))); + } + Ok(Some(&buf[py_minor_idx + 1..])) +} + +/// `LogicalExtensionCodec` parked on every `SessionContext`. Holds +/// the Python-aware encoding hooks for logical-layer types +/// (`LogicalPlan`, `Expr`) and delegates everything it does not +/// handle to the composable `inner` codec — typically +/// `DefaultLogicalExtensionCodec`, or a downstream FFI codec +/// installed via `SessionContext.with_logical_extension_codec(...)`. +/// +/// Sitting at the top of the session's logical codec stack means +/// every serializer that reads `session.logical_codec()` automatically +/// picks up Python-aware encoding for free. +#[derive(Debug)] +pub struct PythonLogicalCodec { + inner: Arc, + python_udf_inlining: bool, +} + +impl PythonLogicalCodec { + pub fn new(inner: Arc) -> Self { + Self { + inner, + python_udf_inlining: true, + } + } + + pub fn inner(&self) -> &Arc { + &self.inner + } + + /// Toggle inline encoding of Python UDFs. See + /// `SessionContext.with_python_udf_inlining` (Python) for full + /// behavior and use cases. + /// + /// Security scope: strict mode (`false`) narrows only the codec + /// layer — it stops `Expr::from_bytes` from invoking + /// `cloudpickle.loads` on the inline `DFPY*` payload. It does + /// **not** make `pickle.loads(untrusted_bytes)` safe; treat every + /// `pickle.loads` on untrusted input as unsafe regardless of this + /// setting. See `docs/source/user-guide/io/distributing_work.rst` + /// (Security section) for the full threat model, and Python's + /// [pickle module security warning][1] for why `pickle.loads` is + /// unsafe in general. + /// + /// [1]: https://docs.python.org/3/library/pickle.html#module-pickle + pub fn with_python_udf_inlining(mut self, enabled: bool) -> Self { + self.python_udf_inlining = enabled; + self + } + + pub fn python_udf_inlining(&self) -> bool { + self.python_udf_inlining + } +} + +impl Default for PythonLogicalCodec { + fn default() -> Self { + Self::new(Arc::new(DefaultLogicalExtensionCodec {})) + } +} + +impl LogicalExtensionCodec for PythonLogicalCodec { + fn try_decode( + &self, + buf: &[u8], + inputs: &[LogicalPlan], + ctx: &TaskContext, + ) -> Result { + self.inner.try_decode(buf, inputs, ctx) + } + + fn try_encode(&self, node: &Extension, buf: &mut Vec) -> Result<()> { + self.inner.try_encode(node, buf) + } + + fn try_decode_table_provider( + &self, + buf: &[u8], + table_ref: &TableReference, + schema: SchemaRef, + ctx: &TaskContext, + ) -> Result> { + self.inner + .try_decode_table_provider(buf, table_ref, schema, ctx) + } + + fn try_encode_table_provider( + &self, + table_ref: &TableReference, + node: Arc, + buf: &mut Vec, + ) -> Result<()> { + self.inner.try_encode_table_provider(table_ref, node, buf) + } + + fn try_decode_file_format( + &self, + buf: &[u8], + ctx: &TaskContext, + ) -> Result> { + self.inner.try_decode_file_format(buf, ctx) + } + + fn try_encode_file_format( + &self, + buf: &mut Vec, + node: Arc, + ) -> Result<()> { + self.inner.try_encode_file_format(buf, node) + } + + fn try_encode_udf(&self, node: &ScalarUDF, buf: &mut Vec) -> Result<()> { + if self.python_udf_inlining && try_encode_python_scalar_udf(node, buf)? { + return Ok(()); + } + self.inner.try_encode_udf(node, buf) + } + + fn try_decode_udf(&self, name: &str, buf: &[u8]) -> Result> { + if self.python_udf_inlining { + if let Some(udf) = try_decode_python_scalar_udf(buf)? { + return Ok(udf); + } + } else { + refuse_if_inline(buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", name)?; + } + self.inner.try_decode_udf(name, buf) + } + + fn try_encode_udaf(&self, node: &AggregateUDF, buf: &mut Vec) -> Result<()> { + if self.python_udf_inlining && try_encode_python_udaf(node, buf)? { + return Ok(()); + } + self.inner.try_encode_udaf(node, buf) + } + + fn try_decode_udaf(&self, name: &str, buf: &[u8]) -> Result> { + if self.python_udf_inlining { + if let Some(udaf) = try_decode_python_udaf(buf)? { + return Ok(udaf); + } + } else { + refuse_if_inline(buf, PY_AGG_UDF_FAMILY, "aggregate UDF", name)?; + } + self.inner.try_decode_udaf(name, buf) + } + + fn try_encode_udwf(&self, node: &WindowUDF, buf: &mut Vec) -> Result<()> { + if self.python_udf_inlining && try_encode_python_udwf(node, buf)? { + return Ok(()); + } + self.inner.try_encode_udwf(node, buf) + } + + fn try_decode_udwf(&self, name: &str, buf: &[u8]) -> Result> { + if self.python_udf_inlining { + if let Some(udwf) = try_decode_python_udwf(buf)? { + return Ok(udwf); + } + } else { + refuse_if_inline(buf, PY_WINDOW_UDF_FAMILY, "window UDF", name)?; + } + self.inner.try_decode_udwf(name, buf) + } +} + +/// Strict-mode gate: if `buf` is a well-framed inline payload for +/// `family`, return the strict-refusal error; otherwise return +/// `Ok(())` so the caller can delegate to its `inner` codec. +/// +/// Routing through [`read_framed_payload`] (rather than a bare +/// `starts_with` probe) means malformed inline bytes — wrong +/// wire-format version, mismatched Python version, truncated header — +/// surface *their* diagnostic instead of the strict-mode message. +/// The strict message implies sender intent ("inlining is disabled"), +/// so it should fire only when the bytes really would have decoded. +/// +/// Fast path: short-circuit on the family-magic prefix before +/// acquiring the GIL. Plans with many non-Python UDFs would otherwise +/// pay a GIL acquisition per decode call just to confirm "not a +/// Python UDF". `read_framed_payload` itself rejects buffers that +/// don't start with `family`, so this is purely an optimization. +fn refuse_if_inline(buf: &[u8], family: &[u8], kind: &str, name: &str) -> Result<()> { + if !buf.starts_with(family) { + return Ok(()); + } + Python::attach(|py| match read_framed_payload(py, buf, family, kind)? { + Some(_) => Err(refuse_inline_payload(kind, name)), + None => Ok(()), + }) +} + +/// Build the error returned by a strict codec when it receives an +/// inline Python-UDF payload it has been told not to deserialize. +fn refuse_inline_payload(kind: &str, name: &str) -> datafusion::error::DataFusionError { + // `Execution`, not `Plan`: this is a wire-format decode refusal at + // codec time, not a planner-stage failure. Downstream error + // classification keys off the variant — surfacing this as a planner + // error would mis-route it into "fix your SQL" buckets. + datafusion::error::DataFusionError::Execution(format!( + "Refusing to deserialize inline Python {kind} '{name}': Python UDF \ + inlining is disabled on this session. Two remediations: \ + (1) ask the sender to re-encode with inlining disabled so '{name}' \ + travels by name, and register '{name}' on this receiver; or \ + (2) enable inlining on this receiver (accepts the cloudpickle \ + execution risk on inbound payloads). Receivers cannot re-encode \ + bytes they did not produce." + )) +} + +/// `PhysicalExtensionCodec` mirror of [`PythonLogicalCodec`] parked +/// on the same `SessionContext`. Carries the Python-aware encoding +/// hooks for physical-layer types (`ExecutionPlan`, `PhysicalExpr`) +/// and delegates the rest to `inner`. +/// +/// The `PhysicalExtensionCodec` trait has its own `try_encode_udf` +/// / `try_decode_udf` pair distinct from the logical one, so a +/// `ScalarUDF` referenced inside a physical plan needs Python-aware +/// encoding on this layer too — otherwise a plan with a Python UDF +/// would round-trip at the logical level but break at the physical +/// level. Both layers reuse the shared payload framing +/// ([`PY_SCALAR_UDF_FAMILY`] et al.) so the wire format is identical. +#[derive(Debug)] +pub struct PythonPhysicalCodec { + inner: Arc, + python_udf_inlining: bool, +} + +impl PythonPhysicalCodec { + pub fn new(inner: Arc) -> Self { + Self { + inner, + python_udf_inlining: true, + } + } + + pub fn inner(&self) -> &Arc { + &self.inner + } + + /// Toggle inline encoding of Python UDFs on this physical codec. + /// + /// Mirrors [`PythonLogicalCodec::with_python_udf_inlining`]; see + /// that method for the full security and portability discussion. + pub fn with_python_udf_inlining(mut self, enabled: bool) -> Self { + self.python_udf_inlining = enabled; + self + } + + pub fn python_udf_inlining(&self) -> bool { + self.python_udf_inlining + } +} + +impl Default for PythonPhysicalCodec { + fn default() -> Self { + Self::new(Arc::new(DefaultPhysicalExtensionCodec {})) + } +} + +impl PhysicalExtensionCodec for PythonPhysicalCodec { + fn try_decode( + &self, + buf: &[u8], + inputs: &[Arc], + ctx: &TaskContext, + proto_converter: &dyn PhysicalProtoConverterExtension, + ) -> Result> { + self.inner.try_decode(buf, inputs, ctx, proto_converter) + } + + fn try_encode( + &self, + node: Arc, + buf: &mut Vec, + proto_converter: &dyn PhysicalProtoConverterExtension, + ) -> Result<()> { + self.inner.try_encode(node, buf, proto_converter) + } + + fn try_encode_udf(&self, node: &ScalarUDF, buf: &mut Vec) -> Result<()> { + if self.python_udf_inlining && try_encode_python_scalar_udf(node, buf)? { + return Ok(()); + } + self.inner.try_encode_udf(node, buf) + } + + fn try_decode_udf(&self, name: &str, buf: &[u8]) -> Result> { + if self.python_udf_inlining { + if let Some(udf) = try_decode_python_scalar_udf(buf)? { + return Ok(udf); + } + } else { + refuse_if_inline(buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", name)?; + } + self.inner.try_decode_udf(name, buf) + } + + fn try_encode_expr( + &self, + node: &Arc, + buf: &mut Vec, + ctx: &PhysicalExprEncodeCtx<'_>, + ) -> Result<()> { + self.inner.try_encode_expr(node, buf, ctx) + } + + fn try_decode_expr( + &self, + buf: &[u8], + inputs: &[Arc], + ctx: &PhysicalExprDecodeCtx<'_>, + ) -> Result> { + self.inner.try_decode_expr(buf, inputs, ctx) + } + + fn try_encode_udaf(&self, node: &AggregateUDF, buf: &mut Vec) -> Result<()> { + if self.python_udf_inlining && try_encode_python_udaf(node, buf)? { + return Ok(()); + } + self.inner.try_encode_udaf(node, buf) + } + + fn try_decode_udaf(&self, name: &str, buf: &[u8]) -> Result> { + if self.python_udf_inlining { + if let Some(udaf) = try_decode_python_udaf(buf)? { + return Ok(udaf); + } + } else { + refuse_if_inline(buf, PY_AGG_UDF_FAMILY, "aggregate UDF", name)?; + } + self.inner.try_decode_udaf(name, buf) + } + + fn try_encode_udwf(&self, node: &WindowUDF, buf: &mut Vec) -> Result<()> { + if self.python_udf_inlining && try_encode_python_udwf(node, buf)? { + return Ok(()); + } + self.inner.try_encode_udwf(node, buf) + } + + fn try_decode_udwf(&self, name: &str, buf: &[u8]) -> Result> { + if self.python_udf_inlining { + if let Some(udwf) = try_decode_python_udwf(buf)? { + return Ok(udwf); + } + } else { + refuse_if_inline(buf, PY_WINDOW_UDF_FAMILY, "window UDF", name)?; + } + self.inner.try_decode_udwf(name, buf) + } +} + +// ============================================================================= +// Shared Python scalar UDF encode / decode helpers +// +// Both `PythonLogicalCodec` and `PythonPhysicalCodec` consult these on +// every `try_encode_udf` / `try_decode_udf` call. Same wire format on +// both layers — a Python `ScalarUDF` referenced inside a `LogicalPlan` +// or an `ExecutionPlan` round-trips identically. +// ============================================================================= + +/// Encode a Python scalar UDF inline if `node` is one. Returns +/// `Ok(true)` when the payload (`DFPYUDF` family prefix, version byte, +/// cloudpickled tuple) was written and the caller should skip its +/// inner codec. Returns `Ok(false)` for any non-Python UDF, signalling +/// the caller to delegate to its `inner`. +pub(crate) fn try_encode_python_scalar_udf(node: &ScalarUDF, buf: &mut Vec) -> Result { + let Some(py_udf) = node.inner().downcast_ref::() else { + return Ok(false); + }; + + Python::attach(|py| -> Result { + let bytes = encode_python_scalar_udf(py, py_udf).map_err(to_datafusion_err)?; + append_framed_payload(py, buf, PY_SCALAR_UDF_FAMILY, &bytes)?; + Ok(true) + }) +} + +/// Decode an inline Python scalar UDF payload. Returns `Ok(None)` +/// when `buf` does not carry the `DFPYUDF` family prefix, signalling +/// the caller to delegate to its `inner` codec (and eventually the +/// `FunctionRegistry`). +pub(crate) fn try_decode_python_scalar_udf(buf: &[u8]) -> Result>> { + if !buf.starts_with(PY_SCALAR_UDF_FAMILY) { + return Ok(None); + } + Python::attach(|py| -> Result>> { + let Some(payload) = read_framed_payload(py, buf, PY_SCALAR_UDF_FAMILY, "scalar UDF")? + else { + return Ok(None); + }; + let udf = decode_python_scalar_udf(py, payload).map_err(to_datafusion_err)?; + Ok(Some(Arc::new(ScalarUDF::new_from_impl(udf)))) + }) +} + +/// Build the cloudpickle payload for a `PythonFunctionScalarUDF`. +/// +/// Layout: `cloudpickle.dumps((name, func, input_schema_bytes, +/// return_schema_bytes, volatility_str))`. Schema blobs are produced +/// by arrow-rs's native IPC stream writer (no pyarrow round-trip) and +/// decoded with the matching stream reader on the receiver. See +/// [`build_input_schema_bytes`] for what the input blob carries. +fn encode_python_scalar_udf(py: Python<'_>, udf: &PythonFunctionScalarUDF) -> PyResult> { + let signature = udf.signature(); + let input_dtypes = signature_input_dtypes(signature, "PythonFunctionScalarUDF")?; + let input_schema_bytes = build_input_schema_bytes(&input_dtypes)?; + let return_schema_bytes = build_single_field_schema_bytes(udf.return_field().as_ref())?; + let volatility = volatility_wire_str(signature.volatility); + + let payload = PyTuple::new( + py, + [ + udf.name().into_pyobject(py)?.into_any(), + udf.func().bind(py).clone().into_any(), + PyBytes::new(py, &input_schema_bytes).into_any(), + PyBytes::new(py, &return_schema_bytes).into_any(), + volatility.into_pyobject(py)?.into_any(), + ], + )?; + + cloudpickle(py)? + .call_method1("dumps", (payload,))? + .extract::>() +} + +/// Inverse of [`encode_python_scalar_udf`]. +fn decode_python_scalar_udf(py: Python<'_>, payload: &[u8]) -> PyResult { + let tuple = cloudpickle(py)? + .call_method1("loads", (PyBytes::new(py, payload),))? + .cast_into::()?; + + let name: String = tuple.get_item(0)?.extract()?; + let func: Py = tuple.get_item(1)?.unbind(); + let input_schema_bytes: Vec = tuple.get_item(2)?.extract()?; + let return_schema_bytes: Vec = tuple.get_item(3)?.extract()?; + let volatility_str: String = tuple.get_item(4)?.extract()?; + + let input_types = read_input_dtypes(&input_schema_bytes)?; + let return_field = read_single_return_field(&return_schema_bytes, "PythonFunctionScalarUDF")?; + let volatility = parse_volatility_str(&volatility_str)?; + + Ok(PythonFunctionScalarUDF::from_parts( + name, + func, + input_types, + return_field, + volatility, + )) +} + +/// Serialize a `Schema` to a self-contained IPC stream containing +/// only the schema message (no record batches). Inverse: +/// [`schema_from_ipc_bytes`]. +fn schema_to_ipc_bytes(schema: &Schema) -> arrow::error::Result> { + let mut buf: Vec = Vec::new(); + { + let mut writer = StreamWriter::try_new(&mut buf, schema)?; + writer.finish()?; + } + Ok(buf) +} + +/// Decode an IPC stream containing only a schema message back into a +/// `Schema`. Inverse: [`schema_to_ipc_bytes`]. +fn schema_from_ipc_bytes(bytes: &[u8]) -> arrow::error::Result { + let reader = StreamReader::try_new(std::io::Cursor::new(bytes), None)?; + Ok(reader.schema().as_ref().clone()) +} + +/// Extract the per-arg `DataType`s from a `Signature` known to be +/// `TypeSignature::Exact` (all Python-defined UDFs are constructed +/// with `Signature::exact`). Any other variant indicates the impl was +/// not built by this crate's UDF/UDAF/UDWF constructors. +fn signature_input_dtypes(signature: &Signature, kind: &str) -> PyResult> { + match &signature.type_signature { + TypeSignature::Exact(types) => Ok(types.clone()), + other => Err(pyo3::exceptions::PyValueError::new_err(format!( + "{kind} expected Signature::Exact, got {other:?}" + ))), + } +} + +/// Wrap per-arg `DataType`s in synthetic `arg_{i}` fields and emit +/// the IPC schema blob the encoder writes into the cloudpickle tuple. +/// +/// The names and `nullable: true` are arbitrary: the underlying +/// `TypeSignature::Exact` carries no per-input nullability or +/// metadata, and the receiver collapses these fields back to +/// `Vec` via [`read_input_dtypes`], so anything set here +/// beyond the data type is discarded on decode. +fn build_input_schema_bytes(dtypes: &[DataType]) -> PyResult> { + let fields: Vec = dtypes + .iter() + .enumerate() + .map(|(i, dt)| Field::new(format!("arg_{i}"), dt.clone(), true)) + .collect(); + schema_to_ipc_bytes(&Schema::new(fields)).map_err(arrow_to_py_err) +} + +/// Emit a single-field IPC schema blob. Used for return-type and +/// state-field payloads where the receiver needs to recover field +/// metadata (names, nullability, key/value attributes) verbatim. +fn build_single_field_schema_bytes(field: &Field) -> PyResult> { + schema_to_ipc_bytes(&Schema::new(vec![field.clone()])).map_err(arrow_to_py_err) +} + +/// Emit a multi-field IPC schema blob. +fn build_schema_bytes(fields: Vec) -> PyResult> { + schema_to_ipc_bytes(&Schema::new(fields)).map_err(arrow_to_py_err) +} + +/// Decode the per-arg `DataType`s the encoder wrote via +/// [`build_input_schema_bytes`]. +fn read_input_dtypes(bytes: &[u8]) -> PyResult> { + let schema = schema_from_ipc_bytes(bytes).map_err(arrow_to_py_err)?; + Ok(schema + .fields() + .iter() + .map(|f| f.data_type().clone()) + .collect()) +} + +/// Decode a single-field IPC schema blob and return that field by +/// value. `kind` names the UDF flavor in the error message produced +/// when the blob is empty (should be unreachable for sender-side +/// payloads built via [`build_single_field_schema_bytes`]). +fn read_single_return_field(bytes: &[u8], kind: &str) -> PyResult { + let schema = schema_from_ipc_bytes(bytes).map_err(arrow_to_py_err)?; + let field = schema.fields().first().ok_or_else(|| { + pyo3::exceptions::PyValueError::new_err(format!( + "{kind} return schema must contain exactly one field" + )) + })?; + Ok(field.as_ref().clone()) +} + +fn arrow_to_py_err(e: arrow::error::ArrowError) -> PyErr { + pyo3::exceptions::PyValueError::new_err(format!("{e}")) +} + +fn parse_volatility_str(s: &str) -> PyResult { + datafusion_python_util::parse_volatility(s) + .map_err(|e| pyo3::exceptions::PyValueError::new_err(format!("{e}"))) +} + +/// Stable wire-format string for a `Volatility`. Pinned to the three +/// tokens [`datafusion_python_util::parse_volatility`] accepts, so an +/// upstream change to `Volatility`'s `Debug` repr cannot silently +/// produce bytes the decoder rejects. +fn volatility_wire_str(v: Volatility) -> &'static str { + match v { + Volatility::Immutable => "immutable", + Volatility::Stable => "stable", + Volatility::Volatile => "volatile", + } +} + +/// Read the interpreter's `sys.version_info` as `(major, minor)`. +/// +/// Used by encoder/decoder to stamp and verify the Python version a +/// cloudpickle payload was produced on. cloudpickle is not portable +/// across Python minor versions; the wire header carries these bytes +/// so a mismatch surfaces an actionable error instead of an opaque +/// `marshal` failure at `cloudpickle.loads` time. +fn current_python_version(py: Python<'_>) -> PyResult<(u8, u8)> { + let version_info = py.import("sys")?.getattr("version_info")?; + let major: u8 = version_info.getattr("major")?.extract()?; + let minor: u8 = version_info.getattr("minor")?.extract()?; + Ok((major, minor)) +} + +/// Stamp `buf` with the framing header for `family` plus the current +/// Python `(major, minor)`, then append `payload`. Bundles the +/// `current_python_version` lookup with the header write so each +/// encoder call site stays one line. +fn append_framed_payload( + py: Python<'_>, + buf: &mut Vec, + family: &[u8], + payload: &[u8], +) -> Result<()> { + let py_version = current_python_version(py).map_err(to_datafusion_err)?; + write_wire_header(buf, family, py_version); + buf.extend_from_slice(payload); + Ok(()) +} + +/// Inspect `buf`'s framing against `family` + the current Python +/// `(major, minor)`. Returns `Ok(None)` when `buf` does not carry +/// `family` (caller should delegate); `Ok(Some(payload))` when the +/// framing matches; `Err(_)` for a recognised family at the wrong +/// wire-format or Python version (see [`strip_wire_header`]). +fn read_framed_payload<'a>( + py: Python<'_>, + buf: &'a [u8], + family: &[u8], + kind: &str, +) -> Result> { + let py_version = current_python_version(py).map_err(to_datafusion_err)?; + strip_wire_header(buf, family, kind, py_version) +} + +/// Cached handle to the `cloudpickle` module. +/// +/// The encode/decode helpers above would otherwise re-resolve the +/// module on every call. `py.import` is backed by `sys.modules` and +/// therefore cheap, but each call still walks a dict and re-binds the +/// result; a plan with many Python UDFs pays that cost per UDF. +/// +/// `PyOnceLock` scopes the cached `Py` to the current +/// interpreter, so the slot drops cleanly on interpreter teardown +/// (relevant under CPython subinterpreters, PEP 684) instead of +/// resurrecting a `Py` rooted in a dead interpreter on the next call. +fn cloudpickle<'py>(py: Python<'py>) -> PyResult> { + static CLOUDPICKLE: PyOnceLock> = PyOnceLock::new(); + CLOUDPICKLE + .get_or_try_init(py, || Ok(py.import("cloudpickle")?.unbind().into_any())) + .map(|cached| cached.bind(py).clone()) +} + +// ============================================================================= +// Shared Python window UDF encode / decode helpers +// +// Cloudpickle tuple shape: `(name, evaluator_factory, input_schema_bytes, +// return_schema_bytes, volatility_str)`. The evaluator factory is the +// Python callable that produces a new evaluator instance per partition. +// ============================================================================= + +pub(crate) fn try_encode_python_udwf(node: &WindowUDF, buf: &mut Vec) -> Result { + let Some(py_udf) = node.inner().downcast_ref::() else { + return Ok(false); + }; + + Python::attach(|py| -> Result { + let bytes = encode_python_udwf(py, py_udf).map_err(to_datafusion_err)?; + append_framed_payload(py, buf, PY_WINDOW_UDF_FAMILY, &bytes)?; + Ok(true) + }) +} + +pub(crate) fn try_decode_python_udwf(buf: &[u8]) -> Result>> { + if !buf.starts_with(PY_WINDOW_UDF_FAMILY) { + return Ok(None); + } + Python::attach(|py| -> Result>> { + let Some(payload) = read_framed_payload(py, buf, PY_WINDOW_UDF_FAMILY, "window UDF")? + else { + return Ok(None); + }; + let udf = decode_python_udwf(py, payload).map_err(to_datafusion_err)?; + Ok(Some(Arc::new(WindowUDF::new_from_impl(udf)))) + }) +} + +fn encode_python_udwf(py: Python<'_>, udf: &PythonFunctionWindowUDF) -> PyResult> { + let signature = WindowUDFImpl::signature(udf); + let input_dtypes = signature_input_dtypes(signature, "PythonFunctionWindowUDF")?; + let input_schema_bytes = build_input_schema_bytes(&input_dtypes)?; + let return_field = Field::new("result", udf.return_type().clone(), true); + let return_schema_bytes = build_single_field_schema_bytes(&return_field)?; + let volatility = volatility_wire_str(signature.volatility); + + let payload = PyTuple::new( + py, + [ + WindowUDFImpl::name(udf).into_pyobject(py)?.into_any(), + udf.evaluator().bind(py).clone().into_any(), + PyBytes::new(py, &input_schema_bytes).into_any(), + PyBytes::new(py, &return_schema_bytes).into_any(), + volatility.into_pyobject(py)?.into_any(), + ], + )?; + + cloudpickle(py)? + .call_method1("dumps", (payload,))? + .extract::>() +} + +fn decode_python_udwf(py: Python<'_>, payload: &[u8]) -> PyResult { + let tuple = cloudpickle(py)? + .call_method1("loads", (PyBytes::new(py, payload),))? + .cast_into::()?; + + let name: String = tuple.get_item(0)?.extract()?; + let evaluator: Py = tuple.get_item(1)?.unbind(); + let input_schema_bytes: Vec = tuple.get_item(2)?.extract()?; + let return_schema_bytes: Vec = tuple.get_item(3)?.extract()?; + let volatility_str: String = tuple.get_item(4)?.extract()?; + + let input_types = read_input_dtypes(&input_schema_bytes)?; + let return_type = read_single_return_field(&return_schema_bytes, "PythonFunctionWindowUDF")? + .data_type() + .clone(); + let volatility = parse_volatility_str(&volatility_str)?; + + Ok(PythonFunctionWindowUDF::new( + name, + evaluator, + input_types, + return_type, + volatility, + )) +} + +// ============================================================================= +// Shared Python aggregate UDF encode / decode helpers +// +// Cloudpickle tuple shape: `(name, accumulator_factory, input_schema_bytes, +// return_schema_bytes, state_schema_bytes, volatility_str)`. The accumulator +// factory is the Python callable that produces a new accumulator instance +// per partition. +// ============================================================================= + +pub(crate) fn try_encode_python_udaf(node: &AggregateUDF, buf: &mut Vec) -> Result { + let Some(py_udf) = node.inner().downcast_ref::() else { + return Ok(false); + }; + + Python::attach(|py| -> Result { + let bytes = encode_python_udaf(py, py_udf).map_err(to_datafusion_err)?; + append_framed_payload(py, buf, PY_AGG_UDF_FAMILY, &bytes)?; + Ok(true) + }) +} + +pub(crate) fn try_decode_python_udaf(buf: &[u8]) -> Result>> { + if !buf.starts_with(PY_AGG_UDF_FAMILY) { + return Ok(None); + } + Python::attach(|py| -> Result>> { + let Some(payload) = read_framed_payload(py, buf, PY_AGG_UDF_FAMILY, "aggregate UDF")? + else { + return Ok(None); + }; + let udf = decode_python_udaf(py, payload).map_err(to_datafusion_err)?; + Ok(Some(Arc::new(AggregateUDF::new_from_impl(udf)))) + }) +} + +fn encode_python_udaf(py: Python<'_>, udf: &PythonFunctionAggregateUDF) -> PyResult> { + let signature = AggregateUDFImpl::signature(udf); + let input_dtypes = signature_input_dtypes(signature, "PythonFunctionAggregateUDF")?; + let input_schema_bytes = build_input_schema_bytes(&input_dtypes)?; + let return_field = Field::new("result", udf.return_type().clone(), true); + let return_schema_bytes = build_single_field_schema_bytes(&return_field)?; + let state_fields: Vec = udf + .state_fields_ref() + .iter() + .map(|f| f.as_ref().clone()) + .collect(); + let state_schema_bytes = build_schema_bytes(state_fields)?; + let volatility = volatility_wire_str(signature.volatility); + + let payload = PyTuple::new( + py, + [ + AggregateUDFImpl::name(udf).into_pyobject(py)?.into_any(), + udf.accumulator().bind(py).clone().into_any(), + PyBytes::new(py, &input_schema_bytes).into_any(), + PyBytes::new(py, &return_schema_bytes).into_any(), + PyBytes::new(py, &state_schema_bytes).into_any(), + volatility.into_pyobject(py)?.into_any(), + ], + )?; + + cloudpickle(py)? + .call_method1("dumps", (payload,))? + .extract::>() +} + +fn decode_python_udaf(py: Python<'_>, payload: &[u8]) -> PyResult { + let tuple = cloudpickle(py)? + .call_method1("loads", (PyBytes::new(py, payload),))? + .cast_into::()?; + + let name: String = tuple.get_item(0)?.extract()?; + let accumulator: Py = tuple.get_item(1)?.unbind(); + let input_schema_bytes: Vec = tuple.get_item(2)?.extract()?; + let return_schema_bytes: Vec = tuple.get_item(3)?.extract()?; + let state_schema_bytes: Vec = tuple.get_item(4)?.extract()?; + let volatility_str: String = tuple.get_item(5)?.extract()?; + + let input_types = read_input_dtypes(&input_schema_bytes)?; + let return_type = read_single_return_field(&return_schema_bytes, "PythonFunctionAggregateUDF")? + .data_type() + .clone(); + // Preserve the encoded state field metadata (names, nullability, + // arbitrary key/value attributes) so the post-decode UDF reports + // the same state schema as the sender's instance — important for + // accumulators whose `StateFieldsArgs` consumers key off names or + // nullability rather than positional `DataType`. + let state_schema = schema_from_ipc_bytes(&state_schema_bytes).map_err(arrow_to_py_err)?; + let state_fields: Vec = + state_schema.fields().iter().cloned().collect(); + let volatility = parse_volatility_str(&volatility_str)?; + + Ok(PythonFunctionAggregateUDF::from_parts( + name, + accumulator, + input_types, + return_type, + state_fields, + volatility, + )) +} + +#[cfg(test)] +mod wire_header_tests { + use super::*; + + const TEST_PY: (u8, u8) = (3, 12); + + #[test] + fn strip_returns_none_when_family_absent() { + let buf = b"OTHER_PAYLOAD"; + assert!(matches!( + strip_wire_header(buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY), + Ok(None) + )); + } + + #[test] + fn strip_errors_on_truncated_version_byte() { + let buf = PY_SCALAR_UDF_FAMILY; + let err = strip_wire_header(buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY).unwrap_err(); + assert!(format!("{err}").contains("missing wire-format version byte")); + } + + #[test] + fn strip_errors_on_too_new_version() { + let mut buf = PY_SCALAR_UDF_FAMILY.to_vec(); + buf.push(WIRE_VERSION_CURRENT.saturating_add(1)); + buf.push(TEST_PY.0); + buf.push(TEST_PY.1); + buf.extend_from_slice(b"payload"); + let err = strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY).unwrap_err(); + let msg = format!("{err}"); + assert!(msg.contains("wire-format version v")); + assert!(msg.contains("supports")); + assert!(msg.contains("Align datafusion-python versions")); + } + + #[test] + fn strip_errors_on_too_old_version() { + if WIRE_VERSION_MIN_SUPPORTED == 0 { + return; + } + let mut buf = PY_SCALAR_UDF_FAMILY.to_vec(); + buf.push(WIRE_VERSION_MIN_SUPPORTED - 1); + buf.push(TEST_PY.0); + buf.push(TEST_PY.1); + buf.extend_from_slice(b"payload"); + assert!(strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY).is_err()); + } + + #[test] + fn strip_errors_on_truncated_py_major() { + let mut buf = PY_SCALAR_UDF_FAMILY.to_vec(); + buf.push(WIRE_VERSION_CURRENT); + let err = strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY).unwrap_err(); + assert!(format!("{err}").contains("missing Python major version byte")); + } + + #[test] + fn strip_errors_on_truncated_py_minor() { + let mut buf = PY_SCALAR_UDF_FAMILY.to_vec(); + buf.push(WIRE_VERSION_CURRENT); + buf.push(TEST_PY.0); + let err = strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY).unwrap_err(); + assert!(format!("{err}").contains("missing Python minor version byte")); + } + + #[test] + fn strip_errors_on_py_minor_mismatch() { + let mut buf = Vec::new(); + write_wire_header(&mut buf, PY_SCALAR_UDF_FAMILY, (3, 11)); + buf.extend_from_slice(b"payload"); + let err = strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", (3, 12)).unwrap_err(); + let msg = format!("{err}"); + assert!(msg.contains("Python 3.11")); + assert!(msg.contains("Python 3.12")); + assert!(msg.contains("not portable across Python minor versions")); + } + + #[test] + fn strip_errors_on_py_major_mismatch() { + let mut buf = Vec::new(); + write_wire_header(&mut buf, PY_SCALAR_UDF_FAMILY, (3, 12)); + buf.extend_from_slice(b"payload"); + assert!(strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", (4, 0)).is_err()); + } + + #[test] + fn write_then_strip_round_trips_scalar_payload() { + let mut buf = Vec::new(); + write_wire_header(&mut buf, PY_SCALAR_UDF_FAMILY, TEST_PY); + buf.extend_from_slice(b"scalar-payload"); + + let payload = strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY) + .unwrap() + .unwrap(); + assert_eq!(payload, b"scalar-payload"); + } + + #[test] + fn write_then_strip_round_trips_agg_payload() { + let mut buf = Vec::new(); + write_wire_header(&mut buf, PY_AGG_UDF_FAMILY, TEST_PY); + buf.extend_from_slice(b"agg-payload"); + + let payload = strip_wire_header(&buf, PY_AGG_UDF_FAMILY, "aggregate UDF", TEST_PY) + .unwrap() + .unwrap(); + assert_eq!(payload, b"agg-payload"); + } + + #[test] + fn write_then_strip_round_trips_window_payload() { + let mut buf = Vec::new(); + write_wire_header(&mut buf, PY_WINDOW_UDF_FAMILY, TEST_PY); + buf.extend_from_slice(b"window-payload"); + + let payload = strip_wire_header(&buf, PY_WINDOW_UDF_FAMILY, "window UDF", TEST_PY) + .unwrap() + .unwrap(); + assert_eq!(payload, b"window-payload"); + } + + #[test] + fn strip_does_not_match_a_different_family() { + let mut buf = Vec::new(); + write_wire_header(&mut buf, PY_SCALAR_UDF_FAMILY, TEST_PY); + buf.extend_from_slice(b"payload"); + assert!(matches!( + strip_wire_header(&buf, PY_WINDOW_UDF_FAMILY, "window UDF", TEST_PY), + Ok(None) + )); + } +} diff --git a/src/common.rs b/crates/core/src/common.rs similarity index 100% rename from src/common.rs rename to crates/core/src/common.rs diff --git a/src/common/data_type.rs b/crates/core/src/common/data_type.rs similarity index 99% rename from src/common/data_type.rs rename to crates/core/src/common/data_type.rs index af4179806..e79aea4ef 100644 --- a/src/common/data_type.rs +++ b/crates/core/src/common/data_type.rs @@ -334,6 +334,9 @@ impl DataTypeMap { Ok(DataType::Interval(IntervalUnit::MonthDayNano)) } ScalarValue::List(arr) => Ok(arr.data_type().to_owned()), + ScalarValue::LargeList(arr) => Ok(arr.data_type().to_owned()), + ScalarValue::ListView(arr) => Ok(arr.data_type().to_owned()), + ScalarValue::LargeListView(arr) => Ok(arr.data_type().to_owned()), ScalarValue::Struct(_fields) => Err(PyNotImplementedError::new_err( "ScalarValue::Struct".to_string(), )), @@ -346,9 +349,6 @@ impl DataTypeMap { "ScalarValue::FixedSizeList".to_string(), )) } - ScalarValue::LargeList(_) => Err(PyNotImplementedError::new_err( - "ScalarValue::LargeList".to_string(), - )), ScalarValue::DurationSecond(_) => Ok(DataType::Duration(TimeUnit::Second)), ScalarValue::DurationMillisecond(_) => Ok(DataType::Duration(TimeUnit::Millisecond)), ScalarValue::DurationMicrosecond(_) => Ok(DataType::Duration(TimeUnit::Microsecond)), diff --git a/src/common/df_schema.rs b/crates/core/src/common/df_schema.rs similarity index 100% rename from src/common/df_schema.rs rename to crates/core/src/common/df_schema.rs diff --git a/src/common/function.rs b/crates/core/src/common/function.rs similarity index 100% rename from src/common/function.rs rename to crates/core/src/common/function.rs diff --git a/src/common/schema.rs b/crates/core/src/common/schema.rs similarity index 99% rename from src/common/schema.rs rename to crates/core/src/common/schema.rs index 29a27b204..94b3ce0ae 100644 --- a/src/common/schema.rs +++ b/crates/core/src/common/schema.rs @@ -15,7 +15,6 @@ // specific language governing permissions and limitations // under the License. -use std::any::Any; use std::borrow::Cow; use std::fmt::{self, Display, Formatter}; use std::sync::Arc; @@ -219,10 +218,6 @@ impl SqlTableSource { /// Implement TableSource, used in the logical query plan and in logical query optimizations impl TableSource for SqlTableSource { - fn as_any(&self) -> &dyn Any { - self - } - fn schema(&self) -> SchemaRef { self.schema.clone() } diff --git a/src/context.rs b/crates/core/src/context.rs similarity index 60% rename from src/context.rs rename to crates/core/src/context.rs index 2eaf5a737..7bbeed2f1 100644 --- a/src/context.rs +++ b/crates/core/src/context.rs @@ -16,7 +16,7 @@ // under the License. use std::collections::{HashMap, HashSet}; -use std::path::PathBuf; +use std::path::{Path, PathBuf}; use std::ptr::NonNull; use std::str::FromStr; use std::sync::Arc; @@ -27,35 +27,44 @@ use arrow::pyarrow::FromPyArrow; use datafusion::arrow::datatypes::{DataType, Schema, SchemaRef}; use datafusion::arrow::pyarrow::PyArrowType; use datafusion::arrow::record_batch::RecordBatch; -use datafusion::catalog::{CatalogProvider, CatalogProviderList}; -use datafusion::common::{ScalarValue, TableReference, exec_err}; +use datafusion::catalog::{CatalogProvider, CatalogProviderList, TableProviderFactory}; +use datafusion::common::{DFSchema, ScalarValue, TableReference, exec_err}; use datafusion::datasource::file_format::file_compression_type::FileCompressionType; use datafusion::datasource::file_format::parquet::ParquetFormat; use datafusion::datasource::listing::{ ListingOptions, ListingTable, ListingTableConfig, ListingTableUrl, }; use datafusion::datasource::{MemTable, TableProvider}; -use datafusion::execution::TaskContextProvider; use datafusion::execution::context::{ DataFilePaths, SQLOptions, SessionConfig, SessionContext, TaskContext, }; use datafusion::execution::disk_manager::DiskManagerMode; use datafusion::execution::memory_pool::{FairSpillPool, GreedyMemoryPool, UnboundedMemoryPool}; -use datafusion::execution::options::ReadOptions; +use datafusion::execution::options::{ArrowReadOptions, ReadOptions}; use datafusion::execution::runtime_env::RuntimeEnvBuilder; use datafusion::execution::session_state::SessionStateBuilder; +use datafusion::execution::{FunctionRegistry, TaskContextProvider}; use datafusion::prelude::{ AvroReadOptions, CsvReadOptions, DataFrame, JsonReadOptions, ParquetReadOptions, }; use datafusion_ffi::catalog_provider::FFI_CatalogProvider; use datafusion_ffi::catalog_provider_list::FFI_CatalogProviderList; +use datafusion_ffi::config::extension_options::FFI_ExtensionOptions; use datafusion_ffi::execution::FFI_TaskContextProvider; use datafusion_ffi::proto::logical_extension_codec::FFI_LogicalExtensionCodec; -use datafusion_proto::logical_plan::DefaultLogicalExtensionCodec; +use datafusion_ffi::proto::physical_extension_codec::FFI_PhysicalExtensionCodec; +use datafusion_ffi::table_provider_factory::FFI_TableProviderFactory; +use datafusion_proto::logical_plan::LogicalExtensionCodec; +use datafusion_proto::physical_plan::PhysicalExtensionCodec; +use datafusion_python_util::{ + create_logical_extension_capsule, create_physical_extension_capsule, + ffi_logical_codec_from_pycapsule, get_global_ctx, get_tokio_runtime, + physical_codec_from_pycapsule, physical_optimizer_rule_from_pycapsule, spawn_future, + wait_for_future, +}; use object_store::ObjectStore; use pyo3::IntoPyObjectExt; -use pyo3::exceptions::{PyKeyError, PyValueError}; -use pyo3::ffi::c_str; +use pyo3::exceptions::{PyKeyError, PyRuntimeError, PyValueError}; use pyo3::prelude::*; use pyo3::types::{PyCapsule, PyDict, PyList, PyTuple}; use url::Url; @@ -64,12 +73,15 @@ use uuid::Uuid; use crate::catalog::{ PyCatalog, PyCatalogList, RustWrappedPyCatalogProvider, RustWrappedPyCatalogProviderList, }; +use crate::codec::{PythonLogicalCodec, PythonPhysicalCodec}; use crate::common::data_type::PyScalarValue; +use crate::common::df_schema::PyDFSchema; use crate::dataframe::PyDataFrame; use crate::dataset::Dataset; use crate::errors::{ PyDataFusionError, PyDataFusionResult, from_datafusion_error, py_datafusion_err, }; +use crate::expr::PyExpr; use crate::expr::sort_expr::PySortExpr; use crate::options::PyCsvReadOptions; use crate::physical_plan::PyExecutionPlan; @@ -77,15 +89,11 @@ use crate::record_batch::PyRecordBatchStream; use crate::sql::logical::PyLogicalPlan; use crate::sql::util::replace_placeholders_with_strings; use crate::store::StorageContexts; -use crate::table::PyTable; +use crate::table::{PyTable, RustWrappedPyTableProviderFactory}; use crate::udaf::PyAggregateUDF; use crate::udf::PyScalarUDF; use crate::udtf::PyTableFunction; use crate::udwf::PyWindowUDF; -use crate::utils::{ - create_logical_extension_capsule, extract_logical_extension_codec, get_global_ctx, - get_tokio_runtime, spawn_future, validate_pycapsule, wait_for_future, -}; /// Configuration options for a SessionContext #[pyclass( @@ -184,6 +192,33 @@ impl PySessionConfig { fn set(&self, key: &str, value: &str) -> Self { Self::from(self.config.clone().set_str(key, value)) } + + pub fn with_extension(&self, extension: Bound) -> PyResult { + if !extension.hasattr("__datafusion_extension_options__")? { + return Err(pyo3::exceptions::PyAttributeError::new_err( + "Expected extension object to define __datafusion_extension_options__()", + )); + } + let capsule = extension.call_method0("__datafusion_extension_options__")?; + let capsule = capsule.cast::()?; + + let extension: NonNull = capsule + .pointer_checked(Some(c"datafusion_extension_options"))? + .cast(); + let mut extension = unsafe { extension.as_ref() }.clone(); + + let mut config = self.config.clone(); + let options = config.options_mut(); + if let Some(prior_extension) = options.extensions.get::() { + extension + .merge(prior_extension) + .map_err(py_datafusion_err)?; + } + + options.extensions.insert(extension); + + Ok(Self::from(config)) + } } /// Runtime options for a SessionContext @@ -335,7 +370,8 @@ impl PySQLOptions { #[derive(Clone)] pub struct PySessionContext { pub ctx: Arc, - logical_codec: Arc, + logical_codec: Arc, + physical_codec: Arc, } #[pymethods] @@ -361,16 +397,21 @@ impl PySessionContext { .with_config(config) .with_runtime_env(runtime) .with_default_features() + .with_analyzer_rule(Arc::new(crate::analyzer::ResolveLambdaVariables::new())) .build(); let ctx = Arc::new(SessionContext::new_with_state(session_state)); - let logical_codec = Self::default_logical_codec(&ctx); - Ok(PySessionContext { ctx, logical_codec }) + Ok(PySessionContext { + ctx, + logical_codec: Arc::new(PythonLogicalCodec::default()), + physical_codec: Arc::new(PythonPhysicalCodec::default()), + }) } pub fn enable_url_table(&self) -> PyResult { Ok(PySessionContext { ctx: Arc::new(self.ctx.as_ref().clone().enable_url_table()), logical_codec: Arc::clone(&self.logical_codec), + physical_codec: Arc::clone(&self.physical_codec), }) } @@ -378,8 +419,11 @@ impl PySessionContext { #[pyo3(signature = ())] pub fn global_ctx() -> PyResult { let ctx = get_global_ctx().clone(); - let logical_codec = Self::default_logical_codec(&ctx); - Ok(Self { ctx, logical_codec }) + Ok(Self { + ctx, + logical_codec: Arc::new(PythonLogicalCodec::default()), + physical_codec: Arc::new(PythonPhysicalCodec::default()), + }) } /// Register an object store with the given name @@ -406,11 +450,25 @@ impl PySessionContext { &upstream_host }; let url_string = format!("{scheme}{derived_host}"); - let url = Url::parse(&url_string).unwrap(); + let url = Url::parse(&url_string).map_err(|e| PyValueError::new_err(e.to_string()))?; self.ctx.runtime_env().register_object_store(&url, store); Ok(()) } + /// Deregister an object store with the given url + #[pyo3(signature = (scheme, host=None))] + pub fn deregister_object_store( + &self, + scheme: &str, + host: Option<&str>, + ) -> PyDataFusionResult<()> { + let host = host.unwrap_or(""); + let url_string = format!("{scheme}{host}"); + let url = Url::parse(&url_string).map_err(|e| PyDataFusionError::Common(e.to_string()))?; + self.ctx.runtime_env().deregister_object_store(&url)?; + Ok(()) + } + #[allow(clippy::too_many_arguments)] #[pyo3(signature = (name, path, table_partition_cols=vec![], file_extension=".parquet", @@ -419,7 +477,7 @@ impl PySessionContext { pub fn register_listing_table( &self, name: &str, - path: &str, + path: PathBuf, table_partition_cols: Vec<(String, PyArrowType)>, file_extension: &str, schema: Option>, @@ -428,20 +486,9 @@ impl PySessionContext { ) -> PyDataFusionResult<()> { let options = ListingOptions::new(Arc::new(ParquetFormat::new())) .with_file_extension(file_extension) - .with_table_partition_cols( - table_partition_cols - .into_iter() - .map(|(name, ty)| (name, ty.0)) - .collect::>(), - ) - .with_file_sort_order( - file_sort_order - .unwrap_or_default() - .into_iter() - .map(|e| e.into_iter().map(|f| f.into()).collect()) - .collect(), - ); - let table_path = ListingTableUrl::parse(path)?; + .with_table_partition_cols(convert_partition_cols(table_partition_cols)) + .with_file_sort_order(convert_file_sort_order(file_sort_order)); + let table_path = ListingTableUrl::parse(path_to_str(&path)?)?; let resolved_schema: SchemaRef = match schema { Some(s) => Arc::new(s.0), None => { @@ -464,6 +511,10 @@ impl PySessionContext { self.ctx.register_udtf(&name, func); } + pub fn deregister_udtf(&self, name: &str) { + self.ctx.deregister_udtf(name); + } + #[pyo3(signature = (query, options=None, param_values=HashMap::default(), param_strings=HashMap::default()))] pub fn sql_with_options( &self, @@ -659,34 +710,68 @@ impl PySessionContext { Ok(()) } + pub fn register_table_factory( + &self, + format: &str, + mut factory: Bound<'_, PyAny>, + ) -> PyDataFusionResult<()> { + if factory.hasattr("__datafusion_table_provider_factory__")? { + let py = factory.py(); + let ffi = self.ffi_logical_codec(); + let codec_capsule = create_logical_extension_capsule(py, ffi.as_ref())?; + factory = factory + .getattr("__datafusion_table_provider_factory__")? + .call1((codec_capsule,))?; + } + + let factory: Arc = + if let Ok(capsule) = factory.cast::().map_err(py_datafusion_err) { + let data: NonNull = capsule + .pointer_checked(Some(c"datafusion_table_provider_factory"))? + .cast(); + let factory = unsafe { data.as_ref() }; + factory.into() + } else { + Arc::new(RustWrappedPyTableProviderFactory::new( + factory.into(), + self.ffi_logical_codec(), + )) + }; + + let st = self.ctx.state_ref(); + let mut lock = st.write(); + lock.table_factories_mut() + .insert(format.to_owned(), factory); + + Ok(()) + } + pub fn register_catalog_provider_list( &self, mut provider: Bound, ) -> PyDataFusionResult<()> { if provider.hasattr("__datafusion_catalog_provider_list__")? { let py = provider.py(); - let codec_capsule = create_logical_extension_capsule(py, self.logical_codec.as_ref())?; + let ffi = self.ffi_logical_codec(); + let codec_capsule = create_logical_extension_capsule(py, ffi.as_ref())?; provider = provider .getattr("__datafusion_catalog_provider_list__")? .call1((codec_capsule,))?; } - let provider = if let Ok(capsule) = provider.cast::().map_err(py_datafusion_err) - { - validate_pycapsule(capsule, "datafusion_catalog_provider_list")?; - + let provider = if let Ok(capsule) = provider.cast::() { let data: NonNull = capsule - .pointer_checked(Some(c_str!("datafusion_catalog_provider_list")))? + .pointer_checked(Some(c"datafusion_catalog_provider_list"))? .cast(); let provider = unsafe { data.as_ref() }; - let provider: Arc = provider.into(); - provider as Arc + let provider: Arc = provider.into(); + provider } else { match provider.extract::() { Ok(py_catalog_list) => py_catalog_list.catalog_list, Err(_) => Arc::new(RustWrappedPyCatalogProviderList::new( provider.into(), - Arc::clone(&self.logical_codec), + self.ffi_logical_codec(), )) as Arc, } }; @@ -703,28 +788,26 @@ impl PySessionContext { ) -> PyDataFusionResult<()> { if provider.hasattr("__datafusion_catalog_provider__")? { let py = provider.py(); - let codec_capsule = create_logical_extension_capsule(py, self.logical_codec.as_ref())?; + let ffi = self.ffi_logical_codec(); + let codec_capsule = create_logical_extension_capsule(py, ffi.as_ref())?; provider = provider .getattr("__datafusion_catalog_provider__")? .call1((codec_capsule,))?; } - let provider = if let Ok(capsule) = provider.cast::().map_err(py_datafusion_err) - { - validate_pycapsule(capsule, "datafusion_catalog_provider")?; - + let provider = if let Ok(capsule) = provider.cast::() { let data: NonNull = capsule - .pointer_checked(Some(c_str!("datafusion_catalog_provider")))? + .pointer_checked(Some(c"datafusion_catalog_provider"))? .cast(); let provider = unsafe { data.as_ref() }; - let provider: Arc = provider.into(); - provider as Arc + let provider: Arc = provider.into(); + provider } else { match provider.extract::() { Ok(py_catalog) => py_catalog.catalog, Err(_) => Arc::new(RustWrappedPyCatalogProvider::new( provider.into(), - Arc::clone(&self.logical_codec), + self.ffi_logical_codec(), )) as Arc, } }; @@ -749,12 +832,31 @@ impl PySessionContext { name: &str, partitions: PyArrowType>>, ) -> PyDataFusionResult<()> { - let schema = partitions.0[0][0].schema(); + // Take the schema from the first available batch; error instead of + // panicking when the partitions hold no batches. + let schema = partitions + .0 + .iter() + .find_map(|partition| partition.first()) + .ok_or_else(|| { + PyValueError::new_err( + "Cannot register record batches without a schema: the \ + provided partitions contain no record batches.", + ) + })? + .schema(); let table = MemTable::try_new(schema, partitions.0)?; self.ctx.register_table(name, Arc::new(table))?; Ok(()) } + pub fn read_batches( + &self, + batches: PyArrowType>, + ) -> PyDataFusionResult { + Ok(PyDataFrame::new(self.ctx.read_batches(batches.0)?)) + } + #[allow(clippy::too_many_arguments)] #[pyo3(signature = (name, path, table_partition_cols=vec![], parquet_pruning=true, @@ -765,7 +867,7 @@ impl PySessionContext { pub fn register_parquet( &self, name: &str, - path: &str, + path: PathBuf, table_partition_cols: Vec<(String, PyArrowType)>, parquet_pruning: bool, file_extension: &str, @@ -774,25 +876,19 @@ impl PySessionContext { file_sort_order: Option>>, py: Python, ) -> PyDataFusionResult<()> { - let mut options = ParquetReadOptions::default() - .table_partition_cols( - table_partition_cols - .into_iter() - .map(|(name, ty)| (name, ty.0)) - .collect::>(), - ) - .parquet_pruning(parquet_pruning) - .skip_metadata(skip_metadata); - options.file_extension = file_extension; - options.schema = schema.as_ref().map(|x| &x.0); - options.file_sort_order = file_sort_order - .unwrap_or_default() - .into_iter() - .map(|e| e.into_iter().map(|f| f.into()).collect()) - .collect(); - - let result = self.ctx.register_parquet(name, path, options); - wait_for_future(py, result)??; + let options = build_parquet_options( + table_partition_cols, + parquet_pruning, + file_extension, + skip_metadata, + &schema, + file_sort_order, + ); + wait_for_future( + py, + self.ctx + .register_parquet(name, path_to_str(&path)?, options), + )??; Ok(()) } @@ -806,19 +902,24 @@ impl PySessionContext { options: Option<&PyCsvReadOptions>, py: Python, ) -> PyDataFusionResult<()> { - let options = options - .map(|opts| opts.try_into()) - .transpose()? - .unwrap_or_default(); + let options = convert_csv_options(options)?; if path.is_instance_of::() { - let paths = path.extract::>()?; - let result = self.register_csv_from_multiple_paths(name, paths, options); - wait_for_future(py, result)??; + let paths = path + .extract::>()? + .iter() + .map(|p| path_to_str(p).map(str::to_owned)) + .collect::>>()?; + wait_for_future( + py, + self.register_csv_from_multiple_paths(name, paths, options), + )??; } else { - let path = path.extract::()?; - let result = self.ctx.register_csv(name, &path, options); - wait_for_future(py, result)??; + let path = path.extract::()?; + wait_for_future( + py, + self.ctx.register_csv(name, path_to_str(&path)?, options), + )??; } Ok(()) @@ -843,25 +944,17 @@ impl PySessionContext { file_compression_type: Option, py: Python, ) -> PyDataFusionResult<()> { - let path = path - .to_str() - .ok_or_else(|| PyValueError::new_err("Unable to convert path to a string"))?; - - let mut options = JsonReadOptions::default() - .file_compression_type(parse_file_compression_type(file_compression_type)?) - .table_partition_cols( - table_partition_cols - .into_iter() - .map(|(name, ty)| (name, ty.0)) - .collect::>(), - ); - options.schema_infer_max_records = schema_infer_max_records; - options.file_extension = file_extension; - options.schema = schema.as_ref().map(|x| &x.0); - - let result = self.ctx.register_json(name, path, options); - wait_for_future(py, result)??; - + let options = build_json_options( + table_partition_cols, + file_compression_type, + schema_infer_max_records, + file_extension, + &schema, + )?; + wait_for_future( + py, + self.ctx.register_json(name, path_to_str(&path)?, options), + )??; Ok(()) } @@ -880,22 +973,38 @@ impl PySessionContext { table_partition_cols: Vec<(String, PyArrowType)>, py: Python, ) -> PyDataFusionResult<()> { - let path = path - .to_str() - .ok_or_else(|| PyValueError::new_err("Unable to convert path to a string"))?; - - let mut options = AvroReadOptions::default().table_partition_cols( - table_partition_cols - .into_iter() - .map(|(name, ty)| (name, ty.0)) - .collect::>(), - ); - options.file_extension = file_extension; - options.schema = schema.as_ref().map(|x| &x.0); + let options = build_avro_options(table_partition_cols, file_extension, &schema); + wait_for_future( + py, + self.ctx.register_avro(name, path_to_str(&path)?, options), + )??; + Ok(()) + } - let result = self.ctx.register_avro(name, path, options); - wait_for_future(py, result)??; + #[pyo3(signature = (name, path, schema=None, file_extension=".arrow", table_partition_cols=vec![]))] + pub fn register_arrow( + &self, + name: &str, + path: PathBuf, + schema: Option>, + file_extension: &str, + table_partition_cols: Vec<(String, PyArrowType)>, + py: Python, + ) -> PyDataFusionResult<()> { + let options = build_arrow_options(table_partition_cols, file_extension, &schema); + wait_for_future( + py, + self.ctx.register_arrow(name, path_to_str(&path)?, options), + )??; + Ok(()) + } + pub fn register_batch( + &self, + name: &str, + batch: PyArrowType, + ) -> PyDataFusionResult<()> { + self.ctx.register_batch(name, batch.0)?; Ok(()) } @@ -918,31 +1027,96 @@ impl PySessionContext { Ok(()) } + pub fn deregister_udf(&self, name: &str) { + self.ctx.deregister_udf(name); + } + pub fn register_udaf(&self, udaf: PyAggregateUDF) -> PyResult<()> { self.ctx.register_udaf(udaf.function); Ok(()) } + /// Register all `datafusion-spark` UDFs/UDAFs/UDWFs, overriding any built-in + /// DataFusion functions of the same name with their Spark-semantics version. + pub fn enable_spark_functions(&self) -> PyResult<()> { + for udf in datafusion_spark::all_default_scalar_functions() { + self.ctx.register_udf((*udf).clone()); + } + for udaf in datafusion_spark::all_default_aggregate_functions() { + self.ctx.register_udaf((*udaf).clone()); + } + for udwf in datafusion_spark::all_default_window_functions() { + self.ctx.register_udwf((*udwf).clone()); + } + Ok(()) + } + + pub fn deregister_udaf(&self, name: &str) { + self.ctx.deregister_udaf(name); + } + pub fn register_udwf(&self, udwf: PyWindowUDF) -> PyResult<()> { self.ctx.register_udwf(udwf.function); Ok(()) } + pub fn deregister_udwf(&self, name: &str) { + self.ctx.deregister_udwf(name); + } + + pub fn udf(&self, name: &str) -> PyResult { + if !self.ctx.udfs().contains(name) { + return Err(PyKeyError::new_err(format!("no UDF named '{name}'"))); + } + let function = (*self.ctx.udf(name).map_err(py_datafusion_err)?).clone(); + Ok(PyScalarUDF { function }) + } + + pub fn udaf(&self, name: &str) -> PyResult { + if !self.ctx.udafs().contains(name) { + return Err(PyKeyError::new_err(format!("no UDAF named '{name}'"))); + } + let function = (*self.ctx.udaf(name).map_err(py_datafusion_err)?).clone(); + Ok(PyAggregateUDF { function }) + } + + pub fn udwf(&self, name: &str) -> PyResult { + if !self.ctx.udwfs().contains(name) { + return Err(PyKeyError::new_err(format!("no UDWF named '{name}'"))); + } + let function = (*self.ctx.udwf(name).map_err(py_datafusion_err)?).clone(); + Ok(PyWindowUDF { function }) + } + + pub fn udfs(&self) -> Vec { + let mut names: Vec = self.ctx.udfs().into_iter().collect(); + names.sort(); + names + } + + pub fn udafs(&self) -> Vec { + let mut names: Vec = self.ctx.udafs().into_iter().collect(); + names.sort(); + names + } + + pub fn udwfs(&self) -> Vec { + let mut names: Vec = self.ctx.udwfs().into_iter().collect(); + names.sort(); + names + } + #[pyo3(signature = (name="datafusion"))] pub fn catalog(&self, py: Python, name: &str) -> PyResult> { let catalog = self.ctx.catalog(name).ok_or(PyKeyError::new_err(format!( "Catalog with name {name} doesn't exist." )))?; - match catalog - .as_any() - .downcast_ref::() - { + match catalog.downcast_ref::() { Some(wrapped_schema) => Ok(wrapped_schema.catalog_provider.clone_ref(py)), - None => Ok( - PyCatalog::new_from_parts(catalog, Arc::clone(&self.logical_codec)) - .into_py_any(py)?, - ), + None => { + Ok(PyCatalog::new_from_parts(catalog, self.ffi_logical_codec()).into_py_any(py)?) + } } } @@ -950,21 +1124,6 @@ impl PySessionContext { self.ctx.catalog_names().into_iter().collect() } - pub fn tables(&self) -> HashSet { - self.ctx - .catalog_names() - .into_iter() - .filter_map(|name| self.ctx.catalog(&name)) - .flat_map(move |catalog| { - catalog - .schema_names() - .into_iter() - .filter_map(move |name| catalog.schema(&name)) - }) - .flat_map(|schema| schema.table_names()) - .collect() - } - pub fn table(&self, name: &str, py: Python) -> PyResult { let res = wait_for_future(py, self.ctx.table(name)) .map_err(|e| PyKeyError::new_err(e.to_string()))?; @@ -993,6 +1152,74 @@ impl PySessionContext { self.ctx.session_id() } + /// Return a copy of the active `SessionConfig`. Mutating the returned + /// config does not affect this context. + pub fn copied_config(&self) -> PySessionConfig { + self.ctx.copied_config().into() + } + + /// Parse a string like `"100M"`, `"1.5G"`, or `"512K"` into a byte count. + /// `"0"` is accepted and returns 0. Use this when constructing a + /// `RuntimeEnvBuilder` from a human-friendly size string. + #[staticmethod] + pub fn parse_capacity_limit(config_name: &str, limit: &str) -> PyDataFusionResult { + Ok(SessionContext::parse_capacity_limit(config_name, limit)?) + } + + pub fn session_start_time(&self) -> String { + self.ctx.session_start_time().to_rfc3339() + } + + pub fn enable_ident_normalization(&self) -> bool { + self.ctx.enable_ident_normalization() + } + + pub fn parse_sql_expr(&self, sql: &str, schema: PyDFSchema) -> PyDataFusionResult { + let df_schema: DFSchema = schema.into(); + Ok(self.ctx.parse_sql_expr(sql, &df_schema)?.into()) + } + + pub fn execute_logical_plan( + &self, + plan: PyLogicalPlan, + py: Python, + ) -> PyDataFusionResult { + let df = wait_for_future( + py, + self.ctx.execute_logical_plan(plan.plan.as_ref().clone()), + )??; + Ok(PyDataFrame::new(df)) + } + + pub fn refresh_catalogs(&self, py: Python) -> PyDataFusionResult<()> { + wait_for_future(py, self.ctx.refresh_catalogs())??; + Ok(()) + } + + pub fn remove_optimizer_rule(&self, name: &str) -> bool { + self.ctx.remove_optimizer_rule(name) + } + + pub fn add_physical_optimizer_rule(&self, rule: Bound<'_, PyAny>) -> PyDataFusionResult<()> { + let rule = physical_optimizer_rule_from_pycapsule(&rule)?; + let state_ref = self.ctx.state_ref(); + let mut guard = state_ref.write(); + let new_state = SessionStateBuilder::new_from_existing(guard.clone()) + .with_physical_optimizer_rule(rule) + .build(); + *guard = new_state; + Ok(()) + } + + pub fn table_provider(&self, name: &str, py: Python) -> PyResult { + let provider = wait_for_future(py, self.ctx.table_provider(name)) + // Outer error: runtime/async failure + .map_err(|e| PyRuntimeError::new_err(e.to_string()))? + // Inner error: table not found + .map_err(|e| PyKeyError::new_err(e.to_string()))?; + Ok(PyTable { table: provider }) + } + #[allow(clippy::too_many_arguments)] #[pyo3(signature = (path, schema=None, schema_infer_max_records=1000, file_extension=".json", table_partition_cols=vec![], file_compression_type=None))] pub fn read_json( @@ -1005,27 +1232,14 @@ impl PySessionContext { file_compression_type: Option, py: Python, ) -> PyDataFusionResult { - let path = path - .to_str() - .ok_or_else(|| PyValueError::new_err("Unable to convert path to a string"))?; - let mut options = JsonReadOptions::default() - .table_partition_cols( - table_partition_cols - .into_iter() - .map(|(name, ty)| (name, ty.0)) - .collect::>(), - ) - .file_compression_type(parse_file_compression_type(file_compression_type)?); - options.schema_infer_max_records = schema_infer_max_records; - options.file_extension = file_extension; - let df = if let Some(schema) = schema { - options.schema = Some(&schema.0); - let result = self.ctx.read_json(path, options); - wait_for_future(py, result)?? - } else { - let result = self.ctx.read_json(path, options); - wait_for_future(py, result)?? - }; + let options = build_json_options( + table_partition_cols, + file_compression_type, + schema_infer_max_records, + file_extension, + &schema, + )?; + let df = wait_for_future(py, self.ctx.read_json(path_to_str(&path)?, options))??; Ok(PyDataFrame::new(df)) } @@ -1038,23 +1252,18 @@ impl PySessionContext { options: Option<&PyCsvReadOptions>, py: Python, ) -> PyDataFusionResult { - let options = options - .map(|opts| opts.try_into()) - .transpose()? - .unwrap_or_default(); + let options = convert_csv_options(options)?; - if path.is_instance_of::() { - let paths = path.extract::>()?; - let paths = paths.iter().map(|p| p as &str).collect::>(); - let result = self.ctx.read_csv(paths, options); - let df = PyDataFrame::new(wait_for_future(py, result)??); - Ok(df) + let paths: Vec = if path.is_instance_of::() { + path.extract::>()? + .iter() + .map(|p| path_to_str(p).map(str::to_owned)) + .collect::>()? } else { - let path = path.extract::()?; - let result = self.ctx.read_csv(path, options); - let df = PyDataFrame::new(wait_for_future(py, result)??); - Ok(df) - } + vec![path_to_str(&path.extract::()?)?.to_owned()] + }; + let df = wait_for_future(py, self.ctx.read_csv(paths, options))??; + Ok(PyDataFrame::new(df)) } #[allow(clippy::too_many_arguments)] @@ -1068,7 +1277,7 @@ impl PySessionContext { file_sort_order=None))] pub fn read_parquet( &self, - path: &str, + path: PathBuf, table_partition_cols: Vec<(String, PyArrowType)>, parquet_pruning: bool, file_extension: &str, @@ -1077,25 +1286,18 @@ impl PySessionContext { file_sort_order: Option>>, py: Python, ) -> PyDataFusionResult { - let mut options = ParquetReadOptions::default() - .table_partition_cols( - table_partition_cols - .into_iter() - .map(|(name, ty)| (name, ty.0)) - .collect::>(), - ) - .parquet_pruning(parquet_pruning) - .skip_metadata(skip_metadata); - options.file_extension = file_extension; - options.schema = schema.as_ref().map(|x| &x.0); - options.file_sort_order = file_sort_order - .unwrap_or_default() - .into_iter() - .map(|e| e.into_iter().map(|f| f.into()).collect()) - .collect(); - - let result = self.ctx.read_parquet(path, options); - let df = PyDataFrame::new(wait_for_future(py, result)??); + let options = build_parquet_options( + table_partition_cols, + parquet_pruning, + file_extension, + skip_metadata, + &schema, + file_sort_order, + ); + let df = PyDataFrame::new(wait_for_future( + py, + self.ctx.read_parquet(path_to_str(&path)?, options), + )??); Ok(df) } @@ -1103,27 +1305,28 @@ impl PySessionContext { #[pyo3(signature = (path, schema=None, table_partition_cols=vec![], file_extension=".avro"))] pub fn read_avro( &self, - path: &str, + path: PathBuf, schema: Option>, table_partition_cols: Vec<(String, PyArrowType)>, file_extension: &str, py: Python, ) -> PyDataFusionResult { - let mut options = AvroReadOptions::default().table_partition_cols( - table_partition_cols - .into_iter() - .map(|(name, ty)| (name, ty.0)) - .collect::>(), - ); - options.file_extension = file_extension; - let df = if let Some(schema) = schema { - options.schema = Some(&schema.0); - let read_future = self.ctx.read_avro(path, options); - wait_for_future(py, read_future)?? - } else { - let read_future = self.ctx.read_avro(path, options); - wait_for_future(py, read_future)?? - }; + let options = build_avro_options(table_partition_cols, file_extension, &schema); + let df = wait_for_future(py, self.ctx.read_avro(path_to_str(&path)?, options))??; + Ok(PyDataFrame::new(df)) + } + + #[pyo3(signature = (path, schema=None, file_extension=".arrow", table_partition_cols=vec![]))] + pub fn read_arrow( + &self, + path: PathBuf, + schema: Option>, + file_extension: &str, + table_partition_cols: Vec<(String, PyArrowType)>, + py: Python, + ) -> PyDataFusionResult { + let options = build_arrow_options(table_partition_cols, file_extension, &schema); + let df = wait_for_future(py, self.ctx.read_arrow(path_to_str(&path)?, options))??; Ok(PyDataFrame::new(df)) } @@ -1168,35 +1371,72 @@ impl PySessionContext { &self, py: Python<'py>, ) -> PyResult> { - let name = cr"datafusion_task_context_provider".into(); - let ctx_provider = Arc::clone(&self.ctx) as Arc; let ffi_ctx_provider = FFI_TaskContextProvider::from(&ctx_provider); - PyCapsule::new(py, ffi_ctx_provider, Some(name)) + PyCapsule::new_with_value(py, ffi_ctx_provider, cr"datafusion_task_context_provider") } pub fn __datafusion_logical_extension_codec__<'py>( &self, py: Python<'py>, ) -> PyResult> { - create_logical_extension_capsule(py, self.logical_codec.as_ref()) + let ffi = self.ffi_logical_codec(); + create_logical_extension_capsule(py, ffi.as_ref()) } pub fn with_logical_extension_codec<'py>( &self, codec: Bound<'py, PyAny>, ) -> PyDataFusionResult { - let py = codec.py(); - let logical_codec = extract_logical_extension_codec(py, Some(codec))?; + let inner_ffi = ffi_logical_codec_from_pycapsule(codec)?; + let inner: Arc = (&inner_ffi).into(); + let logical_codec = Arc::new(PythonLogicalCodec::new(inner)); + + Ok(Self { + ctx: Arc::clone(&self.ctx), + logical_codec, + physical_codec: Arc::clone(&self.physical_codec), + }) + } - Ok({ - Self { - ctx: Arc::clone(&self.ctx), - logical_codec, - } + pub fn __datafusion_physical_extension_codec__<'py>( + &self, + py: Python<'py>, + ) -> PyResult> { + let ffi = self.ffi_physical_codec(); + create_physical_extension_capsule(py, ffi.as_ref()) + } + + pub fn with_physical_extension_codec<'py>( + &self, + codec: Bound<'py, PyAny>, + ) -> PyDataFusionResult { + let inner = physical_codec_from_pycapsule(&codec)?; + let physical_codec = Arc::new(PythonPhysicalCodec::new(inner)); + + Ok(Self { + ctx: Arc::clone(&self.ctx), + logical_codec: Arc::clone(&self.logical_codec), + physical_codec, }) } + + pub fn with_python_udf_inlining(&self, enabled: bool) -> Self { + let logical_codec = Arc::new( + PythonLogicalCodec::new(Arc::clone(self.logical_codec.inner())) + .with_python_udf_inlining(enabled), + ); + let physical_codec = Arc::new( + PythonPhysicalCodec::new(Arc::clone(self.physical_codec.inner())) + .with_python_udf_inlining(enabled), + ); + Self { + ctx: Arc::clone(&self.ctx), + logical_codec, + physical_codec, + } + } } impl PySessionContext { @@ -1224,7 +1464,7 @@ impl PySessionContext { // check if the file extension matches the expected extension for path in &table_paths { let file_path = path.as_str(); - if !file_path.ends_with(option_extension.clone().as_str()) && !path.is_collection() { + if !file_path.ends_with(option_extension.as_str()) && !path.is_collection() { return exec_err!( "File path '{file_path}' does not match the expected extension '{option_extension}'" ); @@ -1244,12 +1484,42 @@ impl PySessionContext { Ok(()) } - fn default_logical_codec(ctx: &Arc) -> Arc { - let codec = Arc::new(DefaultLogicalExtensionCodec {}); - let runtime = get_tokio_runtime().0.handle().clone(); - let ctx_provider = Arc::clone(ctx) as Arc; + /// Session-scoped logical codec. Sibling modules read this when they + /// need to serialize/deserialize logical-layer types (LogicalPlan, + /// Expr) against the user-installed (or default) codec stack. + pub(crate) fn logical_codec(&self) -> &Arc { + &self.logical_codec + } + + /// Session-scoped physical codec. Sibling modules read this for + /// ExecutionPlan / PhysicalExpr serialization. + pub(crate) fn physical_codec(&self) -> &Arc { + &self.physical_codec + } + + /// Build an FFI-wrapped clone of the session's logical codec on demand. + /// Used at every site that exports the codec across an FFI boundary + /// (capsule getters, Rust wrappers for Python-defined providers, etc.). + pub(crate) fn ffi_logical_codec(&self) -> Arc { + let inner: Arc = + Arc::clone(&self.logical_codec) as Arc; + let runtime = get_tokio_runtime().handle().clone(); + let ctx_provider = Arc::clone(&self.ctx) as Arc; Arc::new(FFI_LogicalExtensionCodec::new( - codec, + inner, + Some(runtime), + &ctx_provider, + )) + } + + /// Build an FFI-wrapped clone of the session's physical codec on demand. + pub(crate) fn ffi_physical_codec(&self) -> Arc { + let inner: Arc = + Arc::clone(&self.physical_codec) as Arc; + let runtime = get_tokio_runtime().handle().clone(); + let ctx_provider = Arc::clone(&self.ctx) as Arc; + Arc::new(FFI_PhysicalExtensionCodec::new( + inner, Some(runtime), &ctx_provider, )) @@ -1259,10 +1529,100 @@ impl PySessionContext { pub fn parse_file_compression_type( file_compression_type: Option, ) -> Result { - FileCompressionType::from_str(&*file_compression_type.unwrap_or("".to_string()).as_str()) - .map_err(|_| { - PyValueError::new_err("file_compression_type must one of: gzip, bz2, xz, zstd") - }) + FileCompressionType::from_str(&file_compression_type.unwrap_or_default()).map_err(|_| { + PyValueError::new_err("file_compression_type must be one of: gzip, bz2, xz, zstd") + }) +} + +fn path_to_str(path: &Path) -> PyDataFusionResult<&str> { + path.to_str() + .ok_or_else(|| PyValueError::new_err("Unable to convert path to a string").into()) +} + +fn convert_csv_options( + options: Option<&PyCsvReadOptions>, +) -> PyDataFusionResult> { + Ok(options + .map(|opts| opts.try_into()) + .transpose()? + .unwrap_or_default()) +} + +fn convert_partition_cols( + table_partition_cols: Vec<(String, PyArrowType)>, +) -> Vec<(String, DataType)> { + table_partition_cols + .into_iter() + .map(|(name, ty)| (name, ty.0)) + .collect() +} + +fn convert_file_sort_order( + file_sort_order: Option>>, +) -> Vec> { + file_sort_order + .unwrap_or_default() + .into_iter() + .map(|e| e.into_iter().map(|f| f.into()).collect()) + .collect() +} + +fn build_parquet_options<'a>( + table_partition_cols: Vec<(String, PyArrowType)>, + parquet_pruning: bool, + file_extension: &'a str, + skip_metadata: bool, + schema: &'a Option>, + file_sort_order: Option>>, +) -> ParquetReadOptions<'a> { + let mut options = ParquetReadOptions::default() + .table_partition_cols(convert_partition_cols(table_partition_cols)) + .parquet_pruning(parquet_pruning) + .skip_metadata(skip_metadata); + options.file_extension = file_extension; + options.schema = schema.as_ref().map(|x| &x.0); + options.file_sort_order = convert_file_sort_order(file_sort_order); + options +} + +fn build_json_options<'a>( + table_partition_cols: Vec<(String, PyArrowType)>, + file_compression_type: Option, + schema_infer_max_records: usize, + file_extension: &'a str, + schema: &'a Option>, +) -> Result, PyErr> { + let mut options = JsonReadOptions::default() + .table_partition_cols(convert_partition_cols(table_partition_cols)) + .file_compression_type(parse_file_compression_type(file_compression_type)?); + options.schema_infer_max_records = schema_infer_max_records; + options.file_extension = file_extension; + options.schema = schema.as_ref().map(|x| &x.0); + Ok(options) +} + +fn build_arrow_options<'a>( + table_partition_cols: Vec<(String, PyArrowType)>, + file_extension: &'a str, + schema: &'a Option>, +) -> ArrowReadOptions<'a> { + let mut options = ArrowReadOptions::default() + .table_partition_cols(convert_partition_cols(table_partition_cols)); + options.file_extension = file_extension; + options.schema = schema.as_ref().map(|x| &x.0); + options +} + +fn build_avro_options<'a>( + table_partition_cols: Vec<(String, PyArrowType)>, + file_extension: &'a str, + schema: &'a Option>, +) -> AvroReadOptions<'a> { + let mut options = AvroReadOptions::default() + .table_partition_cols(convert_partition_cols(table_partition_cols)); + options.file_extension = file_extension; + options.schema = schema.as_ref().map(|x| &x.0); + options } impl From for SessionContext { @@ -1273,9 +1633,10 @@ impl From for SessionContext { impl From for PySessionContext { fn from(ctx: SessionContext) -> PySessionContext { - let ctx = Arc::new(ctx); - let logical_codec = Self::default_logical_codec(&ctx); - - PySessionContext { ctx, logical_codec } + PySessionContext { + ctx: Arc::new(ctx), + logical_codec: Arc::new(PythonLogicalCodec::default()), + physical_codec: Arc::new(PythonPhysicalCodec::default()), + } } } diff --git a/src/dataframe.rs b/crates/core/src/dataframe.rs similarity index 86% rename from src/dataframe.rs rename to crates/core/src/dataframe.rs index eb1fa4a81..b1f305551 100644 --- a/src/dataframe.rs +++ b/crates/core/src/dataframe.rs @@ -16,7 +16,7 @@ // under the License. use std::collections::HashMap; -use std::ffi::{CStr, CString}; +use std::ffi::CStr; use std::ptr::NonNull; use std::str::FromStr; use std::sync::Arc; @@ -37,15 +37,21 @@ use datafusion::config::{CsvOptions, ParquetColumnOptions, ParquetOptions, Table use datafusion::dataframe::{DataFrame, DataFrameWriteOptions}; use datafusion::error::DataFusionError; use datafusion::execution::SendableRecordBatchStream; -use datafusion::logical_expr::SortExpr; +use datafusion::execution::context::TaskContext; use datafusion::logical_expr::dml::InsertOp; +use datafusion::logical_expr::{LogicalPlan, SortExpr}; use datafusion::parquet::basic::{BrotliLevel, Compression, GzipLevel, ZstdLevel}; +use datafusion::physical_plan::{ + ExecutionPlan as DFExecutionPlan, collect as df_collect, + collect_partitioned as df_collect_partitioned, execute_stream as df_execute_stream, + execute_stream_partitioned as df_execute_stream_partitioned, +}; use datafusion::prelude::*; +use datafusion_python_util::{is_ipython_env, spawn_future, wait_for_future}; use futures::{StreamExt, TryStreamExt}; use parking_lot::Mutex; use pyo3::PyErr; use pyo3::exceptions::PyValueError; -use pyo3::ffi::c_str; use pyo3::prelude::*; use pyo3::pybacked::PyBackedStr; use pyo3::types::{PyCapsule, PyList, PyTuple, PyTupleMethods}; @@ -58,7 +64,6 @@ use crate::physical_plan::PyExecutionPlan; use crate::record_batch::{PyRecordBatchStream, poll_next_batch}; use crate::sql::logical::PyLogicalPlan; use crate::table::{PyTable, TempViewTable}; -use crate::utils::{is_ipython_env, spawn_future, validate_pycapsule, wait_for_future}; /// File-level static CStr for the Arrow array stream capsule name. static ARROW_ARRAY_STREAM_NAME: &CStr = cstr!("arrow_array_stream"); @@ -309,6 +314,9 @@ pub struct PyDataFrame { // In IPython environment cache batches between __repr__ and _repr_html_ calls. batches: SharedCachedBatches, + + // Cache the last physical plan so that metrics are available after execution. + last_plan: Arc>>>, } impl PyDataFrame { @@ -317,6 +325,7 @@ impl PyDataFrame { Self { df: Arc::new(df), batches: Arc::new(Mutex::new(None)), + last_plan: Arc::new(Mutex::new(None)), } } @@ -388,6 +397,20 @@ impl PyDataFrame { Ok(html_str) } + /// Create the physical plan, cache it in `last_plan`, and return the plan together + /// with a task context. Centralises the repeated three-line pattern that appears in + /// `collect`, `collect_partitioned`, `execute_stream`, and `execute_stream_partitioned`. + fn create_and_cache_plan( + &self, + py: Python, + ) -> PyDataFusionResult<(Arc, Arc)> { + let df = self.df.as_ref().clone(); + let new_plan = wait_for_future(py, df.create_physical_plan())??; + *self.last_plan.lock() = Some(Arc::clone(&new_plan)); + let task_ctx = Arc::new(self.df.as_ref().task_ctx()); + Ok((new_plan, task_ctx)) + } + async fn collect_column_inner(&self, column: &str) -> Result { let batches = self .df @@ -469,17 +492,17 @@ impl PyDataFrame { fn __getitem__(&self, key: Bound<'_, PyAny>) -> PyDataFusionResult { if let Ok(key) = key.extract::() { // df[col] - self.select_columns(vec![key]) + self.select_exprs(vec![key]) } else if let Ok(tuple) = key.cast::() { // df[col1, col2, col3] let keys = tuple .iter() .map(|item| item.extract::()) .collect::>>()?; - self.select_columns(keys) + self.select_exprs(keys) } else if let Ok(keys) = key.extract::>() { // df[[col1, col2, col3]] - self.select_columns(keys) + self.select_exprs(keys) } else { let message = "DataFrame can only be indexed by string index or indices".to_string(); Err(PyDataFusionError::Common(message)) @@ -555,10 +578,8 @@ impl PyDataFrame { Ok(PyTable::from(table_provider)) } - #[pyo3(signature = (*args))] - fn select_columns(&self, args: Vec) -> PyDataFusionResult { - let args = args.iter().map(|s| s.as_ref()).collect::>(); - let df = self.df.as_ref().clone().select_columns(&args)?; + fn alias(&self, alias: &str) -> PyDataFusionResult { + let df = self.df.as_ref().clone().alias(alias)?; Ok(Self::new(df)) } @@ -583,6 +604,14 @@ impl PyDataFrame { Ok(Self::new(df)) } + /// Apply window function expressions to the DataFrame + #[pyo3(signature = (*exprs))] + fn window(&self, exprs: Vec) -> PyDataFusionResult { + let window_exprs = exprs.into_iter().map(|e| e.into()).collect(); + let df = self.df.as_ref().clone().window(window_exprs)?; + Ok(Self::new(df)) + } + fn filter(&self, predicate: PyExpr) -> PyDataFusionResult { let df = self.df.as_ref().clone().filter(predicate.into())?; Ok(Self::new(df)) @@ -646,8 +675,9 @@ impl PyDataFrame { /// Unless some order is specified in the plan, there is no /// guarantee of the order of the result. fn collect<'py>(&self, py: Python<'py>) -> PyResult>> { - let batches = wait_for_future(py, self.df.as_ref().clone().collect())? - .map_err(PyDataFusionError::from)?; + let (plan, task_ctx) = self.create_and_cache_plan(py)?; + let batches = + wait_for_future(py, df_collect(plan, task_ctx))?.map_err(PyDataFusionError::from)?; // cannot use PyResult> return type due to // https://github.com/PyO3/pyo3/issues/1813 batches.into_iter().map(|rb| rb.to_pyarrow(py)).collect() @@ -662,7 +692,8 @@ impl PyDataFrame { /// Executes this DataFrame and collects all results into a vector of vector of RecordBatch /// maintaining the input partitioning. fn collect_partitioned<'py>(&self, py: Python<'py>) -> PyResult>>> { - let batches = wait_for_future(py, self.df.as_ref().clone().collect_partitioned())? + let (plan, task_ctx) = self.create_and_cache_plan(py)?; + let batches = wait_for_future(py, df_collect_partitioned(plan, task_ctx))? .map_err(PyDataFusionError::from)?; batches @@ -681,7 +712,15 @@ impl PyDataFrame { /// Print the result, 20 lines by default #[pyo3(signature = (num=20))] fn show(&self, py: Python, num: usize) -> PyDataFusionResult<()> { - let df = self.df.as_ref().clone().limit(0, Some(num))?; + let mut df = self.df.as_ref().clone(); + df = match self.df.logical_plan() { + LogicalPlan::Explain(_) | LogicalPlan::Analyze(_) => { + // Explain and Analyzer require they are at the top + // of the plan, so do not add a limit. + df + } + _ => df.limit(0, Some(num))?, + }; print_dataframe(py, df) } @@ -805,9 +844,27 @@ impl PyDataFrame { } /// Print the query plan - #[pyo3(signature = (verbose=false, analyze=false))] - fn explain(&self, py: Python, verbose: bool, analyze: bool) -> PyDataFusionResult<()> { - let df = self.df.as_ref().clone().explain(verbose, analyze)?; + #[pyo3(signature = (verbose=false, analyze=false, format=None))] + fn explain( + &self, + py: Python, + verbose: bool, + analyze: bool, + format: Option<&str>, + ) -> PyDataFusionResult<()> { + let explain_format = match format { + Some(f) => f + .parse::() + .map_err(|e| { + PyDataFusionError::Common(format!("Invalid explain format '{}': {}", f, e)) + })?, + None => datafusion::common::format::ExplainFormat::Indent, + }; + let opts = datafusion::logical_expr::ExplainOption::default() + .with_verbose(verbose) + .with_analyze(analyze) + .with_format(explain_format); + let df = self.df.as_ref().clone().explain_with_options(opts)?; print_dataframe(py, df) } @@ -822,7 +879,13 @@ impl PyDataFrame { } /// Get the execution plan for this `DataFrame` + /// + /// If the DataFrame has already been executed (e.g. via `collect()`), + /// returns the cached plan which includes populated metrics. fn execution_plan(&self, py: Python) -> PyDataFusionResult { + if let Some(plan) = self.last_plan.lock().as_ref() { + return Ok(PyExecutionPlan::new(Arc::clone(plan))); + } let plan = wait_for_future(py, self.df.as_ref().clone().create_physical_plan())??; Ok(plan.into()) } @@ -865,39 +928,14 @@ impl PyDataFrame { Ok(Self::new(new_df)) } - /// Calculate the distinct union of two `DataFrame`s. The - /// two `DataFrame`s must have exactly the same schema - fn union_distinct(&self, py_df: PyDataFrame) -> PyDataFusionResult { - let new_df = self - .df - .as_ref() - .clone() - .union_distinct(py_df.df.as_ref().clone())?; - Ok(Self::new(new_df)) - } - - #[pyo3(signature = (column, preserve_nulls=true))] - fn unnest_column(&self, column: &str, preserve_nulls: bool) -> PyDataFusionResult { - // TODO: expose RecursionUnnestOptions - // REF: https://github.com/apache/datafusion/pull/11577 - let unnest_options = UnnestOptions::default().with_preserve_nulls(preserve_nulls); - let df = self - .df - .as_ref() - .clone() - .unnest_columns_with_options(&[column], unnest_options)?; - Ok(Self::new(df)) - } - - #[pyo3(signature = (columns, preserve_nulls=true))] + #[pyo3(signature = (columns, preserve_nulls=true, recursions=None))] fn unnest_columns( &self, columns: Vec, preserve_nulls: bool, + recursions: Option>, ) -> PyDataFusionResult { - // TODO: expose RecursionUnnestOptions - // REF: https://github.com/apache/datafusion/pull/11577 - let unnest_options = UnnestOptions::default().with_preserve_nulls(preserve_nulls); + let unnest_options = build_unnest_options(preserve_nulls, recursions); let cols = columns.iter().map(|s| s.as_ref()).collect::>(); let df = self .df @@ -908,21 +946,79 @@ impl PyDataFrame { } /// Calculate the intersection of two `DataFrame`s. The two `DataFrame`s must have exactly the same schema - fn intersect(&self, py_df: PyDataFrame) -> PyDataFusionResult { - let new_df = self - .df - .as_ref() - .clone() - .intersect(py_df.df.as_ref().clone())?; + #[pyo3(signature = (py_df, distinct=false))] + fn intersect(&self, py_df: PyDataFrame, distinct: bool) -> PyDataFusionResult { + let base = self.df.as_ref().clone(); + let other = py_df.df.as_ref().clone(); + let new_df = if distinct { + base.intersect_distinct(other)? + } else { + base.intersect(other)? + }; Ok(Self::new(new_df)) } /// Calculate the exception of two `DataFrame`s. The two `DataFrame`s must have exactly the same schema - fn except_all(&self, py_df: PyDataFrame) -> PyDataFusionResult { - let new_df = self.df.as_ref().clone().except(py_df.df.as_ref().clone())?; + #[pyo3(signature = (py_df, distinct=false))] + fn except_all(&self, py_df: PyDataFrame, distinct: bool) -> PyDataFusionResult { + let base = self.df.as_ref().clone(); + let other = py_df.df.as_ref().clone(); + let new_df = if distinct { + base.except_distinct(other)? + } else { + base.except(other)? + }; Ok(Self::new(new_df)) } + /// Union two DataFrames matching columns by name + #[pyo3(signature = (py_df, distinct=false))] + fn union_by_name(&self, py_df: PyDataFrame, distinct: bool) -> PyDataFusionResult { + let base = self.df.as_ref().clone(); + let other = py_df.df.as_ref().clone(); + let new_df = if distinct { + base.union_by_name_distinct(other)? + } else { + base.union_by_name(other)? + }; + Ok(Self::new(new_df)) + } + + /// Deduplicate rows based on specific columns, keeping the first row per group + fn distinct_on( + &self, + on_expr: Vec, + select_expr: Vec, + sort_expr: Option>, + ) -> PyDataFusionResult { + let on_expr = on_expr.into_iter().map(|e| e.into()).collect(); + let select_expr = select_expr.into_iter().map(|e| e.into()).collect(); + let sort_expr = sort_expr.map(to_sort_expressions); + let df = self + .df + .as_ref() + .clone() + .distinct_on(on_expr, select_expr, sort_expr)?; + Ok(Self::new(df)) + } + + /// Sort by column expressions with ascending order and nulls last + fn sort_by(&self, exprs: Vec) -> PyDataFusionResult { + let exprs = exprs.into_iter().map(|e| e.into()).collect(); + let df = self.df.as_ref().clone().sort_by(exprs)?; + Ok(Self::new(df)) + } + + /// Return fully qualified column expressions for the given column names + fn find_qualified_columns(&self, names: Vec) -> PyDataFusionResult> { + let name_refs: Vec<&str> = names.iter().map(|s| s.as_str()).collect(); + let qualified = self.df.find_qualified_columns(&name_refs)?; + Ok(qualified + .into_iter() + .map(|q| Expr::Column(Column::from(q)).into()) + .collect()) + } + /// Write a `DataFrame` to a CSV file. fn write_csv( &self, @@ -1117,10 +1213,8 @@ impl PyDataFrame { let mut projection: Option = None; if let Some(schema_capsule) = requested_schema { - validate_pycapsule(&schema_capsule, "arrow_schema")?; - let data: NonNull = schema_capsule - .pointer_checked(Some(c_str!("arrow_schema")))? + .pointer_checked(Some(c"arrow_schema"))? .cast(); let schema_ptr = unsafe { data.as_ref() }; let desired_schema = Schema::try_from(schema_ptr)?; @@ -1143,20 +1237,22 @@ impl PyDataFrame { // destructor provided by PyO3 will drop the stream unless ownership is // transferred to PyArrow during import. let stream = FFI_ArrowArrayStream::new(reader); - let name = CString::new(ARROW_ARRAY_STREAM_NAME.to_bytes()).unwrap(); - let capsule = PyCapsule::new(py, stream, Some(name))?; + let capsule = PyCapsule::new_with_value(py, stream, ARROW_ARRAY_STREAM_NAME)?; Ok(capsule) } fn execute_stream(&self, py: Python) -> PyDataFusionResult { - let df = self.df.as_ref().clone(); - let stream = spawn_future(py, async move { df.execute_stream().await })?; + let (plan, task_ctx) = self.create_and_cache_plan(py)?; + let stream = spawn_future(py, async move { df_execute_stream(plan, task_ctx) })?; Ok(PyRecordBatchStream::new(stream)) } fn execute_stream_partitioned(&self, py: Python) -> PyResult> { - let df = self.df.as_ref().clone(); - let streams = spawn_future(py, async move { df.execute_stream_partitioned().await })?; + let (plan, task_ctx) = self.create_and_cache_plan(py)?; + let streams = spawn_future( + py, + async move { df_execute_stream_partitioned(plan, task_ctx) }, + )?; Ok(streams.into_iter().map(PyRecordBatchStream::new).collect()) } @@ -1220,7 +1316,8 @@ impl PyDataFrame { None => Vec::new(), // Empty vector means fill null for all columns }; - let df = self.df.as_ref().clone().fill_null(scalar_value.0, cols)?; + let cols = cols.iter().map(String::as_str).collect::>(); + let df = self.df.as_ref().fill_null(&scalar_value.0, &cols)?; Ok(Self::new(df)) } } @@ -1298,6 +1395,26 @@ impl PyDataFrameWriteOptions { } } +fn build_unnest_options( + preserve_nulls: bool, + recursions: Option>, +) -> UnnestOptions { + let mut opts = UnnestOptions::default().with_preserve_nulls(preserve_nulls); + if let Some(recs) = recursions { + opts.recursions = recs + .into_iter() + .map( + |(input, output, depth)| datafusion::common::RecursionUnnestOption { + input_column: datafusion::common::Column::from(input.as_str()), + output_column: datafusion::common::Column::from(output.as_str()), + depth, + }, + ) + .collect(); + } + opts +} + /// Print DataFrame fn print_dataframe(py: Python, df: DataFrame) -> PyDataFusionResult<()> { // Get string representation of record batches diff --git a/src/dataset.rs b/crates/core/src/dataset.rs similarity index 95% rename from src/dataset.rs rename to crates/core/src/dataset.rs index dbeafcd9f..2a5770338 100644 --- a/src/dataset.rs +++ b/crates/core/src/dataset.rs @@ -15,7 +15,6 @@ // specific language governing permissions and limitations // under the License. -use std::any::Any; use std::sync::Arc; use async_trait::async_trait; @@ -62,12 +61,6 @@ impl Dataset { #[async_trait] impl TableProvider for Dataset { - /// Returns the table provider as [`Any`](std::any::Any) so that it can be - /// downcast to a specific implementation. - fn as_any(&self) -> &dyn Any { - self - } - /// Get a reference to the schema for this table fn schema(&self) -> SchemaRef { Python::attach(|py| { diff --git a/src/dataset_exec.rs b/crates/core/src/dataset_exec.rs similarity index 95% rename from src/dataset_exec.rs rename to crates/core/src/dataset_exec.rs index e3c058c07..963603c8b 100644 --- a/src/dataset_exec.rs +++ b/crates/core/src/dataset_exec.rs @@ -15,18 +15,18 @@ // specific language governing permissions and limitations // under the License. -use std::any::Any; use std::sync::Arc; use datafusion::arrow::datatypes::SchemaRef; use datafusion::arrow::error::{ArrowError, Result as ArrowResult}; use datafusion::arrow::pyarrow::PyArrowType; use datafusion::arrow::record_batch::RecordBatch; +use datafusion::common::tree_node::TreeNodeRecursion; use datafusion::error::{DataFusionError as InnerDataFusionError, Result as DFResult}; use datafusion::execution::context::TaskContext; use datafusion::logical_expr::Expr; use datafusion::logical_expr::utils::conjunction; -use datafusion::physical_expr::{EquivalenceProperties, LexOrdering}; +use datafusion::physical_expr::{EquivalenceProperties, LexOrdering, PhysicalExpr}; use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType}; use datafusion::physical_plan::stream::RecordBatchStreamAdapter; use datafusion::physical_plan::{ @@ -111,7 +111,7 @@ impl DatasetExec { let scanner = dataset.call_method("scanner", (), Some(&kwargs))?; - let schema = Arc::new( + let schema: SchemaRef = Arc::new( scanner .getattr("projected_schema")? .extract::>()? @@ -156,11 +156,6 @@ impl ExecutionPlan for DatasetExec { Self::static_name() } - /// Return a reference to Any that can be used for downcasting - fn as_any(&self) -> &dyn Any { - self - } - /// Get the schema for this execution plan fn schema(&self) -> SchemaRef { self.schema.clone() @@ -171,6 +166,15 @@ impl ExecutionPlan for DatasetExec { vec![] } + fn apply_expressions( + &self, + _f: &mut dyn FnMut(&Arc) -> DFResult, + ) -> DFResult { + // Any filters are pushed down into the PyArrow dataset scanner as pyarrow + // expressions, so this node owns no physical expressions to visit. + Ok(TreeNodeRecursion::Continue) + } + fn with_new_children( self: Arc, _: Vec>, @@ -235,8 +239,8 @@ impl ExecutionPlan for DatasetExec { }) } - fn partition_statistics(&self, _partition: Option) -> DFResult { - Ok(self.projected_statistics.clone()) + fn partition_statistics(&self, _partition: Option) -> DFResult> { + Ok(Arc::new(self.projected_statistics.clone())) } fn properties(&self) -> &Arc { diff --git a/src/expr/grouping_set.rs b/crates/core/src/errors.rs similarity index 59% rename from src/expr/grouping_set.rs rename to crates/core/src/errors.rs index 549a866ed..8babc5a56 100644 --- a/src/expr/grouping_set.rs +++ b/crates/core/src/errors.rs @@ -15,29 +15,4 @@ // specific language governing permissions and limitations // under the License. -use datafusion::logical_expr::GroupingSet; -use pyo3::prelude::*; - -#[pyclass( - from_py_object, - frozen, - name = "GroupingSet", - module = "datafusion.expr", - subclass -)] -#[derive(Clone)] -pub struct PyGroupingSet { - grouping_set: GroupingSet, -} - -impl From for GroupingSet { - fn from(grouping_set: PyGroupingSet) -> Self { - grouping_set.grouping_set - } -} - -impl From for PyGroupingSet { - fn from(grouping_set: GroupingSet) -> PyGroupingSet { - PyGroupingSet { grouping_set } - } -} +pub use datafusion_python_util::errors::*; diff --git a/src/expr.rs b/crates/core/src/expr.rs similarity index 88% rename from src/expr.rs rename to crates/core/src/expr.rs index c4f2a12da..cab997d7a 100644 --- a/src/expr.rs +++ b/crates/core/src/expr.rs @@ -23,17 +23,21 @@ use datafusion::arrow::datatypes::{DataType, Field}; use datafusion::arrow::pyarrow::PyArrowType; use datafusion::functions::core::expr_ext::FieldAccessor; use datafusion::logical_expr::expr::{ - AggregateFunction, AggregateFunctionParams, FieldMetadata, InList, InSubquery, ScalarFunction, - SetComparison, WindowFunction, + AggregateFunction, AggregateFunctionParams, FieldMetadata, HigherOrderFunction, InList, + InSubquery, Lambda, ScalarFunction, SetComparison, WindowFunction, }; use datafusion::logical_expr::utils::exprlist_to_fields; use datafusion::logical_expr::{ Between, BinaryExpr, Case, Cast, Expr, ExprFuncBuilder, ExprFunctionExt, Like, LogicalPlan, Operator, TryCast, WindowFunctionDefinition, col, lit, lit_with_metadata, }; +use datafusion_proto::logical_plan::{from_proto, to_proto}; +use prost::Message; use pyo3::IntoPyObjectExt; use pyo3::basic::CompareOp; +use pyo3::exceptions::PyRuntimeError; use pyo3::prelude::*; +use pyo3::types::PyBytes; use window::PyWindowFrame; use self::alias::PyAlias; @@ -43,7 +47,9 @@ use self::bool_expr::{ }; use self::like::{PyILike, PyLike, PySimilarTo}; use self::scalar_variable::PyScalarVariable; +use crate::codec::PythonLogicalCodec; use crate::common::data_type::{DataTypeMap, NullTreatment, PyScalarValue, RexType}; +use crate::context::PySessionContext; use crate::errors::{PyDataFusionResult, py_runtime_err, py_type_err, py_unsupported_variant_err}; use crate::expr::aggregate_expr::PyAggregateFunction; use crate::expr::binary_expr::PyBinaryExpr; @@ -85,9 +91,12 @@ pub mod explain; pub mod extension; pub mod filter; pub mod grouping_set; +pub mod higher_order_function; pub mod in_list; pub mod in_subquery; pub mod join; +pub mod lambda; +pub mod lambda_variable; pub mod like; pub mod limit; pub mod literal; @@ -220,6 +229,14 @@ impl PyExpr { Expr::SetComparison(value) => { Ok(set_comparison::PySetComparison::from(value.clone()).into_bound_py_any(py)?) } + Expr::HigherOrderFunction(value) => Ok( + higher_order_function::PyHigherOrderFunction::from(value.clone()) + .into_bound_py_any(py)?, + ), + Expr::Lambda(value) => Ok(lambda::PyLambda::from(value.clone()).into_bound_py_any(py)?), + Expr::LambdaVariable(value) => { + Ok(lambda_variable::PyLambdaVariable::from(value.clone()).into_bound_py_any(py)?) + } }) } @@ -341,6 +358,11 @@ impl PyExpr { expr.into() } + pub fn try_cast(&self, to: PyArrowType) -> PyExpr { + let expr = Expr::TryCast(TryCast::new(Box::new(self.expr.clone()), to.0)); + expr.into() + } + #[pyo3(signature = (low, high, negated=false))] pub fn between(&self, low: PyExpr, high: PyExpr, negated: bool) -> PyExpr { let expr = Expr::Between(Between::new( @@ -387,7 +409,10 @@ impl PyExpr { | Expr::OuterReferenceColumn(_, _) | Expr::Unnest(_) | Expr::IsNotUnknown(_) - | Expr::SetComparison(_) => RexType::Call, + | Expr::SetComparison(_) + | Expr::HigherOrderFunction(..) + | Expr::Lambda(..) => RexType::Call, + Expr::LambdaVariable(..) => RexType::Reference, Expr::ScalarSubquery(..) => RexType::ScalarSubquery, #[allow(deprecated)] Expr::Wildcard { .. } => { @@ -408,7 +433,7 @@ impl PyExpr { Expr::Literal(scalar_value, _) => scalar_to_pyarrow(scalar_value, py), _ => Err(py_type_err(format!( "Non Expr::Literal encountered in types: {:?}", - &self.expr + self.expr ))), } } @@ -419,9 +444,10 @@ impl PyExpr { pub fn rex_call_operands(&self) -> PyResult> { match &self.expr { // Expr variants that are themselves the operand to return - Expr::Column(..) | Expr::ScalarVariable(..) | Expr::Literal(..) => { - Ok(vec![PyExpr::from(self.expr.clone())]) - } + Expr::Column(..) + | Expr::ScalarVariable(..) + | Expr::Literal(..) + | Expr::LambdaVariable(..) => Ok(vec![PyExpr::from(self.expr.clone())]), Expr::Alias(alias) => Ok(vec![PyExpr::from(*alias.expr.clone())]), @@ -448,13 +474,15 @@ impl PyExpr { params: AggregateFunctionParams { args, .. }, .. }) - | Expr::ScalarFunction(ScalarFunction { args, .. }) => { + | Expr::ScalarFunction(ScalarFunction { args, .. }) + | Expr::HigherOrderFunction(HigherOrderFunction { args, .. }) => { Ok(args.iter().map(|arg| PyExpr::from(arg.clone())).collect()) } Expr::WindowFunction(boxed_window_fn) => { let args = &boxed_window_fn.params.args; Ok(args.iter().map(|arg| PyExpr::from(arg.clone())).collect()) } + Expr::Lambda(Lambda { body, .. }) => Ok(vec![PyExpr::from(*body.clone())]), // Expr(s) that require more specific processing Expr::Case(Case { @@ -544,6 +572,10 @@ impl PyExpr { right: _, }) => format!("{op}"), Expr::ScalarFunction(ScalarFunction { func, args: _ }) => func.name().to_string(), + Expr::HigherOrderFunction(HigherOrderFunction { func, args: _ }) => { + func.name().to_string() + } + Expr::Lambda(..) => "lambda".to_string(), Expr::Cast { .. } => "cast".to_string(), Expr::Between { .. } => "between".to_string(), Expr::Case { .. } => "case".to_string(), @@ -580,7 +612,7 @@ impl PyExpr { _ => { return Err(py_type_err(format!( "Catch all triggered in get_operator_name: {:?}", - &self.expr + self.expr ))); } }) @@ -660,6 +692,55 @@ impl PyExpr { .into()), } } + + /// Serialize this `Expr` to protobuf bytes. + /// + /// When `ctx` is supplied, encoding routes through the session's + /// installed `LogicalExtensionCodec` so user FFI codecs see the + /// encode path. Without `ctx` a default-inner Python codec is + /// used; Python scalar UDFs still inline when in-band encoding + /// lands, non-Python UDFs fall through to the default codec. + #[pyo3(signature = (ctx=None))] + pub fn to_bytes<'py>( + &'py self, + py: Python<'py>, + ctx: Option, + ) -> PyDataFusionResult> { + let default_codec; + let codec: &dyn datafusion_proto::logical_plan::LogicalExtensionCodec = match ctx { + Some(ref ctx) => ctx.logical_codec().as_ref(), + None => { + default_codec = PythonLogicalCodec::default(); + &default_codec + } + }; + let proto = to_proto::serialize_expr(&self.expr, codec) + .map_err(|e| PyRuntimeError::new_err(format!("Unable to serialize expr: {e}")))?; + let bytes = proto.encode_to_vec(); + Ok(PyBytes::new(py, &bytes)) + } + + /// Decode an `Expr` from protobuf bytes against the session's + /// function registry and logical codec. + #[staticmethod] + pub fn from_bytes( + ctx: PySessionContext, + proto_msg: Bound<'_, PyBytes>, + ) -> PyDataFusionResult { + let bytes: &[u8] = proto_msg.extract().map_err(Into::::into)?; + let proto_expr = + datafusion_proto::protobuf::LogicalExprNode::decode(bytes).map_err(|e| { + PyRuntimeError::new_err(format!( + "Unable to decode expression from serialized bytes: {e}" + )) + })?; + + let codec = ctx.logical_codec(); + let task_ctx = ctx.ctx.task_ctx(); + let expr = from_proto::parse_expr(&proto_expr, task_ctx.as_ref(), codec.as_ref()) + .map_err(|e| PyRuntimeError::new_err(format!("Unable to decode expr: {e}")))?; + Ok(Self { expr }) + } } #[pyclass( @@ -782,7 +863,9 @@ impl PyExpr { | Operator::QuestionPipe | Operator::Colon => Err(py_type_err(format!("Unsupported expr: ${op}"))), }, - Expr::Cast(Cast { expr: _, data_type }) => DataTypeMap::map_from_arrow_type(data_type), + Expr::Cast(Cast { expr: _, field }) => { + DataTypeMap::map_from_arrow_type(field.data_type()) + } Expr::Literal(scalar_value, _) => DataTypeMap::map_from_scalar_value(scalar_value), _ => Err(py_type_err(format!( "Non Expr::Literal encountered in types: {expr:?}" @@ -838,6 +921,9 @@ pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; m.add_class::()?; + m.add_class::()?; + m.add_class::()?; + m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; diff --git a/src/expr/aggregate.rs b/crates/core/src/expr/aggregate.rs similarity index 98% rename from src/expr/aggregate.rs rename to crates/core/src/expr/aggregate.rs index 5a6a771a7..7177fb469 100644 --- a/src/expr/aggregate.rs +++ b/crates/core/src/expr/aggregate.rs @@ -65,8 +65,8 @@ impl Display for PyAggregate { \nAggregates(s): {:?} \nInput: {:?} \nProjected Schema: {:?}", - &self.aggregate.group_expr, - &self.aggregate.aggr_expr, + self.aggregate.group_expr, + self.aggregate.aggr_expr, self.aggregate.input, self.aggregate.schema ) diff --git a/src/expr/aggregate_expr.rs b/crates/core/src/expr/aggregate_expr.rs similarity index 100% rename from src/expr/aggregate_expr.rs rename to crates/core/src/expr/aggregate_expr.rs diff --git a/src/expr/alias.rs b/crates/core/src/expr/alias.rs similarity index 97% rename from src/expr/alias.rs rename to crates/core/src/expr/alias.rs index b76e82e22..391c94dbc 100644 --- a/src/expr/alias.rs +++ b/crates/core/src/expr/alias.rs @@ -53,7 +53,7 @@ impl Display for PyAlias { "Alias \nExpr: `{:?}` \nAlias Name: `{}`", - &self.alias.expr, &self.alias.name + self.alias.expr, self.alias.name ) } } diff --git a/src/expr/analyze.rs b/crates/core/src/expr/analyze.rs similarity index 100% rename from src/expr/analyze.rs rename to crates/core/src/expr/analyze.rs diff --git a/src/expr/between.rs b/crates/core/src/expr/between.rs similarity index 95% rename from src/expr/between.rs rename to crates/core/src/expr/between.rs index 6943b6c3b..80f6c70da 100644 --- a/src/expr/between.rs +++ b/crates/core/src/expr/between.rs @@ -55,7 +55,7 @@ impl Display for PyBetween { Negated: {:?} Low: {:?} High: {:?}", - &self.between.expr, &self.between.negated, &self.between.low, &self.between.high + self.between.expr, self.between.negated, self.between.low, self.between.high ) } } diff --git a/src/expr/binary_expr.rs b/crates/core/src/expr/binary_expr.rs similarity index 100% rename from src/expr/binary_expr.rs rename to crates/core/src/expr/binary_expr.rs diff --git a/src/expr/bool_expr.rs b/crates/core/src/expr/bool_expr.rs similarity index 96% rename from src/expr/bool_expr.rs rename to crates/core/src/expr/bool_expr.rs index 9e374c7e2..d1cd7bcf7 100644 --- a/src/expr/bool_expr.rs +++ b/crates/core/src/expr/bool_expr.rs @@ -46,7 +46,7 @@ impl Display for PyNot { f, "Not Expr: {}", - &self.expr + self.expr ) } } @@ -82,7 +82,7 @@ impl Display for PyIsNotNull { f, "IsNotNull Expr: {}", - &self.expr + self.expr ) } } @@ -118,7 +118,7 @@ impl Display for PyIsNull { f, "IsNull Expr: {}", - &self.expr + self.expr ) } } @@ -154,7 +154,7 @@ impl Display for PyIsTrue { f, "IsTrue Expr: {}", - &self.expr + self.expr ) } } @@ -190,7 +190,7 @@ impl Display for PyIsFalse { f, "IsFalse Expr: {}", - &self.expr + self.expr ) } } @@ -226,7 +226,7 @@ impl Display for PyIsUnknown { f, "IsUnknown Expr: {}", - &self.expr + self.expr ) } } @@ -262,7 +262,7 @@ impl Display for PyIsNotTrue { f, "IsNotTrue Expr: {}", - &self.expr + self.expr ) } } @@ -298,7 +298,7 @@ impl Display for PyIsNotFalse { f, "IsNotFalse Expr: {}", - &self.expr + self.expr ) } } @@ -334,7 +334,7 @@ impl Display for PyIsNotUnknown { f, "IsNotUnknown Expr: {}", - &self.expr + self.expr ) } } @@ -370,7 +370,7 @@ impl Display for PyNegative { f, "Negative Expr: {}", - &self.expr + self.expr ) } } diff --git a/src/expr/case.rs b/crates/core/src/expr/case.rs similarity index 100% rename from src/expr/case.rs rename to crates/core/src/expr/case.rs diff --git a/src/expr/cast.rs b/crates/core/src/expr/cast.rs similarity index 94% rename from src/expr/cast.rs rename to crates/core/src/expr/cast.rs index 37d603538..484d0c059 100644 --- a/src/expr/cast.rs +++ b/crates/core/src/expr/cast.rs @@ -52,7 +52,7 @@ impl PyCast { } fn data_type(&self) -> PyResult { - Ok(self.cast.data_type.clone().into()) + Ok(self.cast.field.data_type().clone().into()) } } @@ -81,6 +81,6 @@ impl PyTryCast { } fn data_type(&self) -> PyResult { - Ok(self.try_cast.data_type.clone().into()) + Ok(self.try_cast.field.data_type().clone().into()) } } diff --git a/src/expr/column.rs b/crates/core/src/expr/column.rs similarity index 100% rename from src/expr/column.rs rename to crates/core/src/expr/column.rs diff --git a/src/expr/conditional_expr.rs b/crates/core/src/expr/conditional_expr.rs similarity index 100% rename from src/expr/conditional_expr.rs rename to crates/core/src/expr/conditional_expr.rs diff --git a/src/expr/copy_to.rs b/crates/core/src/expr/copy_to.rs similarity index 100% rename from src/expr/copy_to.rs rename to crates/core/src/expr/copy_to.rs diff --git a/src/expr/create_catalog.rs b/crates/core/src/expr/create_catalog.rs similarity index 100% rename from src/expr/create_catalog.rs rename to crates/core/src/expr/create_catalog.rs diff --git a/src/expr/create_catalog_schema.rs b/crates/core/src/expr/create_catalog_schema.rs similarity index 100% rename from src/expr/create_catalog_schema.rs rename to crates/core/src/expr/create_catalog_schema.rs diff --git a/src/expr/create_external_table.rs b/crates/core/src/expr/create_external_table.rs similarity index 97% rename from src/expr/create_external_table.rs rename to crates/core/src/expr/create_external_table.rs index 980eea131..e78b836bf 100644 --- a/src/expr/create_external_table.rs +++ b/crates/core/src/expr/create_external_table.rs @@ -88,7 +88,7 @@ impl PyCreateExternalTable { let create = CreateExternalTable { schema: Arc::new(schema.into()), name: name.into(), - location, + locations: vec![location], file_type, table_partition_cols, if_not_exists, @@ -118,8 +118,8 @@ impl PyCreateExternalTable { Ok(self.create.name.to_string()) } - pub fn location(&self) -> String { - self.create.location.clone() + pub fn locations(&self) -> Vec { + self.create.locations.clone() } pub fn file_type(&self) -> String { diff --git a/src/expr/create_function.rs b/crates/core/src/expr/create_function.rs similarity index 100% rename from src/expr/create_function.rs rename to crates/core/src/expr/create_function.rs diff --git a/src/expr/create_index.rs b/crates/core/src/expr/create_index.rs similarity index 100% rename from src/expr/create_index.rs rename to crates/core/src/expr/create_index.rs diff --git a/src/expr/create_memory_table.rs b/crates/core/src/expr/create_memory_table.rs similarity index 95% rename from src/expr/create_memory_table.rs rename to crates/core/src/expr/create_memory_table.rs index 3214dab0e..c27c11834 100644 --- a/src/expr/create_memory_table.rs +++ b/crates/core/src/expr/create_memory_table.rs @@ -57,10 +57,7 @@ impl Display for PyCreateMemoryTable { Input: {:?} if_not_exists: {:?} or_replace: {:?}", - &self.create.name, - &self.create.input, - &self.create.if_not_exists, - &self.create.or_replace, + self.create.name, self.create.input, self.create.if_not_exists, self.create.or_replace, ) } } diff --git a/src/expr/create_view.rs b/crates/core/src/expr/create_view.rs similarity index 96% rename from src/expr/create_view.rs rename to crates/core/src/expr/create_view.rs index 6941ef769..2f0315c40 100644 --- a/src/expr/create_view.rs +++ b/crates/core/src/expr/create_view.rs @@ -58,7 +58,7 @@ impl Display for PyCreateView { input: {:?} or_replace: {:?} definition: {:?}", - &self.create.name, &self.create.input, &self.create.or_replace, &self.create.definition, + self.create.name, self.create.input, self.create.or_replace, self.create.definition, ) } } diff --git a/src/expr/describe_table.rs b/crates/core/src/expr/describe_table.rs similarity index 100% rename from src/expr/describe_table.rs rename to crates/core/src/expr/describe_table.rs diff --git a/src/expr/distinct.rs b/crates/core/src/expr/distinct.rs similarity index 100% rename from src/expr/distinct.rs rename to crates/core/src/expr/distinct.rs diff --git a/src/expr/dml.rs b/crates/core/src/expr/dml.rs similarity index 80% rename from src/expr/dml.rs rename to crates/core/src/expr/dml.rs index 26f975820..5967d181e 100644 --- a/src/expr/dml.rs +++ b/crates/core/src/expr/dml.rs @@ -18,6 +18,7 @@ use datafusion::logical_expr::dml::InsertOp; use datafusion::logical_expr::{DmlStatement, WriteOp}; use pyo3::IntoPyObjectExt; +use pyo3::exceptions::PyNotImplementedError; use pyo3::prelude::*; use super::logical_node::LogicalNode; @@ -71,8 +72,8 @@ impl PyDmlStatement { }) } - pub fn op(&self) -> PyWriteOp { - self.dml.op.clone().into() + pub fn op(&self) -> PyResult { + self.dml.op.clone().try_into() } pub fn input(&self) -> PyLogicalPlan { @@ -112,16 +113,21 @@ pub enum PyWriteOp { Truncate, } -impl From for PyWriteOp { - fn from(write_op: WriteOp) -> Self { +impl TryFrom for PyWriteOp { + type Error = PyErr; + + fn try_from(write_op: WriteOp) -> Result { match write_op { - WriteOp::Insert(InsertOp::Append) => PyWriteOp::Append, - WriteOp::Insert(InsertOp::Overwrite) => PyWriteOp::Overwrite, - WriteOp::Insert(InsertOp::Replace) => PyWriteOp::Replace, - WriteOp::Update => PyWriteOp::Update, - WriteOp::Delete => PyWriteOp::Delete, - WriteOp::Ctas => PyWriteOp::Ctas, - WriteOp::Truncate => PyWriteOp::Truncate, + WriteOp::Insert(InsertOp::Append) => Ok(PyWriteOp::Append), + WriteOp::Insert(InsertOp::Overwrite) => Ok(PyWriteOp::Overwrite), + WriteOp::Insert(InsertOp::Replace) => Ok(PyWriteOp::Replace), + WriteOp::Update => Ok(PyWriteOp::Update), + WriteOp::Delete => Ok(PyWriteOp::Delete), + WriteOp::Ctas => Ok(PyWriteOp::Ctas), + WriteOp::Truncate => Ok(PyWriteOp::Truncate), + unsupported => Err(PyNotImplementedError::new_err(format!( + "DataFusion write operation {unsupported:?} is not supported" + ))), } } } diff --git a/src/expr/drop_catalog_schema.rs b/crates/core/src/expr/drop_catalog_schema.rs similarity index 97% rename from src/expr/drop_catalog_schema.rs rename to crates/core/src/expr/drop_catalog_schema.rs index fd5105332..f349098b7 100644 --- a/src/expr/drop_catalog_schema.rs +++ b/crates/core/src/expr/drop_catalog_schema.rs @@ -18,9 +18,8 @@ use std::fmt::{self, Display, Formatter}; use std::sync::Arc; -use datafusion::common::SchemaReference; +use datafusion::common::{SchemaReference, TableReference}; use datafusion::logical_expr::DropCatalogSchema; -use datafusion::sql::TableReference; use pyo3::IntoPyObjectExt; use pyo3::exceptions::PyValueError; use pyo3::prelude::*; diff --git a/src/expr/drop_function.rs b/crates/core/src/expr/drop_function.rs similarity index 100% rename from src/expr/drop_function.rs rename to crates/core/src/expr/drop_function.rs diff --git a/src/expr/drop_table.rs b/crates/core/src/expr/drop_table.rs similarity index 97% rename from src/expr/drop_table.rs rename to crates/core/src/expr/drop_table.rs index 46fe67465..156d3bcc2 100644 --- a/src/expr/drop_table.rs +++ b/crates/core/src/expr/drop_table.rs @@ -56,7 +56,7 @@ impl Display for PyDropTable { name: {:?} if_exists: {:?} schema: {:?}", - &self.drop.name, &self.drop.if_exists, &self.drop.schema, + self.drop.name, self.drop.if_exists, self.drop.schema, ) } } diff --git a/src/expr/drop_view.rs b/crates/core/src/expr/drop_view.rs similarity index 100% rename from src/expr/drop_view.rs rename to crates/core/src/expr/drop_view.rs diff --git a/src/expr/empty_relation.rs b/crates/core/src/expr/empty_relation.rs similarity index 97% rename from src/expr/empty_relation.rs rename to crates/core/src/expr/empty_relation.rs index f3c237731..5af4aafeb 100644 --- a/src/expr/empty_relation.rs +++ b/crates/core/src/expr/empty_relation.rs @@ -56,7 +56,7 @@ impl Display for PyEmptyRelation { "Empty Relation Produce One Row: {:?} Schema: {:?}", - &self.empty.produce_one_row, &self.empty.schema + self.empty.produce_one_row, self.empty.schema ) } } diff --git a/src/expr/exists.rs b/crates/core/src/expr/exists.rs similarity index 100% rename from src/expr/exists.rs rename to crates/core/src/expr/exists.rs diff --git a/src/expr/explain.rs b/crates/core/src/expr/explain.rs similarity index 93% rename from src/expr/explain.rs rename to crates/core/src/expr/explain.rs index 6259951de..d6ba1c25c 100644 --- a/src/expr/explain.rs +++ b/crates/core/src/expr/explain.rs @@ -61,11 +61,11 @@ impl Display for PyExplain { stringified_plans: {:?} schema: {:?} logical_optimization_succeeded: {:?}", - &self.explain.verbose, - &self.explain.plan, - &self.explain.stringified_plans, - &self.explain.schema, - &self.explain.logical_optimization_succeeded + self.explain.verbose, + self.explain.plan, + self.explain.stringified_plans, + self.explain.schema, + self.explain.logical_optimization_succeeded ) } } diff --git a/src/expr/extension.rs b/crates/core/src/expr/extension.rs similarity index 100% rename from src/expr/extension.rs rename to crates/core/src/expr/extension.rs diff --git a/src/expr/filter.rs b/crates/core/src/expr/filter.rs similarity index 97% rename from src/expr/filter.rs rename to crates/core/src/expr/filter.rs index 67426806d..1fe5f2c7f 100644 --- a/src/expr/filter.rs +++ b/crates/core/src/expr/filter.rs @@ -57,7 +57,7 @@ impl Display for PyFilter { "Filter Predicate: {:?} Input: {:?}", - &self.filter.predicate, &self.filter.input + self.filter.predicate, self.filter.input ) } } diff --git a/crates/core/src/expr/grouping_set.rs b/crates/core/src/expr/grouping_set.rs new file mode 100644 index 000000000..11d8f4fcd --- /dev/null +++ b/crates/core/src/expr/grouping_set.rs @@ -0,0 +1,78 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use datafusion::logical_expr::{Expr, GroupingSet}; +use pyo3::prelude::*; + +use crate::expr::PyExpr; + +#[pyclass( + from_py_object, + frozen, + name = "GroupingSet", + module = "datafusion.expr", + subclass +)] +#[derive(Clone)] +pub struct PyGroupingSet { + grouping_set: GroupingSet, +} + +#[pymethods] +impl PyGroupingSet { + #[staticmethod] + #[pyo3(signature = (*exprs))] + fn rollup(exprs: Vec) -> PyExpr { + Expr::GroupingSet(GroupingSet::Rollup( + exprs.into_iter().map(|e| e.expr).collect(), + )) + .into() + } + + #[staticmethod] + #[pyo3(signature = (*exprs))] + fn cube(exprs: Vec) -> PyExpr { + Expr::GroupingSet(GroupingSet::Cube( + exprs.into_iter().map(|e| e.expr).collect(), + )) + .into() + } + + #[staticmethod] + #[pyo3(signature = (*expr_lists))] + fn grouping_sets(expr_lists: Vec>) -> PyExpr { + Expr::GroupingSet(GroupingSet::GroupingSets( + expr_lists + .into_iter() + .map(|list| list.into_iter().map(|e| e.expr).collect()) + .collect(), + )) + .into() + } +} + +impl From for GroupingSet { + fn from(grouping_set: PyGroupingSet) -> Self { + grouping_set.grouping_set + } +} + +impl From for PyGroupingSet { + fn from(grouping_set: GroupingSet) -> PyGroupingSet { + PyGroupingSet { grouping_set } + } +} diff --git a/crates/core/src/expr/higher_order_function.rs b/crates/core/src/expr/higher_order_function.rs new file mode 100644 index 000000000..5ba64052a --- /dev/null +++ b/crates/core/src/expr/higher_order_function.rs @@ -0,0 +1,84 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::fmt::{self, Display, Formatter}; + +use datafusion::logical_expr::expr::HigherOrderFunction; +use pyo3::prelude::*; + +use super::PyExpr; + +#[pyclass( + from_py_object, + frozen, + name = "HigherOrderFunction", + module = "datafusion.expr", + subclass +)] +#[derive(Clone)] +pub struct PyHigherOrderFunction { + higher_order: HigherOrderFunction, +} + +impl From for PyHigherOrderFunction { + fn from(higher_order: HigherOrderFunction) -> PyHigherOrderFunction { + PyHigherOrderFunction { higher_order } + } +} + +impl From for HigherOrderFunction { + fn from(higher_order: PyHigherOrderFunction) -> Self { + higher_order.higher_order + } +} + +impl Display for PyHigherOrderFunction { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + write!( + f, + "HigherOrderFunction(name={}, args={:?})", + self.higher_order.name(), + self.higher_order.args, + ) + } +} + +#[pymethods] +impl PyHigherOrderFunction { + /// Name of the higher-order function being invoked. + fn name(&self) -> String { + self.higher_order.name().to_string() + } + + /// Arguments passed to the higher-order function. Some entries may be + /// `Lambda` expressions; others are ordinary value expressions. + fn args(&self) -> Vec { + self.higher_order + .args + .iter() + .map(|e| PyExpr::from(e.clone())) + .collect() + } + + fn __repr__(&self) -> PyResult { + Ok(format!("HigherOrderFunction({self})")) + } + + fn __name__(&self) -> PyResult { + Ok("HigherOrderFunction".to_string()) + } +} diff --git a/src/expr/in_list.rs b/crates/core/src/expr/in_list.rs similarity index 100% rename from src/expr/in_list.rs rename to crates/core/src/expr/in_list.rs diff --git a/src/expr/in_subquery.rs b/crates/core/src/expr/in_subquery.rs similarity index 100% rename from src/expr/in_subquery.rs rename to crates/core/src/expr/in_subquery.rs diff --git a/src/expr/indexed_field.rs b/crates/core/src/expr/indexed_field.rs similarity index 100% rename from src/expr/indexed_field.rs rename to crates/core/src/expr/indexed_field.rs diff --git a/src/expr/join.rs b/crates/core/src/expr/join.rs similarity index 95% rename from src/expr/join.rs rename to crates/core/src/expr/join.rs index b90f2f57d..634e13734 100644 --- a/src/expr/join.rs +++ b/crates/core/src/expr/join.rs @@ -132,14 +132,14 @@ impl Display for PyJoin { JoinConstraint: {:?} Schema: {:?} NullEquality: {:?}", - &self.join.left, - &self.join.right, - &self.join.on, - &self.join.filter, - &self.join.join_type, - &self.join.join_constraint, - &self.join.schema, - &self.join.null_equality, + self.join.left, + self.join.right, + self.join.on, + self.join.filter, + self.join.join_type, + self.join.join_constraint, + self.join.schema, + self.join.null_equality, ) } } diff --git a/crates/core/src/expr/lambda.rs b/crates/core/src/expr/lambda.rs new file mode 100644 index 000000000..7190521d6 --- /dev/null +++ b/crates/core/src/expr/lambda.rs @@ -0,0 +1,78 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::fmt::{self, Display, Formatter}; + +use datafusion::logical_expr::expr::Lambda; +use pyo3::prelude::*; + +use super::PyExpr; + +#[pyclass( + from_py_object, + frozen, + name = "Lambda", + module = "datafusion.expr", + subclass +)] +#[derive(Clone)] +pub struct PyLambda { + lambda: Lambda, +} + +impl From for PyLambda { + fn from(lambda: Lambda) -> PyLambda { + PyLambda { lambda } + } +} + +impl From for Lambda { + fn from(lambda: PyLambda) -> Self { + lambda.lambda + } +} + +impl Display for PyLambda { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + write!( + f, + "Lambda(params={:?}, body={:?})", + self.lambda.params, self.lambda.body, + ) + } +} + +#[pymethods] +impl PyLambda { + /// Parameter names of the lambda. + fn params(&self) -> Vec { + self.lambda.params.clone() + } + + /// Body expression of the lambda. + fn body(&self) -> PyExpr { + (*self.lambda.body).clone().into() + } + + fn __repr__(&self) -> PyResult { + Ok(format!("Lambda({self})")) + } + + fn __name__(&self) -> PyResult { + Ok("Lambda".to_string()) + } +} diff --git a/crates/core/src/expr/lambda_variable.rs b/crates/core/src/expr/lambda_variable.rs new file mode 100644 index 000000000..7baf5f21c --- /dev/null +++ b/crates/core/src/expr/lambda_variable.rs @@ -0,0 +1,67 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::fmt::{self, Display, Formatter}; + +use datafusion::logical_expr::expr::LambdaVariable; +use pyo3::prelude::*; + +#[pyclass( + from_py_object, + frozen, + name = "LambdaVariable", + module = "datafusion.expr", + subclass +)] +#[derive(Clone)] +pub struct PyLambdaVariable { + variable: LambdaVariable, +} + +impl From for PyLambdaVariable { + fn from(variable: LambdaVariable) -> PyLambdaVariable { + PyLambdaVariable { variable } + } +} + +impl From for LambdaVariable { + fn from(variable: PyLambdaVariable) -> Self { + variable.variable + } +} + +impl Display for PyLambdaVariable { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + write!(f, "LambdaVariable({})", self.variable.name) + } +} + +#[pymethods] +impl PyLambdaVariable { + /// Reference name of the lambda parameter. + fn name(&self) -> String { + self.variable.name.clone() + } + + fn __repr__(&self) -> PyResult { + Ok(format!("LambdaVariable({self})")) + } + + fn __name__(&self) -> PyResult { + Ok("LambdaVariable".to_string()) + } +} diff --git a/src/expr/like.rs b/crates/core/src/expr/like.rs similarity index 92% rename from src/expr/like.rs rename to crates/core/src/expr/like.rs index 417dc9182..900551a20 100644 --- a/src/expr/like.rs +++ b/crates/core/src/expr/like.rs @@ -55,10 +55,10 @@ impl Display for PyLike { Expr: {:?} Pattern: {:?} Escape_Char: {:?}", - &self.negated(), - &self.expr(), - &self.pattern(), - &self.escape_char() + self.negated(), + self.expr(), + self.pattern(), + self.escape_char() ) } } @@ -119,10 +119,10 @@ impl Display for PyILike { Expr: {:?} Pattern: {:?} Escape_Char: {:?}", - &self.negated(), - &self.expr(), - &self.pattern(), - &self.escape_char() + self.negated(), + self.expr(), + self.pattern(), + self.escape_char() ) } } @@ -183,10 +183,10 @@ impl Display for PySimilarTo { Expr: {:?} Pattern: {:?} Escape_Char: {:?}", - &self.negated(), - &self.expr(), - &self.pattern(), - &self.escape_char() + self.negated(), + self.expr(), + self.pattern(), + self.escape_char() ) } } diff --git a/src/expr/limit.rs b/crates/core/src/expr/limit.rs similarity index 73% rename from src/expr/limit.rs rename to crates/core/src/expr/limit.rs index c04b8bfa8..f53737b5e 100644 --- a/src/expr/limit.rs +++ b/crates/core/src/expr/limit.rs @@ -22,6 +22,7 @@ use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use crate::common::df_schema::PyDFSchema; +use crate::expr::PyExpr; use crate::expr::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; @@ -57,26 +58,30 @@ impl Display for PyLimit { Skip: {:?} Fetch: {:?} Input: {:?}", - &self.limit.skip, &self.limit.fetch, &self.limit.input + self.limit.skip, self.limit.fetch, self.limit.input ) } } #[pymethods] impl PyLimit { - // NOTE: Upstream now has expressions for skip and fetch - // TODO: Do we still want to expose these? - // REF: https://github.com/apache/datafusion/pull/12836 - - // /// Retrieves the skip value for this `Limit` - // fn skip(&self) -> usize { - // self.limit.skip - // } + // Retrieves the skip expression for this `Limit`, if any. + // + // `LIMIT`/`OFFSET` were changed upstream to support arbitrary + // expressions (not just constants), see + // https://github.com/apache/datafusion/pull/13028. Callers that expect + // a simple literal (the common case, e.g. `OFFSET 5`) should evaluate + // the returned `PyExpr` via `Expr.python_value()`. + fn skip(&self) -> PyResult> { + Ok(self.limit.skip.as_deref().cloned().map(PyExpr::from)) + } - // /// Retrieves the fetch value for this `Limit` - // fn fetch(&self) -> Option { - // self.limit.fetch - // } + // Retrieves the fetch expression for this `Limit`, if any. + // + // See the note on `skip` above regarding expression-based limits. + fn fetch(&self) -> PyResult> { + Ok(self.limit.fetch.as_deref().cloned().map(PyExpr::from)) + } /// Retrieves the input `LogicalPlan` to this `Limit` node fn input(&self) -> PyResult> { diff --git a/src/expr/literal.rs b/crates/core/src/expr/literal.rs similarity index 100% rename from src/expr/literal.rs rename to crates/core/src/expr/literal.rs diff --git a/src/expr/logical_node.rs b/crates/core/src/expr/logical_node.rs similarity index 100% rename from src/expr/logical_node.rs rename to crates/core/src/expr/logical_node.rs diff --git a/src/expr/placeholder.rs b/crates/core/src/expr/placeholder.rs similarity index 100% rename from src/expr/placeholder.rs rename to crates/core/src/expr/placeholder.rs diff --git a/src/expr/projection.rs b/crates/core/src/expr/projection.rs similarity index 97% rename from src/expr/projection.rs rename to crates/core/src/expr/projection.rs index 456e06412..7e22e1e7b 100644 --- a/src/expr/projection.rs +++ b/crates/core/src/expr/projection.rs @@ -65,7 +65,7 @@ impl Display for PyProjection { \nExpr(s): {:?} \nInput: {:?} \nProjected Schema: {:?}", - &self.projection.expr, &self.projection.input, &self.projection.schema, + self.projection.expr, self.projection.input, self.projection.schema, ) } } diff --git a/src/expr/recursive_query.rs b/crates/core/src/expr/recursive_query.rs similarity index 92% rename from src/expr/recursive_query.rs rename to crates/core/src/expr/recursive_query.rs index e03137b80..0b198a191 100644 --- a/src/expr/recursive_query.rs +++ b/crates/core/src/expr/recursive_query.rs @@ -22,6 +22,7 @@ use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; +use crate::errors::PyDataFusionResult; use crate::sql::logical::PyLogicalPlan; #[pyclass( @@ -67,15 +68,15 @@ impl PyRecursiveQuery { static_term: PyLogicalPlan, recursive_term: PyLogicalPlan, is_distinct: bool, - ) -> Self { - Self { - query: RecursiveQuery { + ) -> PyDataFusionResult { + Ok(Self { + query: RecursiveQuery::try_new( name, - static_term: static_term.plan(), - recursive_term: recursive_term.plan(), + static_term.plan(), + recursive_term.plan(), is_distinct, - }, - } + )?, + }) } fn name(&self) -> PyResult { diff --git a/src/expr/repartition.rs b/crates/core/src/expr/repartition.rs similarity index 98% rename from src/expr/repartition.rs rename to crates/core/src/expr/repartition.rs index be39b9978..cbc8a97bb 100644 --- a/src/expr/repartition.rs +++ b/crates/core/src/expr/repartition.rs @@ -82,7 +82,7 @@ impl Display for PyRepartition { "Repartition input: {:?} partitioning_scheme: {:?}", - &self.repartition.input, &self.repartition.partitioning_scheme, + self.repartition.input, self.repartition.partitioning_scheme, ) } } diff --git a/src/expr/scalar_subquery.rs b/crates/core/src/expr/scalar_subquery.rs similarity index 100% rename from src/expr/scalar_subquery.rs rename to crates/core/src/expr/scalar_subquery.rs diff --git a/src/expr/scalar_variable.rs b/crates/core/src/expr/scalar_variable.rs similarity index 100% rename from src/expr/scalar_variable.rs rename to crates/core/src/expr/scalar_variable.rs diff --git a/src/expr/set_comparison.rs b/crates/core/src/expr/set_comparison.rs similarity index 100% rename from src/expr/set_comparison.rs rename to crates/core/src/expr/set_comparison.rs diff --git a/src/expr/signature.rs b/crates/core/src/expr/signature.rs similarity index 100% rename from src/expr/signature.rs rename to crates/core/src/expr/signature.rs diff --git a/src/expr/sort.rs b/crates/core/src/expr/sort.rs similarity index 99% rename from src/expr/sort.rs rename to crates/core/src/expr/sort.rs index 7c1e654c5..1b1065011 100644 --- a/src/expr/sort.rs +++ b/crates/core/src/expr/sort.rs @@ -61,7 +61,7 @@ impl Display for PySort { \nExpr(s): {:?} \nInput: {:?} \nSchema: {:?}", - &self.sort.expr, + self.sort.expr, self.sort.input, self.sort.input.schema() ) diff --git a/src/expr/sort_expr.rs b/crates/core/src/expr/sort_expr.rs similarity index 97% rename from src/expr/sort_expr.rs rename to crates/core/src/expr/sort_expr.rs index 3c3c86bc1..93faffcec 100644 --- a/src/expr/sort_expr.rs +++ b/crates/core/src/expr/sort_expr.rs @@ -54,7 +54,7 @@ impl Display for PySortExpr { Expr: {:?} Asc: {:?} NullsFirst: {:?}", - &self.sort.expr, &self.sort.asc, &self.sort.nulls_first + self.sort.expr, self.sort.asc, self.sort.nulls_first ) } } diff --git a/src/expr/statement.rs b/crates/core/src/expr/statement.rs similarity index 100% rename from src/expr/statement.rs rename to crates/core/src/expr/statement.rs diff --git a/src/expr/subquery.rs b/crates/core/src/expr/subquery.rs similarity index 100% rename from src/expr/subquery.rs rename to crates/core/src/expr/subquery.rs diff --git a/src/expr/subquery_alias.rs b/crates/core/src/expr/subquery_alias.rs similarity index 100% rename from src/expr/subquery_alias.rs rename to crates/core/src/expr/subquery_alias.rs diff --git a/src/expr/table_scan.rs b/crates/core/src/expr/table_scan.rs similarity index 97% rename from src/expr/table_scan.rs rename to crates/core/src/expr/table_scan.rs index 8ba7e4a69..46ce551d2 100644 --- a/src/expr/table_scan.rs +++ b/crates/core/src/expr/table_scan.rs @@ -65,10 +65,10 @@ impl Display for PyTableScan { Projections: {:?} Projected Schema: {:?} Filters: {:?}", - &self.table_scan.table_name, - &self.py_projections(), - &self.py_schema(), - &self.py_filters(), + self.table_scan.table_name, + self.py_projections(), + self.py_schema(), + self.py_filters(), ) } } diff --git a/src/expr/union.rs b/crates/core/src/expr/union.rs similarity index 97% rename from src/expr/union.rs rename to crates/core/src/expr/union.rs index a3b9efe91..bd5770e0a 100644 --- a/src/expr/union.rs +++ b/crates/core/src/expr/union.rs @@ -56,7 +56,7 @@ impl Display for PyUnion { "Union Inputs: {:?} Schema: {:?}", - &self.union_.inputs, &self.union_.schema, + self.union_.inputs, self.union_.schema, ) } } diff --git a/src/expr/unnest.rs b/crates/core/src/expr/unnest.rs similarity index 97% rename from src/expr/unnest.rs rename to crates/core/src/expr/unnest.rs index 880d0a279..540667824 100644 --- a/src/expr/unnest.rs +++ b/crates/core/src/expr/unnest.rs @@ -56,7 +56,7 @@ impl Display for PyUnnest { "Unnest Inputs: {:?} Schema: {:?}", - &self.unnest_.input, &self.unnest_.schema, + self.unnest_.input, self.unnest_.schema, ) } } diff --git a/src/expr/unnest_expr.rs b/crates/core/src/expr/unnest_expr.rs similarity index 98% rename from src/expr/unnest_expr.rs rename to crates/core/src/expr/unnest_expr.rs index 97feef1d1..549257b86 100644 --- a/src/expr/unnest_expr.rs +++ b/crates/core/src/expr/unnest_expr.rs @@ -52,7 +52,7 @@ impl Display for PyUnnestExpr { f, "Unnest Expr: {:?}", - &self.unnest.expr, + self.unnest.expr, ) } } diff --git a/src/expr/values.rs b/crates/core/src/expr/values.rs similarity index 100% rename from src/expr/values.rs rename to crates/core/src/expr/values.rs diff --git a/src/expr/window.rs b/crates/core/src/expr/window.rs similarity index 99% rename from src/expr/window.rs rename to crates/core/src/expr/window.rs index 92d909bfc..e0050f671 100644 --- a/src/expr/window.rs +++ b/crates/core/src/expr/window.rs @@ -105,7 +105,7 @@ impl Display for PyWindowExpr { "Over\n Window Expr: {:?} Schema: {:?}", - &self.window.window_expr, &self.window.schema + self.window.window_expr, self.window.schema ) } } diff --git a/src/functions.rs b/crates/core/src/functions.rs similarity index 82% rename from src/functions.rs rename to crates/core/src/functions.rs index c32134054..e57c7702d 100644 --- a/src/functions.rs +++ b/crates/core/src/functions.rs @@ -18,26 +18,20 @@ use std::collections::HashMap; use datafusion::common::{Column, ScalarValue, TableReference}; -use datafusion::execution::FunctionRegistry; -use datafusion::functions_aggregate::all_default_aggregate_functions; -use datafusion::functions_window::all_default_window_functions; -use datafusion::logical_expr::expr::{ - Alias, FieldMetadata, NullTreatment as DFNullTreatment, WindowFunction, WindowFunctionParams, -}; -use datafusion::logical_expr::{Expr, ExprFunctionExt, WindowFrame, WindowFunctionDefinition, lit}; +use datafusion::logical_expr::expr::{Alias, FieldMetadata, NullTreatment as DFNullTreatment}; +use datafusion::logical_expr::{Expr, ExprFunctionExt, lit}; use datafusion::{functions, functions_aggregate, functions_window}; use pyo3::prelude::*; use pyo3::wrap_pyfunction; use crate::common::data_type::{NullTreatment, PyScalarValue}; -use crate::context::PySessionContext; -use crate::errors::{PyDataFusionError, PyDataFusionResult}; +use crate::errors::PyDataFusionResult; use crate::expr::PyExpr; use crate::expr::conditional_expr::PyCaseBuilder; use crate::expr::sort_expr::{PySortExpr, to_sort_expressions}; use crate::expr::window::PyWindowFrame; -fn add_builder_fns_to_aggregate( +pub(crate) fn add_builder_fns_to_aggregate( agg_fn: Expr, distinct: Option, filter: Option, @@ -93,6 +87,57 @@ fn array_cat(exprs: Vec) -> PyExpr { array_concat(exprs) } +#[pyfunction] +fn array_distance(array1: PyExpr, array2: PyExpr) -> PyExpr { + let args = vec![array1.into(), array2.into()]; + Expr::ScalarFunction(datafusion::logical_expr::expr::ScalarFunction::new_udf( + datafusion::functions_nested::distance::array_distance_udf(), + args, + )) + .into() +} + +#[pyfunction] +fn arrays_zip(exprs: Vec) -> PyExpr { + let exprs = exprs.into_iter().map(|x| x.into()).collect(); + datafusion::functions_nested::expr_fn::arrays_zip(exprs).into() +} + +#[pyfunction] +#[pyo3(signature = (string, delimiter, null_string=None))] +fn string_to_array(string: PyExpr, delimiter: PyExpr, null_string: Option) -> PyExpr { + let mut args = vec![string.into(), delimiter.into()]; + if let Some(null_string) = null_string { + args.push(null_string.into()); + } + Expr::ScalarFunction(datafusion::logical_expr::expr::ScalarFunction::new_udf( + datafusion::functions_nested::string::string_to_array_udf(), + args, + )) + .into() +} + +#[pyfunction] +#[pyo3(signature = (start, stop, step=None))] +fn gen_series(start: PyExpr, stop: PyExpr, step: Option) -> PyExpr { + let mut args = vec![start.into(), stop.into()]; + if let Some(step) = step { + args.push(step.into()); + } + Expr::ScalarFunction(datafusion::logical_expr::expr::ScalarFunction::new_udf( + datafusion::functions_nested::range::gen_series_udf(), + args, + )) + .into() +} + +#[pyfunction] +fn make_map(keys: Vec, values: Vec) -> PyExpr { + let keys = keys.into_iter().map(|x| x.into()).collect(); + let values = values.into_iter().map(|x| x.into()).collect(); + datafusion::functions_nested::map::map(keys, values).into() +} + #[pyfunction] #[pyo3(signature = (array, element, index=None))] fn array_position(array: PyExpr, element: PyExpr, index: Option) -> PyExpr { @@ -114,6 +159,44 @@ fn array_slice(array: PyExpr, begin: PyExpr, end: PyExpr, stride: Option .into() } +/// Create a lambda expression from a list of parameter names and a body +/// expression. The body should reference the parameters via [`lambda_var`]. +/// Exposed to Python as `lambda_` because `lambda` is a reserved keyword. +#[pyfunction] +#[pyo3(name = "lambda_")] +fn py_lambda(params: Vec, body: PyExpr) -> PyExpr { + datafusion::logical_expr::lambda(params, body.into()).into() +} + +/// Create an unresolved lambda variable reference by name. The owning +/// higher-order function resolves it against its lambda parameters during +/// planning. +#[pyfunction] +fn lambda_var(name: String) -> PyExpr { + datafusion::logical_expr::lambda_var(name).into() +} + +/// Higher-order function: apply `transform` (a lambda) to each element of +/// `array`, returning a new array of the results. +#[pyfunction] +fn array_transform(array: PyExpr, transform: PyExpr) -> PyExpr { + datafusion::functions_nested::expr_fn::array_transform(array.into(), transform.into()).into() +} + +/// Higher-order function: return true if any element of `array` satisfies +/// `predicate` (a lambda returning a boolean). +#[pyfunction] +fn array_any_match(array: PyExpr, predicate: PyExpr) -> PyExpr { + datafusion::functions_nested::expr_fn::array_any_match(array.into(), predicate.into()).into() +} + +/// Higher-order function: keep the elements of `array` for which `predicate` +/// (a lambda returning a boolean) is true, returning a new filtered array. +#[pyfunction] +fn array_filter(array: PyExpr, predicate: PyExpr) -> PyExpr { + datafusion::functions_nested::expr_fn::array_filter(array.into(), predicate.into()).into() +} + /// Computes a binary hash of the given data. type is the algorithm to use. /// Standard algorithms are md5, sha224, sha256, sha384, sha512, blake2s, blake2b, and blake3. // #[pyfunction(value, method)] @@ -255,126 +338,6 @@ fn when(when: PyExpr, then: PyExpr) -> PyResult { Ok(PyCaseBuilder::new(None).when(when, then)) } -/// Helper function to find the appropriate window function. -/// -/// Search procedure: -/// 1) Search built in window functions, which are being deprecated. -/// 1) If a session context is provided: -/// 1) search User Defined Aggregate Functions (UDAFs) -/// 1) search registered window functions -/// 1) search registered aggregate functions -/// 1) If no function has been found, search default aggregate functions. -/// -/// NOTE: we search the built-ins first because the `UDAF` versions currently do not have the same behavior. -fn find_window_fn( - name: &str, - ctx: Option, -) -> PyDataFusionResult { - if let Some(ctx) = ctx { - // search UDAFs - let udaf = ctx - .ctx - .udaf(name) - .map(WindowFunctionDefinition::AggregateUDF) - .ok(); - - if let Some(udaf) = udaf { - return Ok(udaf); - } - - let session_state = ctx.ctx.state(); - - // search registered window functions - let window_fn = session_state - .window_functions() - .get(name) - .map(|f| WindowFunctionDefinition::WindowUDF(f.clone())); - - if let Some(window_fn) = window_fn { - return Ok(window_fn); - } - - // search registered aggregate functions - let agg_fn = session_state - .aggregate_functions() - .get(name) - .map(|f| WindowFunctionDefinition::AggregateUDF(f.clone())); - - if let Some(agg_fn) = agg_fn { - return Ok(agg_fn); - } - } - - // search default aggregate functions - let agg_fn = all_default_aggregate_functions() - .iter() - .find(|v| v.name() == name || v.aliases().contains(&name.to_string())) - .map(|f| WindowFunctionDefinition::AggregateUDF(f.clone())); - - if let Some(agg_fn) = agg_fn { - return Ok(agg_fn); - } - - // search default window functions - let window_fn = all_default_window_functions() - .iter() - .find(|v| v.name() == name || v.aliases().contains(&name.to_string())) - .map(|f| WindowFunctionDefinition::WindowUDF(f.clone())); - - if let Some(window_fn) = window_fn { - return Ok(window_fn); - } - - Err(PyDataFusionError::Common(format!( - "window function `{name}` not found" - ))) -} - -/// Creates a new Window function expression -#[allow(clippy::too_many_arguments)] -#[pyfunction] -#[pyo3(signature = (name, args, partition_by=None, order_by=None, window_frame=None, filter=None, distinct=false, ctx=None))] -fn window( - name: &str, - args: Vec, - partition_by: Option>, - order_by: Option>, - window_frame: Option, - filter: Option, - distinct: bool, - ctx: Option, -) -> PyResult { - let fun = find_window_fn(name, ctx)?; - - let window_frame = window_frame - .map(|w| w.into()) - .unwrap_or(WindowFrame::new(order_by.as_ref().map(|v| !v.is_empty()))); - let filter = filter.map(|f| f.expr.into()); - - Ok(PyExpr { - expr: datafusion::logical_expr::Expr::WindowFunction(Box::new(WindowFunction { - fun, - params: WindowFunctionParams { - args: args.into_iter().map(|x| x.expr).collect::>(), - partition_by: partition_by - .unwrap_or_default() - .into_iter() - .map(|x| x.expr) - .collect::>(), - order_by: order_by - .unwrap_or_default() - .into_iter() - .map(|x| x.into()) - .collect::>(), - window_frame, - filter, - distinct, - null_treatment: None, - }, - })), - }) -} - // Generates a [pyo3] wrapper for associated aggregate functions. // All of the builder options are exposed to the python internal // function and we rely on the wrappers to only use those that @@ -494,6 +457,13 @@ expr_fn!(length, string); expr_fn!(char_length, string); expr_fn!(chr, arg, "Returns the character with the given code."); expr_fn_vec!(coalesce); +expr_fn_vec!(greatest); +expr_fn_vec!(least); +expr_fn!( + contains, + string search_str, + "Return true if search_str is found within string (case-sensitive)." +); expr_fn!(cos, num); expr_fn!(cosh, num); expr_fn!(cot, num); @@ -543,6 +513,11 @@ expr_fn!( x y, "Returns x if x is not NULL otherwise returns y." ); +expr_fn!( + nvl2, + x y z, + "Returns y if x is not NULL; otherwise returns z." +); expr_fn!(nullif, arg_1 arg_2); expr_fn!( octet_length, @@ -616,6 +591,7 @@ expr_fn!(date_part, part date); expr_fn!(date_trunc, part date); expr_fn!(date_bin, stride source origin); expr_fn!(make_date, year month day); +expr_fn!(make_time, hour minute second); expr_fn!(to_char, datetime format); expr_fn!(translate, string from to, "Replaces each character in string that matches a character in the from set with the corresponding character in the to set. If from is longer than to, occurrences of the extra characters in from are deleted."); @@ -631,8 +607,34 @@ expr_fn_vec!(named_struct); expr_fn!(from_unixtime, unixtime); expr_fn!(arrow_typeof, arg_1); expr_fn!(arrow_cast, arg_1 datatype); +expr_fn!(arrow_try_cast, arg_1 datatype); +expr_fn!(arrow_field, arg_1); +expr_fn!(cast_to_type, arg_1 reference); +expr_fn!(try_cast_to_type, arg_1 reference); +expr_fn_vec!(arrow_metadata); +expr_fn_vec!(with_metadata); +expr_fn!(union_tag, arg1); expr_fn!(random); +#[pyfunction] +fn get_field(expr: PyExpr, names: Vec) -> PyExpr { + let mut args = vec![expr.into()]; + args.extend(names.into_iter().map(Into::into)); + functions::core::get_field().call(args).into() +} + +#[pyfunction] +fn union_extract(union_expr: PyExpr, field_name: PyExpr) -> PyExpr { + functions::core::union_extract() + .call(vec![union_expr.into(), field_name.into()]) + .into() +} + +#[pyfunction] +fn version() -> PyExpr { + functions::core::version().call(vec![]).into() +} + // Array Functions array_fn!(array_append, array element); array_fn!(array_to_string, array delimiter); @@ -657,14 +659,28 @@ array_fn!(array_replace, array from to); array_fn!(array_replace_n, array from to max); array_fn!(array_replace_all, array from to); array_fn!(array_sort, array desc null_first); +array_fn!(array_compact, array); +array_fn!(array_normalize, array); +array_fn!(cosine_distance, array1 array2); +array_fn!(inner_product, array1 array2); array_fn!(array_intersect, first_array second_array); array_fn!(array_union, array1 array2); array_fn!(array_except, first_array second_array); array_fn!(array_resize, array size value); +array_fn!(array_any_value, array); +array_fn!(array_max, array); +array_fn!(array_min, array); +array_fn!(array_reverse, array); array_fn!(cardinality, array); array_fn!(flatten, array); array_fn!(range, start stop step); +// Map Functions +array_fn!(map_keys, map); +array_fn!(map_values, map); +array_fn!(map_extract, map key); +array_fn!(map_entries, map); + aggregate_function!(array_agg); aggregate_function!(max); aggregate_function!(min); @@ -696,9 +712,10 @@ aggregate_function!(var_pop); aggregate_function!(approx_distinct); aggregate_function!(approx_median); -// Code is commented out since grouping is not yet implemented -// https://github.com/apache/datafusion-python/issues/861 -// aggregate_function!(grouping); +// The grouping function's physical plan is not implemented, but the +// ResolveGroupingFunction analyzer rule rewrites it before the physical +// planner sees it, so it works correctly at runtime. +aggregate_function!(grouping); #[pyfunction] #[pyo3(signature = (sort_expression, percentile, num_centroids=None, filter=None))] @@ -736,6 +753,19 @@ pub fn approx_percentile_cont_with_weight( add_builder_fns_to_aggregate(agg_fn, None, filter, None, None) } +#[pyfunction] +#[pyo3(signature = (sort_expression, percentile, filter=None))] +pub fn percentile_cont( + sort_expression: PySortExpr, + percentile: f64, + filter: Option, +) -> PyDataFusionResult { + let agg_fn = + functions_aggregate::expr_fn::percentile_cont(sort_expression.sort, lit(percentile)); + + add_builder_fns_to_aggregate(agg_fn, None, filter, None, None) +} + // We handle last_value explicitly because the signature expects an order_by // https://github.com/apache/datafusion/issues/12376 #[pyfunction] @@ -936,10 +966,17 @@ pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_wrapped(wrap_pyfunction!(approx_median))?; m.add_wrapped(wrap_pyfunction!(approx_percentile_cont))?; m.add_wrapped(wrap_pyfunction!(approx_percentile_cont_with_weight))?; + m.add_wrapped(wrap_pyfunction!(percentile_cont))?; m.add_wrapped(wrap_pyfunction!(range))?; m.add_wrapped(wrap_pyfunction!(array_agg))?; m.add_wrapped(wrap_pyfunction!(arrow_typeof))?; m.add_wrapped(wrap_pyfunction!(arrow_cast))?; + m.add_wrapped(wrap_pyfunction!(arrow_try_cast))?; + m.add_wrapped(wrap_pyfunction!(arrow_field))?; + m.add_wrapped(wrap_pyfunction!(cast_to_type))?; + m.add_wrapped(wrap_pyfunction!(try_cast_to_type))?; + m.add_wrapped(wrap_pyfunction!(arrow_metadata))?; + m.add_wrapped(wrap_pyfunction!(with_metadata))?; m.add_wrapped(wrap_pyfunction!(ascii))?; m.add_wrapped(wrap_pyfunction!(asin))?; m.add_wrapped(wrap_pyfunction!(asinh))?; @@ -960,6 +997,7 @@ pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_wrapped(wrap_pyfunction!(col))?; m.add_wrapped(wrap_pyfunction!(concat_ws))?; m.add_wrapped(wrap_pyfunction!(concat))?; + m.add_wrapped(wrap_pyfunction!(contains))?; m.add_wrapped(wrap_pyfunction!(corr))?; m.add_wrapped(wrap_pyfunction!(cos))?; m.add_wrapped(wrap_pyfunction!(cosh))?; @@ -974,6 +1012,7 @@ pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_wrapped(wrap_pyfunction!(date_part))?; m.add_wrapped(wrap_pyfunction!(date_trunc))?; m.add_wrapped(wrap_pyfunction!(make_date))?; + m.add_wrapped(wrap_pyfunction!(make_time))?; m.add_wrapped(wrap_pyfunction!(digest))?; m.add_wrapped(wrap_pyfunction!(ends_with))?; m.add_wrapped(wrap_pyfunction!(exp))?; @@ -981,13 +1020,15 @@ pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_wrapped(wrap_pyfunction!(floor))?; m.add_wrapped(wrap_pyfunction!(from_unixtime))?; m.add_wrapped(wrap_pyfunction!(gcd))?; - // m.add_wrapped(wrap_pyfunction!(grouping))?; + m.add_wrapped(wrap_pyfunction!(greatest))?; + m.add_wrapped(wrap_pyfunction!(grouping))?; m.add_wrapped(wrap_pyfunction!(in_list))?; m.add_wrapped(wrap_pyfunction!(initcap))?; m.add_wrapped(wrap_pyfunction!(isnan))?; m.add_wrapped(wrap_pyfunction!(iszero))?; m.add_wrapped(wrap_pyfunction!(levenshtein))?; m.add_wrapped(wrap_pyfunction!(lcm))?; + m.add_wrapped(wrap_pyfunction!(least))?; m.add_wrapped(wrap_pyfunction!(left))?; m.add_wrapped(wrap_pyfunction!(length))?; m.add_wrapped(wrap_pyfunction!(ln))?; @@ -1005,6 +1046,7 @@ pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_wrapped(wrap_pyfunction!(named_struct))?; m.add_wrapped(wrap_pyfunction!(nanvl))?; m.add_wrapped(wrap_pyfunction!(nvl))?; + m.add_wrapped(wrap_pyfunction!(nvl2))?; m.add_wrapped(wrap_pyfunction!(now))?; m.add_wrapped(wrap_pyfunction!(nullif))?; m.add_wrapped(wrap_pyfunction!(octet_length))?; @@ -1063,10 +1105,13 @@ pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_wrapped(wrap_pyfunction!(trim))?; m.add_wrapped(wrap_pyfunction!(trunc))?; m.add_wrapped(wrap_pyfunction!(upper))?; + m.add_wrapped(wrap_pyfunction!(get_field))?; + m.add_wrapped(wrap_pyfunction!(union_extract))?; + m.add_wrapped(wrap_pyfunction!(union_tag))?; + m.add_wrapped(wrap_pyfunction!(version))?; m.add_wrapped(wrap_pyfunction!(self::uuid))?; // Use self to avoid name collision m.add_wrapped(wrap_pyfunction!(var_pop))?; m.add_wrapped(wrap_pyfunction!(var_sample))?; - m.add_wrapped(wrap_pyfunction!(window))?; m.add_wrapped(wrap_pyfunction!(regr_avgx))?; m.add_wrapped(wrap_pyfunction!(regr_avgy))?; m.add_wrapped(wrap_pyfunction!(regr_count))?; @@ -1089,12 +1134,23 @@ pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_wrapped(wrap_pyfunction!(encode))?; m.add_wrapped(wrap_pyfunction!(decode))?; + // Lambda / higher-order functions + m.add_wrapped(wrap_pyfunction!(py_lambda))?; + m.add_wrapped(wrap_pyfunction!(lambda_var))?; + m.add_wrapped(wrap_pyfunction!(array_transform))?; + m.add_wrapped(wrap_pyfunction!(array_any_match))?; + m.add_wrapped(wrap_pyfunction!(array_filter))?; + // Array Functions m.add_wrapped(wrap_pyfunction!(array_append))?; m.add_wrapped(wrap_pyfunction!(array_concat))?; m.add_wrapped(wrap_pyfunction!(array_cat))?; m.add_wrapped(wrap_pyfunction!(array_dims))?; m.add_wrapped(wrap_pyfunction!(array_distinct))?; + m.add_wrapped(wrap_pyfunction!(array_compact))?; + m.add_wrapped(wrap_pyfunction!(array_normalize))?; + m.add_wrapped(wrap_pyfunction!(cosine_distance))?; + m.add_wrapped(wrap_pyfunction!(inner_product))?; m.add_wrapped(wrap_pyfunction!(array_element))?; m.add_wrapped(wrap_pyfunction!(array_empty))?; m.add_wrapped(wrap_pyfunction!(array_length))?; @@ -1121,9 +1177,24 @@ pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_wrapped(wrap_pyfunction!(array_replace_all))?; m.add_wrapped(wrap_pyfunction!(array_sort))?; m.add_wrapped(wrap_pyfunction!(array_slice))?; + m.add_wrapped(wrap_pyfunction!(array_any_value))?; + m.add_wrapped(wrap_pyfunction!(array_distance))?; + m.add_wrapped(wrap_pyfunction!(array_max))?; + m.add_wrapped(wrap_pyfunction!(array_min))?; + m.add_wrapped(wrap_pyfunction!(array_reverse))?; + m.add_wrapped(wrap_pyfunction!(arrays_zip))?; + m.add_wrapped(wrap_pyfunction!(string_to_array))?; + m.add_wrapped(wrap_pyfunction!(gen_series))?; m.add_wrapped(wrap_pyfunction!(flatten))?; m.add_wrapped(wrap_pyfunction!(cardinality))?; + // Map Functions + m.add_wrapped(wrap_pyfunction!(make_map))?; + m.add_wrapped(wrap_pyfunction!(map_keys))?; + m.add_wrapped(wrap_pyfunction!(map_values))?; + m.add_wrapped(wrap_pyfunction!(map_extract))?; + m.add_wrapped(wrap_pyfunction!(map_entries))?; + // Window Functions m.add_wrapped(wrap_pyfunction!(lead))?; m.add_wrapped(wrap_pyfunction!(lag))?; diff --git a/src/lib.rs b/crates/core/src/lib.rs similarity index 87% rename from src/lib.rs rename to crates/core/src/lib.rs index 468243a3d..7f0f9cb39 100644 --- a/src/lib.rs +++ b/crates/core/src/lib.rs @@ -26,28 +26,25 @@ pub use datafusion_substrait; use mimalloc::MiMalloc; use pyo3::prelude::*; -#[allow(clippy::borrow_deref_ref)] +pub mod analyzer; pub mod catalog; +pub mod codec; pub mod common; -#[allow(clippy::borrow_deref_ref)] -mod config; -#[allow(clippy::borrow_deref_ref)] pub mod context; -#[allow(clippy::borrow_deref_ref)] pub mod dataframe; mod dataset; mod dataset_exec; pub mod errors; -#[allow(clippy::borrow_deref_ref)] pub mod expr; -#[allow(clippy::borrow_deref_ref)] mod functions; +pub mod metrics; mod options; pub mod physical_plan; mod pyarrow_filter_expression; pub mod pyarrow_util; mod record_batch; +mod spark_functions; pub mod sql; pub mod store; pub mod table; @@ -56,21 +53,20 @@ pub mod unparser; mod array; #[cfg(feature = "substrait")] pub mod substrait; -#[allow(clippy::borrow_deref_ref)] mod udaf; -#[allow(clippy::borrow_deref_ref)] mod udf; pub mod udtf; mod udwf; -pub mod utils; + +// Re-export helpers previously consumed by downstream Rust crates. +// Modules stay private to keep the public Rust API surface small. +pub use udaf::to_rust_accumulator; +pub use udwf::{MultiColumnWindowUDF, PythonFunctionWindowUDF, to_rust_partition_evaluator}; #[cfg(feature = "mimalloc")] #[global_allocator] static GLOBAL: MiMalloc = MiMalloc; -// Used to define Tokio Runtime as a Python module attribute -pub(crate) struct TokioRuntime(tokio::runtime::Runtime); - /// Low-level DataFusion internal package. /// /// The higher-level public API is defined in pure python files under the @@ -94,8 +90,9 @@ fn _internal(py: Python, m: Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; m.add_class::()?; - m.add_class::()?; m.add_class::()?; + m.add_class::()?; + m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; @@ -121,6 +118,10 @@ fn _internal(py: Python, m: Bound<'_, PyModule>) -> PyResult<()> { // Register the functions as a submodule let funcs = PyModule::new(py, "functions")?; functions::init_module(&funcs)?; + // Spark-compatible functions live under `functions.spark`. + let spark_funcs = PyModule::new(py, "spark")?; + spark_functions::init_module(&spark_funcs)?; + funcs.add_submodule(&spark_funcs)?; m.add_submodule(&funcs)?; let store = PyModule::new(py, "object_store")?; diff --git a/crates/core/src/metrics.rs b/crates/core/src/metrics.rs new file mode 100644 index 000000000..ee0937e25 --- /dev/null +++ b/crates/core/src/metrics.rs @@ -0,0 +1,169 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::collections::HashMap; +use std::sync::Arc; + +use chrono::{Datelike, Timelike}; +use datafusion::physical_plan::metrics::{Metric, MetricValue, MetricsSet, Timestamp}; +use pyo3::prelude::*; + +#[pyclass(from_py_object, frozen, name = "MetricsSet", module = "datafusion")] +#[derive(Debug, Clone)] +pub struct PyMetricsSet { + metrics: MetricsSet, +} + +impl PyMetricsSet { + pub fn new(metrics: MetricsSet) -> Self { + Self { metrics } + } +} + +#[pymethods] +impl PyMetricsSet { + fn metrics(&self) -> Vec { + self.metrics + .iter() + .map(|m| PyMetric::new(Arc::clone(m))) + .collect() + } + + fn output_rows(&self) -> Option { + self.metrics.output_rows() + } + + fn elapsed_compute(&self) -> Option { + self.metrics.elapsed_compute() + } + + fn spill_count(&self) -> Option { + self.metrics.spill_count() + } + + fn spilled_bytes(&self) -> Option { + self.metrics.spilled_bytes() + } + + fn spilled_rows(&self) -> Option { + self.metrics.spilled_rows() + } + + fn sum_by_name(&self, name: &str) -> Option { + self.metrics.sum_by_name(name).map(|v| v.as_usize()) + } + + fn __repr__(&self) -> String { + format!("{}", self.metrics) + } +} + +#[pyclass(from_py_object, frozen, name = "Metric", module = "datafusion")] +#[derive(Debug, Clone)] +pub struct PyMetric { + metric: Arc, +} + +impl PyMetric { + pub fn new(metric: Arc) -> Self { + Self { metric } + } + + fn timestamp_to_pyobject<'py>( + py: Python<'py>, + ts: &Timestamp, + ) -> PyResult>> { + match ts.value() { + Some(dt) => { + let datetime_mod = py.import("datetime")?; + let datetime_cls = datetime_mod.getattr("datetime")?; + let tz_utc = datetime_mod.getattr("timezone")?.getattr("utc")?; + let result = datetime_cls.call1(( + dt.year(), + dt.month(), + dt.day(), + dt.hour(), + dt.minute(), + dt.second(), + dt.timestamp_subsec_micros(), + tz_utc, + ))?; + Ok(Some(result)) + } + None => Ok(None), + } + } +} + +#[pymethods] +impl PyMetric { + #[getter] + fn name(&self) -> String { + self.metric.value().name().to_string() + } + + #[getter] + fn value<'py>(&self, py: Python<'py>) -> PyResult>> { + match self.metric.value() { + MetricValue::OutputRows(c) => Ok(Some(c.value().into_pyobject(py)?.into_any())), + MetricValue::OutputBytes(c) => Ok(Some(c.value().into_pyobject(py)?.into_any())), + MetricValue::ElapsedCompute(t) => Ok(Some(t.value().into_pyobject(py)?.into_any())), + MetricValue::SpillCount(c) => Ok(Some(c.value().into_pyobject(py)?.into_any())), + MetricValue::SpilledBytes(c) => Ok(Some(c.value().into_pyobject(py)?.into_any())), + MetricValue::SpilledRows(c) => Ok(Some(c.value().into_pyobject(py)?.into_any())), + MetricValue::CurrentMemoryUsage(g) => Ok(Some(g.value().into_pyobject(py)?.into_any())), + MetricValue::Count { count, .. } => { + Ok(Some(count.value().into_pyobject(py)?.into_any())) + } + MetricValue::Gauge { gauge, .. } => { + Ok(Some(gauge.value().into_pyobject(py)?.into_any())) + } + MetricValue::Time { time, .. } => Ok(Some(time.value().into_pyobject(py)?.into_any())), + MetricValue::StartTimestamp(ts) | MetricValue::EndTimestamp(ts) => { + Self::timestamp_to_pyobject(py, ts) + } + _ => Ok(None), + } + } + + #[getter] + fn value_as_datetime<'py>(&self, py: Python<'py>) -> PyResult>> { + match self.metric.value() { + MetricValue::StartTimestamp(ts) | MetricValue::EndTimestamp(ts) => { + Self::timestamp_to_pyobject(py, ts) + } + _ => Ok(None), + } + } + + #[getter] + fn partition(&self) -> Option { + self.metric.partition() + } + + fn labels(&self) -> HashMap { + self.metric + .labels() + .iter() + .map(|l| (l.name().to_string(), l.value().to_string())) + .collect() + } + + fn __repr__(&self) -> String { + format!("{}", self.metric.value()) + } +} diff --git a/src/options.rs b/crates/core/src/options.rs similarity index 100% rename from src/options.rs rename to crates/core/src/options.rs diff --git a/src/physical_plan.rs b/crates/core/src/physical_plan.rs similarity index 72% rename from src/physical_plan.rs rename to crates/core/src/physical_plan.rs index 8674a8b55..594655a60 100644 --- a/src/physical_plan.rs +++ b/crates/core/src/physical_plan.rs @@ -18,14 +18,16 @@ use std::sync::Arc; use datafusion::physical_plan::{ExecutionPlan, ExecutionPlanProperties, displayable}; -use datafusion_proto::physical_plan::{AsExecutionPlan, DefaultPhysicalExtensionCodec}; +use datafusion_proto::physical_plan::AsExecutionPlan; use prost::Message; use pyo3::exceptions::PyRuntimeError; use pyo3::prelude::*; use pyo3::types::PyBytes; +use crate::codec::PythonPhysicalCodec; use crate::context::PySessionContext; use crate::errors::PyDataFusionResult; +use crate::metrics::PyMetricsSet; #[pyclass( from_py_object, @@ -67,11 +69,26 @@ impl PyExecutionPlan { format!("{}", d.indent(false)) } - pub fn to_proto<'py>(&'py self, py: Python<'py>) -> PyDataFusionResult> { - let codec = DefaultPhysicalExtensionCodec {}; + #[pyo3(signature = (ctx=None))] + pub fn to_bytes<'py>( + &'py self, + py: Python<'py>, + ctx: Option, + ) -> PyDataFusionResult> { + // Route through the session's physical codec when supplied so + // user FFI codecs registered via + // `with_physical_extension_codec` see the encode path. + let default_codec; + let codec: &dyn datafusion_proto::physical_plan::PhysicalExtensionCodec = match ctx { + Some(ref ctx) => ctx.physical_codec().as_ref(), + None => { + default_codec = PythonPhysicalCodec::default(); + &default_codec + } + }; let proto = datafusion_proto::protobuf::PhysicalPlanNode::try_from_physical_plan( self.plan.clone(), - &codec, + codec, )?; let bytes = proto.encode_to_vec(); @@ -79,7 +96,7 @@ impl PyExecutionPlan { } #[staticmethod] - pub fn from_proto( + pub fn from_bytes( ctx: PySessionContext, proto_msg: Bound<'_, PyBytes>, ) -> PyDataFusionResult { @@ -87,15 +104,20 @@ impl PyExecutionPlan { let proto_plan = datafusion_proto::protobuf::PhysicalPlanNode::decode(bytes).map_err(|e| { PyRuntimeError::new_err(format!( - "Unable to decode logical node from serialized bytes: {e}" + "Unable to decode physical node from serialized bytes: {e}" )) })?; - let codec = DefaultPhysicalExtensionCodec {}; - let plan = proto_plan.try_into_physical_plan(ctx.ctx.task_ctx().as_ref(), &codec)?; + let codec = ctx.physical_codec(); + let plan = + proto_plan.try_into_physical_plan(ctx.ctx.task_ctx().as_ref(), codec.as_ref())?; Ok(Self::new(plan)) } + pub fn metrics(&self) -> Option { + self.plan.metrics().map(PyMetricsSet::new) + } + fn __repr__(&self) -> String { self.display_indent() } diff --git a/src/pyarrow_filter_expression.rs b/crates/core/src/pyarrow_filter_expression.rs similarity index 100% rename from src/pyarrow_filter_expression.rs rename to crates/core/src/pyarrow_filter_expression.rs diff --git a/src/pyarrow_util.rs b/crates/core/src/pyarrow_util.rs similarity index 100% rename from src/pyarrow_util.rs rename to crates/core/src/pyarrow_util.rs diff --git a/src/record_batch.rs b/crates/core/src/record_batch.rs similarity index 98% rename from src/record_batch.rs rename to crates/core/src/record_batch.rs index e8abc641b..0492c6c76 100644 --- a/src/record_batch.rs +++ b/crates/core/src/record_batch.rs @@ -20,6 +20,7 @@ use std::sync::Arc; use datafusion::arrow::pyarrow::ToPyArrow; use datafusion::arrow::record_batch::RecordBatch; use datafusion::physical_plan::SendableRecordBatchStream; +use datafusion_python_util::wait_for_future; use futures::StreamExt; use pyo3::exceptions::{PyStopAsyncIteration, PyStopIteration}; use pyo3::prelude::*; @@ -27,7 +28,6 @@ use pyo3::{PyAny, PyResult, Python, pyclass, pymethods}; use tokio::sync::Mutex; use crate::errors::PyDataFusionError; -use crate::utils::wait_for_future; #[pyclass(name = "RecordBatch", module = "datafusion", subclass, frozen)] pub struct PyRecordBatch { diff --git a/crates/core/src/spark_functions.rs b/crates/core/src/spark_functions.rs new file mode 100644 index 000000000..e7cb94f8c --- /dev/null +++ b/crates/core/src/spark_functions.rs @@ -0,0 +1,363 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! PyO3 wrappers for the [`datafusion-spark`] crate. +//! +//! Exposes Spark-compatible scalar and aggregate function builders for use +//! from Python under `datafusion.functions.spark`. + +use datafusion::logical_expr::Expr; +use datafusion::logical_expr::expr::ScalarFunction; +use datafusion_spark::{expr_fn, function as udf}; +use pyo3::prelude::*; +use pyo3::wrap_pyfunction; + +use crate::common::data_type::NullTreatment; +use crate::errors::PyDataFusionResult; +use crate::expr::PyExpr; +use crate::expr::sort_expr::PySortExpr; +use crate::functions::add_builder_fns_to_aggregate; + +/// Generates a [pyo3] wrapper for [datafusion_spark::expr_fn]. +/// +/// These functions have explicit named arguments and mirror the upstream +/// `expr_fn::$FUNC` signature. +macro_rules! spark_expr_fn { + ($FUNC:ident) => { + spark_expr_fn!($FUNC,); + }; + ($FUNC:ident, $($arg:ident)*) => { + #[pyfunction] + fn $FUNC($($arg: PyExpr),*) -> PyExpr { + expr_fn::$FUNC($($arg.into()),*).into() + } + }; +} + +/// Generates a variadic [pyo3] wrapper that calls the [`ScalarUDF`] factory +/// directly. Required for functions whose upstream `expr_fn` wrapper accepts +/// a single `Expr` instead of `Vec` (an upstream `export_functions!` +/// macro-arm quirk), so we bypass it to get true Python `*args` semantics. +macro_rules! spark_udf_vec { + ($PY_NAME:ident, $UDF_PATH:path) => { + #[pyfunction] + #[pyo3(signature = (*args))] + fn $PY_NAME(args: Vec) -> PyExpr { + let udf = $UDF_PATH(); + let args: Vec = args.into_iter().map(Into::into).collect(); + Expr::ScalarFunction(ScalarFunction::new_udf(udf, args)).into() + } + }; +} + +/// Generates a [pyo3] wrapper for Spark aggregate functions. Mirrors +/// [`crate::functions::aggregate_function`] but points at +/// [`datafusion_spark::expr_fn`]. +macro_rules! spark_aggregate { + ($NAME:ident) => { + spark_aggregate!($NAME, expr); + }; + ($NAME:ident, $($arg:ident)*) => { + #[pyfunction] + #[pyo3(signature = ($($arg),*, distinct=None, filter=None, order_by=None, null_treatment=None))] + fn $NAME( + $($arg: PyExpr),*, + distinct: Option, + filter: Option, + order_by: Option>, + null_treatment: Option, + ) -> PyDataFusionResult { + let agg_fn = expr_fn::$NAME($($arg.into()),*); + add_builder_fns_to_aggregate(agg_fn, distinct, filter, order_by, null_treatment) + } + }; +} + +// --------------------------------------------------------------------------- +// Aggregate functions +// --------------------------------------------------------------------------- + +spark_aggregate!(avg, arg1); +spark_aggregate!(try_sum, arg1); +spark_aggregate!(collect_list, arg1); +spark_aggregate!(collect_set, arg1); + +// --------------------------------------------------------------------------- +// Array functions +// --------------------------------------------------------------------------- + +// Upstream factory is `spark_array_contains`; expose under the Spark SQL +// name `array_contains` on the Python side. +#[pyfunction] +fn array_contains(arr: PyExpr, element: PyExpr) -> PyExpr { + expr_fn::spark_array_contains(arr.into(), element.into()).into() +} +spark_udf_vec!(array, udf::array::array); +spark_expr_fn!(shuffle, arg1); +spark_expr_fn!(array_repeat, element count); +spark_expr_fn!(slice, arr start length); + +// --------------------------------------------------------------------------- +// Bitmap functions +// --------------------------------------------------------------------------- + +spark_expr_fn!(bitmap_count, arg1); +spark_expr_fn!(bitmap_bit_position, arg1); +spark_expr_fn!(bitmap_bucket_number, arg1); + +// --------------------------------------------------------------------------- +// Bitwise functions +// --------------------------------------------------------------------------- + +spark_expr_fn!(bit_get, col pos); +spark_expr_fn!(bit_count, col); +spark_expr_fn!(bitwise_not, col); +spark_expr_fn!(shiftleft, value shift); +spark_expr_fn!(shiftright, value shift); +spark_expr_fn!(shiftrightunsigned, value shift); + +// --------------------------------------------------------------------------- +// Collection / Conditional / Conversion +// --------------------------------------------------------------------------- + +spark_expr_fn!(size, arg1); + +// Python keyword `if` → exposed as `if_`. Upstream Rust ident is `r#if`. +#[pyfunction] +fn if_(condition: PyExpr, if_true: PyExpr, if_false: PyExpr) -> PyExpr { + expr_fn::r#if(condition.into(), if_true.into(), if_false.into()).into() +} + +// `spark_cast` is config-injected by the upstream `expr_fn` helper; defaults +// applied automatically there. +spark_expr_fn!(spark_cast, arg1 arg2); + +// --------------------------------------------------------------------------- +// Datetime functions +// --------------------------------------------------------------------------- + +spark_expr_fn!(add_months, start_date num_months); +spark_expr_fn!(date_add, start_date days); +spark_expr_fn!(date_sub, start_date days); +spark_expr_fn!(hour, arg1); +spark_expr_fn!(minute, arg1); +spark_expr_fn!(second, arg1); +spark_expr_fn!(last_day, arg1); +spark_expr_fn!(make_dt_interval, days hours mins secs); +spark_expr_fn!(make_interval, years months weeks days hours mins secs); +spark_expr_fn!(next_day, start_date day_of_week); +spark_expr_fn!(date_diff, end_date start_date); +spark_expr_fn!(date_trunc, fmt ts); +spark_expr_fn!(time_trunc, fmt t); +spark_expr_fn!(trunc, dt fmt); +spark_expr_fn!(date_part, field source); +spark_expr_fn!(from_utc_timestamp, ts tz); +spark_expr_fn!(to_utc_timestamp, ts tz); +spark_expr_fn!(unix_date, dt); +spark_expr_fn!(unix_micros, ts); +spark_expr_fn!(unix_millis, ts); +spark_expr_fn!(unix_seconds, ts); + +// --------------------------------------------------------------------------- +// Hash functions +// --------------------------------------------------------------------------- + +spark_expr_fn!(crc32, arg1); +spark_expr_fn!(sha1, arg1); +spark_expr_fn!(sha2, arg1 bit_length); +spark_udf_vec!(xxhash64, udf::hash::xxhash64); + +// --------------------------------------------------------------------------- +// JSON functions +// --------------------------------------------------------------------------- + +spark_udf_vec!(json_tuple, udf::json::json_tuple); + +// --------------------------------------------------------------------------- +// Map functions +// --------------------------------------------------------------------------- + +spark_expr_fn!(map_from_arrays, keys values); +spark_expr_fn!(map_from_entries, arg1); +spark_expr_fn!(str_to_map, text pair_delim key_value_delim); + +// --------------------------------------------------------------------------- +// Math functions +// --------------------------------------------------------------------------- + +spark_expr_fn!(abs, arg1); +spark_expr_fn!(ceil, arg1); +spark_expr_fn!(expm1, arg1); +spark_expr_fn!(factorial, arg1); +spark_expr_fn!(floor, arg1); +spark_expr_fn!(hex, arg1); +spark_expr_fn!(modulus, dividend divisor); +spark_expr_fn!(pmod, dividend divisor); +spark_expr_fn!(rint, arg1); +spark_expr_fn!(round, value scale); +spark_expr_fn!(unhex, arg1); +spark_expr_fn!(width_bucket, value min_value max_value num_buckets); +spark_expr_fn!(csc, arg1); +spark_expr_fn!(sec, arg1); +spark_expr_fn!(negative, arg1); +spark_expr_fn!(bin, arg1); + +// --------------------------------------------------------------------------- +// String functions +// --------------------------------------------------------------------------- + +spark_expr_fn!(ascii, arg1); +spark_expr_fn!(base64, bin_input); +// `char` collides with the Rust primitive type in macro hygiene; rename the +// Rust ident and re-expose under the original name to Python. +#[pyfunction] +#[pyo3(name = "char")] +fn char_fn(arg1: PyExpr) -> PyExpr { + expr_fn::char(arg1.into()).into() +} +spark_udf_vec!(concat, udf::string::concat); +spark_udf_vec!(elt, udf::string::elt); +spark_expr_fn!(ilike, str pattern); +spark_expr_fn!(length, arg1); +spark_expr_fn!(like, str pattern); +spark_expr_fn!(luhn_check, arg1); +spark_udf_vec!(format_string, udf::string::format_string); +spark_expr_fn!(space, arg1); +spark_expr_fn!(substring, str pos length); +spark_expr_fn!(unbase64, str); +spark_expr_fn!(soundex, str); +spark_expr_fn!(is_valid_utf8, str); +spark_expr_fn!(make_valid_utf8, str); + +// --------------------------------------------------------------------------- +// URL functions +// --------------------------------------------------------------------------- + +spark_udf_vec!(parse_url, udf::url::parse_url); +spark_udf_vec!(try_parse_url, udf::url::try_parse_url); +spark_udf_vec!(url_decode, udf::url::url_decode); +spark_udf_vec!(try_url_decode, udf::url::try_url_decode); +spark_udf_vec!(url_encode, udf::url::url_encode); + +// --------------------------------------------------------------------------- +// Module init +// --------------------------------------------------------------------------- + +pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { + // Aggregate + m.add_wrapped(wrap_pyfunction!(avg))?; + m.add_wrapped(wrap_pyfunction!(try_sum))?; + m.add_wrapped(wrap_pyfunction!(collect_list))?; + m.add_wrapped(wrap_pyfunction!(collect_set))?; + // Array + m.add_wrapped(wrap_pyfunction!(array_contains))?; + m.add_wrapped(wrap_pyfunction!(array))?; + m.add_wrapped(wrap_pyfunction!(shuffle))?; + m.add_wrapped(wrap_pyfunction!(array_repeat))?; + m.add_wrapped(wrap_pyfunction!(slice))?; + // Bitmap + m.add_wrapped(wrap_pyfunction!(bitmap_count))?; + m.add_wrapped(wrap_pyfunction!(bitmap_bit_position))?; + m.add_wrapped(wrap_pyfunction!(bitmap_bucket_number))?; + // Bitwise + m.add_wrapped(wrap_pyfunction!(bit_get))?; + m.add_wrapped(wrap_pyfunction!(bit_count))?; + m.add_wrapped(wrap_pyfunction!(bitwise_not))?; + m.add_wrapped(wrap_pyfunction!(shiftleft))?; + m.add_wrapped(wrap_pyfunction!(shiftright))?; + m.add_wrapped(wrap_pyfunction!(shiftrightunsigned))?; + // Collection + m.add_wrapped(wrap_pyfunction!(size))?; + // Conditional + m.add_wrapped(wrap_pyfunction!(if_))?; + // Conversion + m.add_wrapped(wrap_pyfunction!(spark_cast))?; + // Datetime + m.add_wrapped(wrap_pyfunction!(add_months))?; + m.add_wrapped(wrap_pyfunction!(date_add))?; + m.add_wrapped(wrap_pyfunction!(date_sub))?; + m.add_wrapped(wrap_pyfunction!(hour))?; + m.add_wrapped(wrap_pyfunction!(minute))?; + m.add_wrapped(wrap_pyfunction!(second))?; + m.add_wrapped(wrap_pyfunction!(last_day))?; + m.add_wrapped(wrap_pyfunction!(make_dt_interval))?; + m.add_wrapped(wrap_pyfunction!(make_interval))?; + m.add_wrapped(wrap_pyfunction!(next_day))?; + m.add_wrapped(wrap_pyfunction!(date_diff))?; + m.add_wrapped(wrap_pyfunction!(date_trunc))?; + m.add_wrapped(wrap_pyfunction!(time_trunc))?; + m.add_wrapped(wrap_pyfunction!(trunc))?; + m.add_wrapped(wrap_pyfunction!(date_part))?; + m.add_wrapped(wrap_pyfunction!(from_utc_timestamp))?; + m.add_wrapped(wrap_pyfunction!(to_utc_timestamp))?; + m.add_wrapped(wrap_pyfunction!(unix_date))?; + m.add_wrapped(wrap_pyfunction!(unix_micros))?; + m.add_wrapped(wrap_pyfunction!(unix_millis))?; + m.add_wrapped(wrap_pyfunction!(unix_seconds))?; + // Hash + m.add_wrapped(wrap_pyfunction!(crc32))?; + m.add_wrapped(wrap_pyfunction!(sha1))?; + m.add_wrapped(wrap_pyfunction!(sha2))?; + m.add_wrapped(wrap_pyfunction!(xxhash64))?; + // JSON + m.add_wrapped(wrap_pyfunction!(json_tuple))?; + // Map + m.add_wrapped(wrap_pyfunction!(map_from_arrays))?; + m.add_wrapped(wrap_pyfunction!(map_from_entries))?; + m.add_wrapped(wrap_pyfunction!(str_to_map))?; + // Math + m.add_wrapped(wrap_pyfunction!(abs))?; + m.add_wrapped(wrap_pyfunction!(ceil))?; + m.add_wrapped(wrap_pyfunction!(expm1))?; + m.add_wrapped(wrap_pyfunction!(factorial))?; + m.add_wrapped(wrap_pyfunction!(floor))?; + m.add_wrapped(wrap_pyfunction!(hex))?; + m.add_wrapped(wrap_pyfunction!(modulus))?; + m.add_wrapped(wrap_pyfunction!(pmod))?; + m.add_wrapped(wrap_pyfunction!(rint))?; + m.add_wrapped(wrap_pyfunction!(round))?; + m.add_wrapped(wrap_pyfunction!(unhex))?; + m.add_wrapped(wrap_pyfunction!(width_bucket))?; + m.add_wrapped(wrap_pyfunction!(csc))?; + m.add_wrapped(wrap_pyfunction!(sec))?; + m.add_wrapped(wrap_pyfunction!(negative))?; + m.add_wrapped(wrap_pyfunction!(bin))?; + // String + m.add_wrapped(wrap_pyfunction!(ascii))?; + m.add_wrapped(wrap_pyfunction!(base64))?; + m.add_wrapped(wrap_pyfunction!(char_fn))?; + m.add_wrapped(wrap_pyfunction!(concat))?; + m.add_wrapped(wrap_pyfunction!(elt))?; + m.add_wrapped(wrap_pyfunction!(ilike))?; + m.add_wrapped(wrap_pyfunction!(length))?; + m.add_wrapped(wrap_pyfunction!(like))?; + m.add_wrapped(wrap_pyfunction!(luhn_check))?; + m.add_wrapped(wrap_pyfunction!(format_string))?; + m.add_wrapped(wrap_pyfunction!(space))?; + m.add_wrapped(wrap_pyfunction!(substring))?; + m.add_wrapped(wrap_pyfunction!(unbase64))?; + m.add_wrapped(wrap_pyfunction!(soundex))?; + m.add_wrapped(wrap_pyfunction!(is_valid_utf8))?; + m.add_wrapped(wrap_pyfunction!(make_valid_utf8))?; + // URL + m.add_wrapped(wrap_pyfunction!(parse_url))?; + m.add_wrapped(wrap_pyfunction!(try_parse_url))?; + m.add_wrapped(wrap_pyfunction!(url_decode))?; + m.add_wrapped(wrap_pyfunction!(try_url_decode))?; + m.add_wrapped(wrap_pyfunction!(url_encode))?; + Ok(()) +} diff --git a/src/sql.rs b/crates/core/src/sql.rs similarity index 100% rename from src/sql.rs rename to crates/core/src/sql.rs diff --git a/src/sql/exceptions.rs b/crates/core/src/sql/exceptions.rs similarity index 100% rename from src/sql/exceptions.rs rename to crates/core/src/sql/exceptions.rs diff --git a/src/sql/logical.rs b/crates/core/src/sql/logical.rs similarity index 91% rename from src/sql/logical.rs rename to crates/core/src/sql/logical.rs index 631aa9b09..12fc43bdc 100644 --- a/src/sql/logical.rs +++ b/crates/core/src/sql/logical.rs @@ -18,12 +18,13 @@ use std::sync::Arc; use datafusion::logical_expr::{DdlStatement, LogicalPlan, Statement}; -use datafusion_proto::logical_plan::{AsLogicalPlan, DefaultLogicalExtensionCodec}; +use datafusion_proto::logical_plan::AsLogicalPlan; use prost::Message; use pyo3::exceptions::PyRuntimeError; use pyo3::prelude::*; use pyo3::types::PyBytes; +use crate::codec::PythonLogicalCodec; use crate::context::PySessionContext; use crate::errors::PyDataFusionResult; use crate::expr::aggregate::PyAggregate; @@ -134,7 +135,7 @@ impl PyLogicalPlan { LogicalPlan::Dml(plan) => PyDmlStatement::from(plan.clone()).to_variant(py), LogicalPlan::Ddl(plan) => match plan { DdlStatement::CreateExternalTable(plan) => { - PyCreateExternalTable::from(plan.clone()).to_variant(py) + PyCreateExternalTable::from(plan.as_ref().clone()).to_variant(py) } DdlStatement::CreateMemoryTable(plan) => { PyCreateMemoryTable::from(plan.clone()).to_variant(py) @@ -153,7 +154,7 @@ impl PyLogicalPlan { PyDropCatalogSchema::from(plan.clone()).to_variant(py) } DdlStatement::CreateFunction(plan) => { - PyCreateFunction::from(plan.clone()).to_variant(py) + PyCreateFunction::from(plan.as_ref().clone()).to_variant(py) } DdlStatement::DropFunction(plan) => { PyDropFunction::from(plan.clone()).to_variant(py) @@ -196,17 +197,29 @@ impl PyLogicalPlan { format!("{}", self.plan.display_graphviz()) } - pub fn to_proto<'py>(&'py self, py: Python<'py>) -> PyDataFusionResult> { - let codec = DefaultLogicalExtensionCodec {}; + #[pyo3(signature = (ctx=None))] + pub fn to_bytes<'py>( + &'py self, + py: Python<'py>, + ctx: Option, + ) -> PyDataFusionResult> { + let default_codec; + let codec: &dyn datafusion_proto::logical_plan::LogicalExtensionCodec = match ctx { + Some(ref ctx) => ctx.logical_codec().as_ref(), + None => { + default_codec = PythonLogicalCodec::default(); + &default_codec + } + }; let proto = - datafusion_proto::protobuf::LogicalPlanNode::try_from_logical_plan(&self.plan, &codec)?; + datafusion_proto::protobuf::LogicalPlanNode::try_from_logical_plan(&self.plan, codec)?; let bytes = proto.encode_to_vec(); Ok(PyBytes::new(py, &bytes)) } #[staticmethod] - pub fn from_proto( + pub fn from_bytes( ctx: PySessionContext, proto_msg: Bound<'_, PyBytes>, ) -> PyDataFusionResult { @@ -218,8 +231,8 @@ impl PyLogicalPlan { )) })?; - let codec = DefaultLogicalExtensionCodec {}; - let plan = proto_plan.try_into_logical_plan(&ctx.ctx.task_ctx(), &codec)?; + let codec = ctx.logical_codec(); + let plan = proto_plan.try_into_logical_plan(&ctx.ctx.task_ctx(), codec.as_ref())?; Ok(Self::new(plan)) } } diff --git a/src/sql/util.rs b/crates/core/src/sql/util.rs similarity index 100% rename from src/sql/util.rs rename to crates/core/src/sql/util.rs diff --git a/src/store.rs b/crates/core/src/store.rs similarity index 100% rename from src/store.rs rename to crates/core/src/store.rs diff --git a/src/substrait.rs b/crates/core/src/substrait.rs similarity index 95% rename from src/substrait.rs rename to crates/core/src/substrait.rs index c2f112520..dcb5587cf 100644 --- a/src/substrait.rs +++ b/crates/core/src/substrait.rs @@ -15,6 +15,7 @@ // specific language governing permissions and limitations // under the License. +use datafusion_python_util::wait_for_future; use datafusion_substrait::logical_plan::{consumer, producer}; use datafusion_substrait::serializer; use datafusion_substrait::substrait::proto::Plan; @@ -25,7 +26,6 @@ use pyo3::types::PyBytes; use crate::context::PySessionContext; use crate::errors::{PyDataFusionError, PyDataFusionResult, py_datafusion_err, to_datafusion_err}; use crate::sql::logical::PyLogicalPlan; -use crate::utils::wait_for_future; #[pyclass( from_py_object, @@ -109,7 +109,7 @@ impl PySubstraitSerializer { ) -> PyDataFusionResult { PySubstraitSerializer::serialize_bytes(sql, ctx, py).and_then(|proto_bytes| { let proto_bytes = proto_bytes.bind(py).cast::().unwrap(); - PySubstraitSerializer::deserialize_bytes(proto_bytes.as_bytes().to_vec(), py) + PySubstraitSerializer::deserialize_bytes(proto_bytes.as_bytes().to_vec()) }) } @@ -131,8 +131,8 @@ impl PySubstraitSerializer { } #[staticmethod] - pub fn deserialize_bytes(proto_bytes: Vec, py: Python) -> PyDataFusionResult { - let plan = wait_for_future(py, serializer::deserialize_bytes(proto_bytes))??; + pub fn deserialize_bytes(proto_bytes: Vec) -> PyDataFusionResult { + let plan = serializer::deserialize_bytes(&proto_bytes)?; Ok(PyPlan { plan: *plan }) } } diff --git a/src/table.rs b/crates/core/src/table.rs similarity index 78% rename from src/table.rs rename to crates/core/src/table.rs index b9f30af9c..e0f0f0d13 100644 --- a/src/table.rs +++ b/crates/core/src/table.rs @@ -15,25 +15,29 @@ // specific language governing permissions and limitations // under the License. -use std::any::Any; use std::sync::Arc; use arrow::datatypes::SchemaRef; use arrow::pyarrow::ToPyArrow; use async_trait::async_trait; -use datafusion::catalog::Session; +use datafusion::catalog::{Session, TableProviderFactory}; use datafusion::common::Column; use datafusion::datasource::{TableProvider, TableType}; -use datafusion::logical_expr::{Expr, LogicalPlanBuilder, TableProviderFilterPushDown}; +use datafusion::logical_expr::{ + CreateExternalTable, Expr, LogicalPlanBuilder, TableProviderFilterPushDown, +}; use datafusion::physical_plan::ExecutionPlan; use datafusion::prelude::DataFrame; +use datafusion_ffi::proto::logical_extension_codec::FFI_LogicalExtensionCodec; +use datafusion_python_util::{create_logical_extension_capsule, table_provider_from_pycapsule}; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use crate::context::PySessionContext; use crate::dataframe::PyDataFrame; use crate::dataset::Dataset; -use crate::utils::table_provider_from_pycapsule; +use crate::errors; +use crate::expr::create_external_table::PyCreateExternalTable; /// This struct is used as a common method for all TableProviders, /// whether they refer to an FFI provider, an internally known @@ -145,10 +149,6 @@ impl TempViewTable { #[async_trait] impl TableProvider for TempViewTable { - fn as_any(&self) -> &dyn Any { - self - } - fn schema(&self) -> SchemaRef { Arc::new(self.df.schema().as_arrow().clone()) } @@ -206,3 +206,51 @@ impl TableProvider for TempViewTable { Ok(vec![TableProviderFilterPushDown::Exact; filters.len()]) } } + +#[derive(Debug)] +pub(crate) struct RustWrappedPyTableProviderFactory { + pub(crate) table_provider_factory: Py, + pub(crate) codec: Arc, +} + +impl RustWrappedPyTableProviderFactory { + pub fn new(table_provider_factory: Py, codec: Arc) -> Self { + Self { + table_provider_factory, + codec, + } + } + + fn create_inner( + &self, + cmd: CreateExternalTable, + codec: Bound, + ) -> PyResult> { + Python::attach(|py| { + let provider = self.table_provider_factory.bind(py); + let cmd = PyCreateExternalTable::from(cmd); + + provider + .call_method1("create", (cmd,)) + .and_then(|t| PyTable::new(t, Some(codec))) + .map(|t| t.table()) + }) + } +} + +#[async_trait] +impl TableProviderFactory for RustWrappedPyTableProviderFactory { + async fn create( + &self, + _: &dyn Session, + cmd: &CreateExternalTable, + ) -> datafusion::common::Result> { + Python::attach(|py| { + let codec = create_logical_extension_capsule(py, self.codec.as_ref()) + .map_err(errors::to_datafusion_err)?; + + self.create_inner(cmd.clone(), codec.into_any()) + .map_err(errors::to_datafusion_err) + }) + } +} diff --git a/src/udaf.rs b/crates/core/src/udaf.rs similarity index 53% rename from src/udaf.rs rename to crates/core/src/udaf.rs index 7ba499c66..caf7b97bc 100644 --- a/src/udaf.rs +++ b/crates/core/src/udaf.rs @@ -19,22 +19,22 @@ use std::ptr::NonNull; use std::sync::Arc; use datafusion::arrow::array::ArrayRef; -use datafusion::arrow::datatypes::DataType; +use datafusion::arrow::datatypes::{DataType, Field, FieldRef}; use datafusion::arrow::pyarrow::{PyArrowType, ToPyArrow}; use datafusion::common::ScalarValue; use datafusion::error::{DataFusionError, Result}; +use datafusion::logical_expr::function::{AccumulatorArgs, StateFieldsArgs}; use datafusion::logical_expr::{ - Accumulator, AccumulatorFactoryFunction, AggregateUDF, AggregateUDFImpl, create_udaf, + Accumulator, AccumulatorFactoryFunction, AggregateUDF, AggregateUDFImpl, Signature, Volatility, }; use datafusion_ffi::udaf::FFI_AggregateUDF; -use pyo3::ffi::c_str; +use datafusion_python_util::parse_volatility; use pyo3::prelude::*; use pyo3::types::{PyCapsule, PyTuple}; use crate::common::data_type::PyScalarValue; use crate::errors::{PyDataFusionResult, py_datafusion_err, to_datafusion_err}; use crate::expr::PyExpr; -use crate::utils::{parse_volatility, validate_pycapsule}; #[derive(Debug)] struct RustAccumulator { @@ -145,22 +145,173 @@ impl Accumulator for RustAccumulator { } } +fn instantiate_accumulator(accum: &Py) -> Result> { + let instance = Python::attach(|py| { + accum + .call0(py) + .map_err(|e| DataFusionError::Execution(format!("{e}"))) + })?; + Ok(Box::new(RustAccumulator::new(instance))) +} + +/// Wrap a Python accumulator factory in an `AccumulatorFactoryFunction`. +/// +/// Retained for downstream callers that previously consumed this +/// helper to build a [`AccumulatorFactoryFunction`] for `create_udaf` +/// or similar factory-based APIs. New in-crate code should construct +/// a [`PythonFunctionAggregateUDF`] directly so the codec can downcast +/// and ship it inline. pub fn to_rust_accumulator(accum: Py) -> AccumulatorFactoryFunction { - Arc::new(move |_args| -> Result> { - let accum = Python::attach(|py| { - accum - .call0(py) - .map_err(|e| DataFusionError::Execution(format!("{e}"))) - })?; - Ok(Box::new(RustAccumulator::new(accum))) - }) + Arc::new(move |_args| instantiate_accumulator(&accum)) } -fn aggregate_udf_from_capsule(capsule: &Bound<'_, PyCapsule>) -> PyDataFusionResult { - validate_pycapsule(capsule, "datafusion_aggregate_udf")?; +/// Named-struct `AggregateUDFImpl` for Python-defined aggregate UDFs. +/// Holds the Python accumulator factory directly so the codec can +/// downcast and cloudpickle it across process boundaries. +#[derive(Debug)] +pub(crate) struct PythonFunctionAggregateUDF { + name: String, + accumulator: Py, + signature: Signature, + return_type: DataType, + state_fields: Vec, +} + +impl PythonFunctionAggregateUDF { + fn new( + name: String, + accumulator: Py, + input_types: Vec, + return_type: DataType, + state_types: Vec, + volatility: Volatility, + ) -> Self { + let signature = Signature::exact(input_types, volatility); + let state_fields = state_types + .into_iter() + .enumerate() + .map(|(i, t)| Arc::new(Field::new(format!("state_{i}"), t, true))) + .collect(); + Self { + name, + accumulator, + signature, + return_type, + state_fields, + } + } + + /// Stored Python callable that returns a fresh accumulator instance + /// per partition. Consumed by the codec to cloudpickle the factory + /// across process boundaries. + pub(crate) fn accumulator(&self) -> &Py { + &self.accumulator + } + + pub(crate) fn return_type(&self) -> &DataType { + &self.return_type + } + + pub(crate) fn state_fields_ref(&self) -> &[FieldRef] { + &self.state_fields + } + + /// Reconstruct a `PythonFunctionAggregateUDF` from the parts emitted + /// by the codec. `state_fields` carries the full state schema + /// (names, data types, nullability, metadata) — the codec extracts + /// it from the IPC payload, so the post-decode state schema is + /// identical to the pre-encode one. Use [`Self::new`] when only + /// `Vec` is available (e.g. the Python constructor path, + /// where field names are synthesized). + pub(crate) fn from_parts( + name: String, + accumulator: Py, + input_types: Vec, + return_type: DataType, + state_fields: Vec, + volatility: Volatility, + ) -> Self { + Self { + name, + accumulator, + signature: Signature::exact(input_types, volatility), + return_type, + state_fields, + } + } +} +impl Eq for PythonFunctionAggregateUDF {} +impl PartialEq for PythonFunctionAggregateUDF { + fn eq(&self, other: &Self) -> bool { + self.name == other.name + && self.signature == other.signature + && self.return_type == other.return_type + && self.state_fields == other.state_fields + // Pointer-identity fast path: `Arc`-shared clones of the + // same UDF skip the GIL roundtrip. Falls through to Python + // `__eq__` only for two distinct callables. + && (self.accumulator.as_ptr() == other.accumulator.as_ptr() + || Python::attach(|py| { + // See `PythonFunctionScalarUDF::eq` for the + // rationale on swallowing the exception as `false` + // and logging at `debug`. FIXME: revisit if + // upstream `AggregateUDFImpl` exposes a fallible + // `PartialEq`. + self.accumulator + .bind(py) + .eq(other.accumulator.bind(py)) + .unwrap_or_else(|e| { + log::debug!( + target: "datafusion_python::udaf", + "PythonFunctionAggregateUDF {:?} __eq__ raised; treating as unequal: {e}", + self.name, + ); + false + }) + })) + } +} + +impl std::hash::Hash for PythonFunctionAggregateUDF { + fn hash(&self, state: &mut H) { + // See `PythonFunctionScalarUDF`'s `Hash` impl for the + // rationale: hash the identifying header only and let + // `PartialEq` disambiguate callables. + self.name.hash(state); + self.signature.hash(state); + self.return_type.hash(state); + for f in &self.state_fields { + f.hash(state); + } + } +} + +impl AggregateUDFImpl for PythonFunctionAggregateUDF { + fn name(&self) -> &str { + &self.name + } + + fn signature(&self) -> &Signature { + &self.signature + } + + fn return_type(&self, _arg_types: &[DataType]) -> Result { + Ok(self.return_type.clone()) + } + + fn accumulator(&self, _acc_args: AccumulatorArgs) -> Result> { + instantiate_accumulator(&self.accumulator) + } + + fn state_fields(&self, _args: StateFieldsArgs) -> Result> { + Ok(self.state_fields.clone()) + } +} + +fn aggregate_udf_from_capsule(capsule: &Bound<'_, PyCapsule>) -> PyDataFusionResult { let data: NonNull = capsule - .pointer_checked(Some(c_str!("datafusion_aggregate_udf")))? + .pointer_checked(Some(c"datafusion_aggregate_udf"))? .cast(); let udaf = unsafe { data.as_ref() }; let udaf: Arc = udaf.into(); @@ -193,14 +344,15 @@ impl PyAggregateUDF { state_type: PyArrowType>, volatility: &str, ) -> PyResult { - let function = create_udaf( - name, + let py_udf = PythonFunctionAggregateUDF::new( + name.to_string(), + accumulator, input_type.0, - Arc::new(return_type.0), + return_type.0, + state_type.0, parse_volatility(volatility)?, - to_rust_accumulator(accumulator), - Arc::new(state_type.0), ); + let function = AggregateUDF::new_from_impl(py_udf); Ok(Self { function }) } @@ -234,4 +386,9 @@ impl PyAggregateUDF { fn __repr__(&self) -> PyResult { Ok(format!("AggregateUDF({})", self.function.name())) } + + #[getter] + fn name(&self) -> &str { + self.function.name() + } } diff --git a/src/udf.rs b/crates/core/src/udf.rs similarity index 64% rename from src/udf.rs rename to crates/core/src/udf.rs index 2d60abc09..2006401db 100644 --- a/src/udf.rs +++ b/crates/core/src/udf.rs @@ -15,7 +15,6 @@ // specific language governing permissions and limitations // under the License. -use std::any::Any; use std::hash::{Hash, Hasher}; use std::ptr::NonNull; use std::sync::Arc; @@ -32,19 +31,18 @@ use datafusion::logical_expr::{ Volatility, }; use datafusion_ffi::udf::FFI_ScalarUDF; -use pyo3::ffi::c_str; +use datafusion_python_util::parse_volatility; use pyo3::prelude::*; use pyo3::types::{PyCapsule, PyTuple}; use crate::array::PyArrowArrayExportable; -use crate::errors::{PyDataFusionResult, py_datafusion_err, to_datafusion_err}; +use crate::errors::{PyDataFusionResult, to_datafusion_err}; use crate::expr::PyExpr; -use crate::utils::{parse_volatility, validate_pycapsule}; /// This struct holds the Python written function that is a /// ScalarUDF. #[derive(Debug)] -struct PythonFunctionScalarUDF { +pub(crate) struct PythonFunctionScalarUDF { name: String, func: Py, signature: Signature, @@ -68,6 +66,37 @@ impl PythonFunctionScalarUDF { return_field: Arc::new(return_field), } } + + /// Stored Python callable. Consumed by the codec to cloudpickle + /// the function body across process boundaries. + pub(crate) fn func(&self) -> &Py { + &self.func + } + + pub(crate) fn return_field(&self) -> &FieldRef { + &self.return_field + } + + /// Reconstruct a `PythonFunctionScalarUDF` from the parts emitted + /// by the codec. Inputs collapse to `Vec` because + /// `Signature::exact` cannot carry per-input nullability or + /// metadata — the encoder is free to discard that side of the + /// schema. `return_field` is kept as a `Field` so the post-decode + /// nullability and metadata match the sender's instance. + pub(crate) fn from_parts( + name: String, + func: Py, + input_types: Vec, + return_field: Field, + volatility: Volatility, + ) -> Self { + Self { + name, + func, + signature: Signature::exact(input_types, volatility), + return_field: Arc::new(return_field), + } + } } impl Eq for PythonFunctionScalarUDF {} @@ -76,29 +105,55 @@ impl PartialEq for PythonFunctionScalarUDF { self.name == other.name && self.signature == other.signature && self.return_field == other.return_field - && Python::attach(|py| self.func.bind(py).eq(other.func.bind(py)).unwrap_or(false)) + // Identical pointers ⇒ same Python object. Most equality + // checks compare `Arc`-shared clones of the same UDF + // (e.g. expression rewriting), so the pointer match short- + // circuits before touching the GIL. + && (self.func.as_ptr() == other.func.as_ptr() + || Python::attach(|py| { + // Rust's `PartialEq` cannot return `Result`, so we + // have to pick a side when Python `__eq__` raises. + // `false` is the conservative choice — better to + // report two UDFs as distinct than to wrongly + // merge them — but the silent miss can still + // surface as expression-dedup or cache-lookup + // anomalies. Log at `debug` so the failure is + // observable without flooding production logs. + // FIXME: revisit if upstream `ScalarUDFImpl` + // exposes a fallible `PartialEq`. + self.func + .bind(py) + .eq(other.func.bind(py)) + .unwrap_or_else(|e| { + log::debug!( + target: "datafusion_python::udf", + "PythonFunctionScalarUDF {:?} __eq__ raised; treating as unequal: {e}", + self.name, + ); + false + }) + })) } } impl Hash for PythonFunctionScalarUDF { fn hash(&self, state: &mut H) { + // Hash only the identifying header (name + signature + return + // field). Skipping `func` is intentional: the Rust `Hash` + // contract requires `a == b ⇒ hash(a) == hash(b)`, not the + // converse, so a coarser hash is sound — `PartialEq` still + // disambiguates two UDFs with the same header but distinct + // callables. Falling back to a sentinel on `py_hash` failure + // (as a prior revision did) silently mapped every unhashable + // closure to the same bucket; that is the worst case for a + // hashmap and is what this rewrite avoids. self.name.hash(state); self.signature.hash(state); self.return_field.hash(state); - - Python::attach(|py| { - let py_hash = self.func.bind(py).hash().unwrap_or(0); // Handle unhashable objects - - state.write_isize(py_hash); - }); } } impl ScalarUDFImpl for PythonFunctionScalarUDF { - fn as_any(&self) -> &dyn Any { - self - } - fn name(&self) -> &str { &self.name } @@ -194,11 +249,9 @@ impl PyScalarUDF { pub fn from_pycapsule(func: Bound<'_, PyAny>) -> PyDataFusionResult { if func.hasattr("__datafusion_scalar_udf__")? { let capsule = func.getattr("__datafusion_scalar_udf__")?.call0()?; - let capsule = capsule.cast::().map_err(py_datafusion_err)?; - validate_pycapsule(capsule, "datafusion_scalar_udf")?; - + let capsule = capsule.cast::().map_err(to_datafusion_err)?; let data: NonNull = capsule - .pointer_checked(Some(c_str!("datafusion_scalar_udf")))? + .pointer_checked(Some(c"datafusion_scalar_udf"))? .cast(); let udf = unsafe { data.as_ref() }; let udf: Arc = udf.into(); @@ -223,4 +276,9 @@ impl PyScalarUDF { fn __repr__(&self) -> PyResult { Ok(format!("ScalarUDF({})", self.function.name())) } + + #[getter] + fn name(&self) -> &str { + self.function.name() + } } diff --git a/crates/core/src/udtf.rs b/crates/core/src/udtf.rs new file mode 100644 index 000000000..cffa0c12a --- /dev/null +++ b/crates/core/src/udtf.rs @@ -0,0 +1,202 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::ptr::NonNull; +use std::sync::Arc; + +use datafusion::catalog::{Session, TableFunctionArgs, TableFunctionImpl, TableProvider}; +use datafusion::error::{DataFusionError, Result as DataFusionResult}; +use datafusion::execution::context::SessionContext; +use datafusion::execution::session_state::SessionState; +use datafusion::logical_expr::Expr; +use datafusion_ffi::udtf::FFI_TableFunction; +use pyo3::IntoPyObjectExt; +use pyo3::exceptions::{PyImportError, PyTypeError}; +use pyo3::prelude::*; +use pyo3::types::{PyCapsule, PyDict, PyTuple, PyType}; + +use crate::context::PySessionContext; +use crate::errors::{py_datafusion_err, to_datafusion_err}; +use crate::expr::PyExpr; +use crate::table::PyTable; + +/// A pure-Python UDTF callable plus the metadata we discovered about it +/// at registration time. +#[derive(Debug, Clone)] +pub(crate) struct PythonTableFunctionCallable { + pub(crate) callable: Arc>, + /// When true, the calling :class:`SessionContext` is passed to the + /// callable as a ``session`` keyword argument on every invocation. + /// Opt-in at registration time via ``with_session=True`` on the + /// Python wrapper. + pub(crate) inject_session_on_call: bool, +} + +/// Represents a user defined table function +#[pyclass(from_py_object, frozen, name = "TableFunction", module = "datafusion")] +#[derive(Debug, Clone)] +pub struct PyTableFunction { + pub(crate) name: String, + pub(crate) inner: PyTableFunctionInner, +} + +#[derive(Debug, Clone)] +pub(crate) enum PyTableFunctionInner { + PythonFunction(PythonTableFunctionCallable), + FFIFunction(Arc), +} + +#[pymethods] +impl PyTableFunction { + #[new] + #[pyo3(signature=(name, func, session, inject_session_on_call=false))] + pub fn new( + name: &str, + func: Bound<'_, PyAny>, + session: Option>, + inject_session_on_call: bool, + ) -> PyResult { + let inner = if func.hasattr("__datafusion_table_function__")? { + let py = func.py(); + let session = match session { + Some(session) => session, + None => PySessionContext::global_ctx()?.into_bound_py_any(py)?, + }; + let capsule = func + .getattr("__datafusion_table_function__")? + .call1((session,)).map_err(|err| { + if err.get_type(py).is(PyType::new::(py)) { + PyImportError::new_err("Incompatible libraries. DataFusion 52.0.0 introduced an incompatible signature change for table functions. Either downgrade DataFusion or upgrade your function library.") + } else { + err + } + })?; + let capsule = capsule.cast::()?; + let data: NonNull = capsule + .pointer_checked(Some(c"datafusion_table_function"))? + .cast(); + let ffi_func = unsafe { data.as_ref() }; + let foreign_func: Arc = ffi_func.to_owned().into(); + + PyTableFunctionInner::FFIFunction(foreign_func) + } else { + PyTableFunctionInner::PythonFunction(PythonTableFunctionCallable { + callable: Arc::new(func.unbind()), + inject_session_on_call, + }) + }; + + Ok(Self { + name: name.to_string(), + inner, + }) + } + + #[pyo3(signature = (*args))] + pub fn __call__(&self, args: Vec) -> PyResult { + let args: Vec = args.iter().map(|e| e.expr.clone()).collect(); + let global = PySessionContext::global_ctx()?; + let state = global.ctx.state(); + let table_provider = self + .call_with_args(TableFunctionArgs::new(&args, &state)) + .map_err(py_datafusion_err)?; + + Ok(PyTable::from(table_provider)) + } + + fn __repr__(&self) -> PyResult { + Ok(format!("TableUDF({})", self.name)) + } +} + +/// Materialize a fresh :class:`PySessionContext` from the borrowed +/// ``&dyn Session`` handed in at call time. +/// +/// Upstream invokes ``call_with_args`` with a trait-object reference +/// rather than an owned context; we downcast it to the canonical +/// :class:`SessionState` impl and rebuild a :class:`SessionContext` +/// (sharing the same registries via the Arc-heavy interior of +/// :class:`SessionState`). +/// +/// The downcast is defensive. Every path that reaches a pure-Python +/// UDTF today hands us a `SessionState`: the SQL planner builds the +/// args from its own `SessionState`, and `PyTableFunction::__call__` +/// uses the global context's state. A non-`SessionState` session +/// (e.g. a `ForeignSession`) would only arrive if this UDTF were +/// exported across the FFI boundary to a foreign-library consumer, +/// which datafusion-python does not do. Should that change, this +/// returns an error rather than silently misbehaving. +fn py_session_from_session(session: &dyn Session) -> DataFusionResult { + let state = session + .as_any() + .downcast_ref::() + .ok_or_else(|| { + DataFusionError::Execution( + "Cannot expose this UDTF's calling session to Python: the \ + session is not a SessionState. Drop the `session` keyword \ + from the callback signature to fall back to the \ + expression-only call form." + .to_string(), + ) + })?; + Ok(PySessionContext::from(SessionContext::new_with_state( + state.clone(), + ))) +} + +#[allow(clippy::result_large_err)] +fn call_python_table_function( + func: &PythonTableFunctionCallable, + args: TableFunctionArgs, +) -> DataFusionResult> { + let py_session = if func.inject_session_on_call { + Some(py_session_from_session(args.session())?) + } else { + None + }; + let py_exprs = args + .exprs() + .iter() + .map(|arg| PyExpr::from(arg.clone())) + .collect::>(); + + Python::attach(|py| { + let py_args = PyTuple::new(py, py_exprs)?; + let provider_obj = if let Some(session) = py_session { + let kwargs = PyDict::new(py); + kwargs.set_item("session", session.into_pyobject(py)?)?; + func.callable.call(py, py_args, Some(&kwargs))? + } else { + func.callable.call1(py, py_args)? + }; + let provider = provider_obj.bind(py).clone(); + + Ok::, PyErr>(PyTable::new(provider, None)?.table) + }) + .map_err(to_datafusion_err) +} + +impl TableFunctionImpl for PyTableFunction { + fn call_with_args(&self, args: TableFunctionArgs) -> DataFusionResult> { + match &self.inner { + PyTableFunctionInner::FFIFunction(func) => func.call_with_args(args), + PyTableFunctionInner::PythonFunction(callable) => { + call_python_table_function(callable, args) + } + } + } +} diff --git a/src/udwf.rs b/crates/core/src/udwf.rs similarity index 69% rename from src/udwf.rs rename to crates/core/src/udwf.rs index de63e2f9a..ebec8f3bd 100644 --- a/src/udwf.rs +++ b/crates/core/src/udwf.rs @@ -15,7 +15,6 @@ // specific language governing permissions and limitations // under the License. -use std::any::Any; use std::ops::Range; use std::ptr::NonNull; use std::sync::Arc; @@ -25,22 +24,20 @@ use datafusion::arrow::datatypes::DataType; use datafusion::arrow::pyarrow::{FromPyArrow, PyArrowType, ToPyArrow}; use datafusion::error::{DataFusionError, Result}; use datafusion::logical_expr::function::{PartitionEvaluatorArgs, WindowUDFFieldArgs}; -use datafusion::logical_expr::ptr_eq::PtrEq; use datafusion::logical_expr::window_state::WindowAggState; use datafusion::logical_expr::{ PartitionEvaluator, PartitionEvaluatorFactory, Signature, Volatility, WindowUDF, WindowUDFImpl, }; use datafusion::scalar::ScalarValue; use datafusion_ffi::udwf::FFI_WindowUDF; +use datafusion_python_util::parse_volatility; use pyo3::exceptions::PyValueError; -use pyo3::ffi::c_str; use pyo3::prelude::*; use pyo3::types::{PyCapsule, PyList, PyTuple}; use crate::common::data_type::PyScalarValue; -use crate::errors::{PyDataFusionResult, py_datafusion_err, to_datafusion_err}; +use crate::errors::{PyDataFusionResult, to_datafusion_err}; use crate::expr::PyExpr; -use crate::utils::{parse_volatility, validate_pycapsule}; #[derive(Debug)] struct RustPartitionEvaluator { @@ -199,15 +196,24 @@ impl PartitionEvaluator for RustPartitionEvaluator { } } +fn instantiate_partition_evaluator(evaluator: &Py) -> Result> { + let instance = Python::attach(|py| { + evaluator + .call0(py) + .map_err(|e| DataFusionError::Execution(e.to_string())) + })?; + Ok(Box::new(RustPartitionEvaluator::new(instance))) +} + +/// Wrap a Python evaluator factory in a `PartitionEvaluatorFactory`. +/// +/// Retained for downstream callers that previously consumed this +/// helper to build a [`PartitionEvaluatorFactory`] for factory-based +/// APIs. New in-crate code should construct a +/// [`PythonFunctionWindowUDF`] directly so the codec can downcast and +/// ship it inline. pub fn to_rust_partition_evaluator(evaluator: Py) -> PartitionEvaluatorFactory { - Arc::new(move || -> Result> { - let evaluator = Python::attach(|py| { - evaluator - .call0(py) - .map_err(|e| DataFusionError::Execution(e.to_string())) - })?; - Ok(Box::new(RustPartitionEvaluator::new(evaluator))) - }) + Arc::new(move || instantiate_partition_evaluator(&evaluator)) } /// Represents an WindowUDF @@ -235,14 +241,14 @@ impl PyWindowUDF { volatility: &str, ) -> PyResult { let return_type = return_type.0; - let input_types = input_types.into_iter().map(|t| t.0).collect(); + let input_types: Vec = input_types.into_iter().map(|t| t.0).collect(); - let function = WindowUDF::from(MultiColumnWindowUDF::new( + let function = WindowUDF::from(PythonFunctionWindowUDF::new( name, + evaluator, input_types, return_type, parse_volatility(volatility)?, - to_rust_partition_evaluator(evaluator), )); Ok(Self { function }) } @@ -262,11 +268,9 @@ impl PyWindowUDF { func }; - let capsule = capsule.cast::().map_err(py_datafusion_err)?; - validate_pycapsule(capsule, "datafusion_window_udf")?; - + let capsule = capsule.cast::().map_err(to_datafusion_err)?; let data: NonNull = capsule - .pointer_checked(Some(c_str!("datafusion_window_udf")))? + .pointer_checked(Some(c"datafusion_window_udf"))? .cast(); let udwf = unsafe { data.as_ref() }; let udwf: Arc = udwf.into(); @@ -279,51 +283,104 @@ impl PyWindowUDF { fn __repr__(&self) -> PyResult { Ok(format!("WindowUDF({})", self.function.name())) } + + #[getter] + fn name(&self) -> &str { + self.function.name() + } } -#[derive(Hash, Eq, PartialEq)] -pub struct MultiColumnWindowUDF { +/// `WindowUDFImpl` for Python-defined window UDFs. +/// +/// Holds the Python evaluator factory directly so the codec can +/// downcast and cloudpickle it across process boundaries. Replaces +/// the prior factory-erased `MultiColumnWindowUDF`; the old name is +/// kept as a type alias below for backward compatibility. +#[derive(Debug)] +pub struct PythonFunctionWindowUDF { name: String, + evaluator: Py, signature: Signature, return_type: DataType, - partition_evaluator_factory: PtrEq, } -impl std::fmt::Debug for MultiColumnWindowUDF { - fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result { - f.debug_struct("WindowUDF") - .field("name", &self.name) - .field("signature", &self.signature) - .field("return_type", &"") - .field("partition_evaluator_factory", &"") - .finish() - } -} +/// Backward-compatible alias for downstream crates that referenced the +/// previous struct name. New code should use [`PythonFunctionWindowUDF`]. +pub type MultiColumnWindowUDF = PythonFunctionWindowUDF; -impl MultiColumnWindowUDF { +impl PythonFunctionWindowUDF { pub fn new( name: impl Into, + evaluator: Py, input_types: Vec, return_type: DataType, volatility: Volatility, - partition_evaluator_factory: PartitionEvaluatorFactory, ) -> Self { let name = name.into(); let signature = Signature::exact(input_types, volatility); Self { name, + evaluator, signature, return_type, - partition_evaluator_factory: partition_evaluator_factory.into(), } } + + /// Stored Python callable that produces a fresh partition + /// evaluator instance per partition. Consumed by the codec to + /// cloudpickle the evaluator factory across process boundaries. + pub(crate) fn evaluator(&self) -> &Py { + &self.evaluator + } + + pub(crate) fn return_type(&self) -> &DataType { + &self.return_type + } } -impl WindowUDFImpl for MultiColumnWindowUDF { - fn as_any(&self) -> &dyn Any { - self +impl Eq for PythonFunctionWindowUDF {} +impl PartialEq for PythonFunctionWindowUDF { + fn eq(&self, other: &Self) -> bool { + self.name == other.name + && self.signature == other.signature + && self.return_type == other.return_type + // Pointer-identity fast path: `Arc`-shared clones of the + // same UDF skip the GIL roundtrip. Falls through to Python + // `__eq__` only for two distinct callables. + && (self.evaluator.as_ptr() == other.evaluator.as_ptr() + || Python::attach(|py| { + // See `PythonFunctionScalarUDF::eq` for the + // rationale on swallowing the exception as `false` + // and logging at `debug`. FIXME: revisit if + // upstream `WindowUDFImpl` exposes a fallible + // `PartialEq`. + self.evaluator + .bind(py) + .eq(other.evaluator.bind(py)) + .unwrap_or_else(|e| { + log::debug!( + target: "datafusion_python::udwf", + "PythonFunctionWindowUDF {:?} __eq__ raised; treating as unequal: {e}", + self.name, + ); + false + }) + })) } +} + +impl std::hash::Hash for PythonFunctionWindowUDF { + fn hash(&self, state: &mut H) { + // See `PythonFunctionScalarUDF`'s `Hash` impl for the + // rationale: hash the identifying header only and let + // `PartialEq` disambiguate evaluators. + self.name.hash(state); + self.signature.hash(state); + self.return_type.hash(state); + } +} +impl WindowUDFImpl for PythonFunctionWindowUDF { fn name(&self) -> &str { &self.name } @@ -342,7 +399,6 @@ impl WindowUDFImpl for MultiColumnWindowUDF { &self, _partition_evaluator_args: PartitionEvaluatorArgs, ) -> Result> { - let _ = _partition_evaluator_args; - (self.partition_evaluator_factory)() + instantiate_partition_evaluator(&self.evaluator) } } diff --git a/src/unparser/dialect.rs b/crates/core/src/unparser/dialect.rs similarity index 100% rename from src/unparser/dialect.rs rename to crates/core/src/unparser/dialect.rs diff --git a/src/unparser/mod.rs b/crates/core/src/unparser/mod.rs similarity index 100% rename from src/unparser/mod.rs rename to crates/core/src/unparser/mod.rs diff --git a/python/datafusion/html_formatter.py b/crates/util/Cargo.toml similarity index 60% rename from python/datafusion/html_formatter.py rename to crates/util/Cargo.toml index 65eb1f042..c23667b0f 100644 --- a/python/datafusion/html_formatter.py +++ b/crates/util/Cargo.toml @@ -15,15 +15,21 @@ # specific language governing permissions and limitations # under the License. -"""Deprecated module for dataframe formatting.""" +[package] +name = "datafusion-python-util" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +description.workspace = true +homepage.workspace = true +repository.workspace = true -import warnings - -from datafusion.dataframe_formatter import * # noqa: F403 - -warnings.warn( - "The module 'html_formatter' is deprecated and will be removed in the next release." - "Please use 'dataframe_formatter' instead.", - DeprecationWarning, - stacklevel=3, -) +[dependencies] +tokio = { workspace = true, features = ["macros", "rt", "rt-multi-thread"] } +pyo3 = { workspace = true } +datafusion = { workspace = true } +datafusion-ffi = { workspace = true } +datafusion-proto = { workspace = true } +arrow = { workspace = true } +prost = { workspace = true } diff --git a/src/errors.rs b/crates/util/src/errors.rs similarity index 100% rename from src/errors.rs rename to crates/util/src/errors.rs diff --git a/src/utils.rs b/crates/util/src/lib.rs similarity index 54% rename from src/utils.rs rename to crates/util/src/lib.rs index 5085018f7..9327d7f2f 100644 --- a/src/utils.rs +++ b/crates/util/src/lib.rs @@ -21,37 +21,41 @@ use std::sync::{Arc, OnceLock}; use std::time::Duration; use datafusion::datasource::TableProvider; +use datafusion::execution::TaskContext; use datafusion::execution::context::SessionContext; use datafusion::logical_expr::Volatility; +use datafusion::physical_optimizer::PhysicalOptimizerRule; +use datafusion_ffi::execution::FFI_TaskContextProvider; +use datafusion_ffi::physical_optimizer::FFI_PhysicalOptimizerRule; use datafusion_ffi::proto::logical_extension_codec::FFI_LogicalExtensionCodec; +use datafusion_ffi::proto::physical_extension_codec::FFI_PhysicalExtensionCodec; use datafusion_ffi::table_provider::FFI_TableProvider; -use pyo3::IntoPyObjectExt; +use datafusion_proto::physical_plan::PhysicalExtensionCodec; use pyo3::exceptions::{PyImportError, PyTypeError, PyValueError}; -use pyo3::ffi::c_str; use pyo3::prelude::*; use pyo3::types::{PyCapsule, PyType}; use tokio::runtime::Runtime; use tokio::task::JoinHandle; use tokio::time::sleep; -use crate::TokioRuntime; -use crate::context::PySessionContext; -use crate::errors::{PyDataFusionError, PyDataFusionResult, py_datafusion_err, to_datafusion_err}; +pub mod errors; +pub use crate::errors::to_datafusion_err; +use crate::errors::{PyDataFusionError, PyDataFusionResult}; /// Utility to get the Tokio Runtime from Python #[inline] -pub(crate) fn get_tokio_runtime() -> &'static TokioRuntime { +pub fn get_tokio_runtime() -> &'static Runtime { // NOTE: Other pyo3 python libraries have had issues with using tokio // behind a forking app-server like `gunicorn` // If we run into that problem, in the future we can look to `delta-rs` // which adds a check in that disallows calls from a forked process // https://github.com/delta-io/delta-rs/blob/87010461cfe01563d91a4b9cd6fa468e2ad5f283/python/src/utils.rs#L10-L31 - static RUNTIME: OnceLock = OnceLock::new(); - RUNTIME.get_or_init(|| TokioRuntime(tokio::runtime::Runtime::new().unwrap())) + static RUNTIME: OnceLock = OnceLock::new(); + RUNTIME.get_or_init(|| Runtime::new().unwrap()) } #[inline] -pub(crate) fn is_ipython_env(py: Python) -> &'static bool { +pub fn is_ipython_env(py: Python) -> &'static bool { static IS_IPYTHON_ENV: OnceLock = OnceLock::new(); IS_IPYTHON_ENV.get_or_init(|| { py.import("IPython") @@ -63,7 +67,7 @@ pub(crate) fn is_ipython_env(py: Python) -> &'static bool { /// Utility to get the Global Datafussion CTX #[inline] -pub(crate) fn get_global_ctx() -> &'static Arc { +pub fn get_global_ctx() -> &'static Arc { static CTX: OnceLock> = OnceLock::new(); CTX.get_or_init(|| Arc::new(SessionContext::new())) } @@ -77,7 +81,7 @@ where F: Future + Send, F::Output: Send, { - let runtime: &Runtime = &get_tokio_runtime().0; + let runtime: &Runtime = get_tokio_runtime(); const INTERVAL_CHECK_SIGNALS: Duration = Duration::from_millis(1_000); // Some fast running processes that generate many `wait_for_future` calls like @@ -111,12 +115,12 @@ where /// Spawn a [`Future`] on the Tokio runtime and wait for completion /// while respecting Python signal handling. -pub(crate) fn spawn_future(py: Python, fut: F) -> PyDataFusionResult +pub fn spawn_future(py: Python, fut: F) -> PyDataFusionResult where F: Future> + Send + 'static, T: Send + 'static, { - let rt = &get_tokio_runtime().0; + let rt = get_tokio_runtime(); let handle: JoinHandle> = rt.spawn(fut); // Wait for the join handle while respecting Python signal handling. // We handle errors in two steps so `?` maps the error types correctly: @@ -138,7 +142,7 @@ where Ok(inner_result?) } -pub(crate) fn parse_volatility(value: &str) -> PyDataFusionResult { +pub fn parse_volatility(value: &str) -> PyDataFusionResult { Ok(match value { "immutable" => Volatility::Immutable, "stable" => Volatility::Stable, @@ -152,7 +156,7 @@ pub(crate) fn parse_volatility(value: &str) -> PyDataFusionResult { }) } -pub(crate) fn validate_pycapsule(capsule: &Bound, name: &str) -> PyResult<()> { +pub fn validate_pycapsule(capsule: &Bound, name: &str) -> PyResult<()> { let capsule_name = capsule.name()?; if capsule_name.is_none() { return Err(PyValueError::new_err(format!( @@ -160,7 +164,8 @@ pub(crate) fn validate_pycapsule(capsule: &Bound, name: &str) -> PyRe ))); } - let capsule_name = unsafe { capsule_name.unwrap().as_cstr().to_str()? }; + let capsule_name = unsafe { capsule_name.unwrap().as_cstr().to_str() } + .map_err(|err| PyValueError::new_err(err.to_string()))?; if capsule_name != name { return Err(PyValueError::new_err(format!( "Expected name '{name}' in PyCapsule, instead got '{capsule_name}'" @@ -170,7 +175,7 @@ pub(crate) fn validate_pycapsule(capsule: &Bound, name: &str) -> PyRe Ok(()) } -pub(crate) fn table_provider_from_pycapsule<'py>( +pub fn table_provider_from_pycapsule<'py>( mut obj: Bound<'py, PyAny>, session: Bound<'py, PyAny>, ) -> PyResult>> { @@ -187,11 +192,9 @@ pub(crate) fn table_provider_from_pycapsule<'py>( })?; } - if let Ok(capsule) = obj.cast::().map_err(py_datafusion_err) { - validate_pycapsule(capsule, "datafusion_table_provider")?; - + if let Ok(capsule) = obj.cast::() { let data: NonNull = capsule - .pointer_checked(Some(c_str!("datafusion_table_provider")))? + .pointer_checked(Some(c"datafusion_table_provider"))? .cast(); let provider = unsafe { data.as_ref() }; let provider: Arc = provider.into(); @@ -202,37 +205,144 @@ pub(crate) fn table_provider_from_pycapsule<'py>( } } -pub(crate) fn extract_logical_extension_codec( - py: Python, - obj: Option>, -) -> PyResult> { - let obj = match obj { - Some(obj) => obj, - None => PySessionContext::global_ctx()?.into_bound_py_any(py)?, - }; - let capsule = if obj.hasattr("__datafusion_logical_extension_codec__")? { - obj.getattr("__datafusion_logical_extension_codec__")? - .call0()? +pub fn create_logical_extension_capsule<'py>( + py: Python<'py>, + codec: &FFI_LogicalExtensionCodec, +) -> PyResult> { + let codec = codec.clone(); + + PyCapsule::new_with_value(py, codec, cr"datafusion_logical_extension_codec") +} + +pub fn ffi_logical_codec_from_pycapsule(obj: Bound) -> PyResult { + let attr_name = "__datafusion_logical_extension_codec__"; + let capsule = if obj.hasattr(attr_name)? { + obj.getattr(attr_name)?.call0()? } else { obj }; - let capsule = capsule.cast::().map_err(py_datafusion_err)?; - - validate_pycapsule(capsule, "datafusion_logical_extension_codec")?; + let capsule = capsule.cast::()?; let data: NonNull = capsule - .pointer_checked(Some(c_str!("datafusion_logical_extension_codec")))? + .pointer_checked(Some(c"datafusion_logical_extension_codec"))? .cast(); let codec = unsafe { data.as_ref() }; - Ok(Arc::new(codec.clone())) + + Ok(codec.clone()) } -pub(crate) fn create_logical_extension_capsule<'py>( +pub fn create_physical_extension_capsule<'py>( py: Python<'py>, - codec: &FFI_LogicalExtensionCodec, + codec: &FFI_PhysicalExtensionCodec, ) -> PyResult> { - let name = cr"datafusion_logical_extension_codec".into(); let codec = codec.clone(); - PyCapsule::new(py, codec, Some(name)) + PyCapsule::new_with_value(py, codec, cr"datafusion_physical_extension_codec") +} + +/// Define a `(obj) -> PyResult>` extractor that +/// accepts either a raw `PyCapsule` carrying `$ffi_type` or any object +/// exposing `____()` that returns one. +/// +/// Use this when `Arc<$output_type>: From<&$ffi_type>` (infallible +/// conversion). For fallible conversions use [`try_from_pycapsule!`] +/// instead. +#[macro_export] +macro_rules! from_pycapsule { + ($fn_name:ident, $capsule_name:literal, $ffi_type:ty, $output_type:ty) => { + pub fn $fn_name( + obj: &$crate::pyo3::Bound<$crate::pyo3::PyAny>, + ) -> $crate::pyo3::PyResult> { + use $crate::pyo3::prelude::*; + use $crate::pyo3::types::PyCapsule; + + let mut obj = obj.clone(); + if obj.hasattr(concat!("__", $capsule_name, "__"))? { + obj = obj.getattr(concat!("__", $capsule_name, "__"))?.call0()?; + } + let capsule = obj.cast::().map_err(|_| { + $crate::errors::py_datafusion_err(concat!( + "Invalid ", + $capsule_name, + ". Does not contain PyCapsule object." + )) + })?; + $crate::validate_pycapsule(&capsule, $capsule_name)?; + + let expected_name = std::ffi::CString::new($capsule_name) + .expect("capsule name must not contain interior NUL bytes"); + let data: std::ptr::NonNull<$ffi_type> = capsule + .pointer_checked(Some(expected_name.as_c_str()))? + .cast(); + let output_obj = unsafe { data.as_ref() }; + let output_obj: std::sync::Arc<$output_type> = output_obj.into(); + + Ok(output_obj) + } + }; +} + +/// Same shape as [`from_pycapsule!`] but for FFI types whose conversion +/// into `Arc<$output_type>` is fallible (uses `TryFrom`). +#[macro_export] +macro_rules! try_from_pycapsule { + ($fn_name:ident, $capsule_name:literal, $ffi_type:ty, $output_type:ty) => { + pub fn $fn_name( + obj: &$crate::pyo3::Bound<$crate::pyo3::PyAny>, + ) -> $crate::pyo3::PyResult> { + use $crate::pyo3::prelude::*; + use $crate::pyo3::types::PyCapsule; + + let mut obj = obj.clone(); + if obj.hasattr(concat!("__", $capsule_name, "__"))? { + obj = obj.getattr(concat!("__", $capsule_name, "__"))?.call0()?; + } + let capsule = obj.cast::().map_err(|_| { + $crate::errors::py_datafusion_err(concat!( + "Invalid ", + $capsule_name, + ". Does not contain PyCapsule object." + )) + })?; + $crate::validate_pycapsule(&capsule, $capsule_name)?; + + let expected_name = std::ffi::CString::new($capsule_name) + .expect("capsule name must not contain interior NUL bytes"); + let data: std::ptr::NonNull<$ffi_type> = capsule + .pointer_checked(Some(expected_name.as_c_str()))? + .cast(); + let output_obj = unsafe { data.as_ref() }; + let output_obj: std::sync::Arc<$output_type> = output_obj + .try_into() + .map_err($crate::errors::py_datafusion_err)?; + + Ok(output_obj) + } + }; } + +// Re-export pyo3 so the macros expand inside downstream crates without +// requiring an explicit pyo3 dep at the call site. +#[doc(hidden)] +pub use pyo3; + +from_pycapsule!( + physical_codec_from_pycapsule, + "datafusion_physical_extension_codec", + FFI_PhysicalExtensionCodec, + dyn PhysicalExtensionCodec +); + +from_pycapsule!( + physical_optimizer_rule_from_pycapsule, + "datafusion_physical_optimizer_rule", + FFI_PhysicalOptimizerRule, + dyn PhysicalOptimizerRule + Send + Sync +); + +try_from_pycapsule!( + task_context_from_pycapsule, + "datafusion_task_context_provider", + FFI_TaskContextProvider, + TaskContext +); diff --git a/dev/changelog/53.0.0.md b/dev/changelog/53.0.0.md new file mode 100644 index 000000000..3e27a852d --- /dev/null +++ b/dev/changelog/53.0.0.md @@ -0,0 +1,107 @@ + + +# Apache DataFusion Python 53.0.0 Changelog + +This release consists of 52 commits from 9 contributors. See credits at the end of this changelog for more information. + +**Breaking changes:** + +- minor: remove deprecated interfaces [#1481](https://github.com/apache/datafusion-python/pull/1481) (timsaucer) + +**Implemented enhancements:** + +- feat: feat: add to_time, to_local_time, to_date functions [#1387](https://github.com/apache/datafusion-python/pull/1387) (mesejo) +- feat: Add FFI_TableProviderFactory support [#1396](https://github.com/apache/datafusion-python/pull/1396) (davisp) + +**Fixed bugs:** + +- fix: satisfy rustfmt check in lib.rs re-exports [#1406](https://github.com/apache/datafusion-python/pull/1406) (kevinjqliu) + +**Documentation updates:** + +- docs: clarify DataFusion 52 FFI session-parameter requirement for provider hooks [#1439](https://github.com/apache/datafusion-python/pull/1439) (kevinjqliu) + +**Other:** + +- Merge release 52.0.0 into main [#1389](https://github.com/apache/datafusion-python/pull/1389) (timsaucer) +- Add workflow to verify release candidate on multiple systems [#1388](https://github.com/apache/datafusion-python/pull/1388) (timsaucer) +- Allow running "verify release candidate" github workflow on Windows [#1392](https://github.com/apache/datafusion-python/pull/1392) (kevinjqliu) +- ci: update pre-commit hooks, fix linting, and refresh dependencies [#1385](https://github.com/apache/datafusion-python/pull/1385) (dariocurr) +- Add CI check for crates.io patches [#1407](https://github.com/apache/datafusion-python/pull/1407) (timsaucer) +- Enable doc tests in local and CI testing [#1409](https://github.com/apache/datafusion-python/pull/1409) (ntjohnson1) +- Upgrade to DataFusion 53 [#1402](https://github.com/apache/datafusion-python/pull/1402) (nuno-faria) +- Catch warnings in FFI unit tests [#1410](https://github.com/apache/datafusion-python/pull/1410) (timsaucer) +- Add docstring examples for Scalar trigonometric functions [#1411](https://github.com/apache/datafusion-python/pull/1411) (ntjohnson1) +- Create workspace with core and util crates [#1414](https://github.com/apache/datafusion-python/pull/1414) (timsaucer) +- Add docstring examples for Scalar regex, crypto, struct and other [#1422](https://github.com/apache/datafusion-python/pull/1422) (ntjohnson1) +- Add docstring examples for Scalar math functions [#1421](https://github.com/apache/datafusion-python/pull/1421) (ntjohnson1) +- Add docstring examples for Common utility functions [#1419](https://github.com/apache/datafusion-python/pull/1419) (ntjohnson1) +- Add docstring examples for Aggregate basic and bitwise/boolean functions [#1416](https://github.com/apache/datafusion-python/pull/1416) (ntjohnson1) +- Fix CI errors on main [#1432](https://github.com/apache/datafusion-python/pull/1432) (timsaucer) +- Add docstring examples for Scalar temporal functions [#1424](https://github.com/apache/datafusion-python/pull/1424) (ntjohnson1) +- Add docstring examples for Aggregate statistical and regression functions [#1417](https://github.com/apache/datafusion-python/pull/1417) (ntjohnson1) +- Add docstring examples for Scalar array/list functions [#1420](https://github.com/apache/datafusion-python/pull/1420) (ntjohnson1) +- Add docstring examples for Scalar string functions [#1423](https://github.com/apache/datafusion-python/pull/1423) (ntjohnson1) +- Add docstring examples for Aggregate window functions [#1418](https://github.com/apache/datafusion-python/pull/1418) (ntjohnson1) +- ci: pin third-party actions to Apache-approved SHAs [#1438](https://github.com/apache/datafusion-python/pull/1438) (kevinjqliu) +- minor: bump datafusion to release version [#1441](https://github.com/apache/datafusion-python/pull/1441) (timsaucer) +- ci: add swap during build, use tpchgen-cli [#1443](https://github.com/apache/datafusion-python/pull/1443) (timsaucer) +- Update remaining existing examples to make testable/standalone executable [#1437](https://github.com/apache/datafusion-python/pull/1437) (ntjohnson1) +- Do not run validate_pycapsule if pointer_checked is used [#1426](https://github.com/apache/datafusion-python/pull/1426) (Tpt) +- Implement configuration extension support [#1391](https://github.com/apache/datafusion-python/pull/1391) (timsaucer) +- Add a working, more complete example of using a catalog (docs) [#1427](https://github.com/apache/datafusion-python/pull/1427) (toppyy) +- chore: update dependencies [#1447](https://github.com/apache/datafusion-python/pull/1447) (timsaucer) +- Complete doc string examples for functions.py [#1435](https://github.com/apache/datafusion-python/pull/1435) (ntjohnson1) +- chore: enforce uv lockfile consistency in CI and pre-commit [#1398](https://github.com/apache/datafusion-python/pull/1398) (mesejo) +- CI: Add CodeQL workflow for GitHub Actions security scanning [#1408](https://github.com/apache/datafusion-python/pull/1408) (kevinjqliu) +- ci: update codespell paths [#1469](https://github.com/apache/datafusion-python/pull/1469) (timsaucer) +- Add missing datetime functions [#1467](https://github.com/apache/datafusion-python/pull/1467) (timsaucer) +- Add AI skill to check current repository against upstream APIs [#1460](https://github.com/apache/datafusion-python/pull/1460) (timsaucer) +- Add missing string function `contains` [#1465](https://github.com/apache/datafusion-python/pull/1465) (timsaucer) +- Add missing conditional functions [#1464](https://github.com/apache/datafusion-python/pull/1464) (timsaucer) +- Reduce peak memory usage during release builds to fix OOM on manylinux runners [#1445](https://github.com/apache/datafusion-python/pull/1445) (kevinjqliu) +- Add missing map functions [#1461](https://github.com/apache/datafusion-python/pull/1461) (timsaucer) +- minor: Fix pytest instructions in the README [#1477](https://github.com/apache/datafusion-python/pull/1477) (nuno-faria) +- Add missing array functions [#1468](https://github.com/apache/datafusion-python/pull/1468) (timsaucer) +- Add missing scalar functions [#1470](https://github.com/apache/datafusion-python/pull/1470) (timsaucer) +- Add missing aggregate functions [#1471](https://github.com/apache/datafusion-python/pull/1471) (timsaucer) +- Add missing Dataframe functions [#1472](https://github.com/apache/datafusion-python/pull/1472) (timsaucer) +- Add missing deregister methods to SessionContext [#1473](https://github.com/apache/datafusion-python/pull/1473) (timsaucer) +- Add missing registration methods [#1474](https://github.com/apache/datafusion-python/pull/1474) (timsaucer) +- Add missing SessionContext utility methods [#1475](https://github.com/apache/datafusion-python/pull/1475) (timsaucer) + +## Credits + +Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. + +``` + 25 Tim Saucer + 13 Nick + 6 Kevin Liu + 2 Daniel Mesejo + 2 Nuno Faria + 1 Paul J. Davis + 1 Thomas Tanon + 1 Topias Pyykkönen + 1 dario curreri +``` + +Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. + diff --git a/dev/changelog/54.0.0.md b/dev/changelog/54.0.0.md new file mode 100644 index 000000000..e73118d8b --- /dev/null +++ b/dev/changelog/54.0.0.md @@ -0,0 +1,106 @@ + + +# Apache DataFusion Python 54.0.0 Changelog + +This release consists of 51 commits from 8 contributors. See credits at the end of this changelog for more information. + +**Breaking changes:** + +- Update datafusion dependency to latest in preparation for DF54 [#1532](https://github.com/apache/datafusion-python/pull/1532) (timsaucer) +- feat: enable pickling for Python aggregate and window UDFs [#1545](https://github.com/apache/datafusion-python/pull/1545) (timsaucer) +- feat: pass calling SessionContext to Python UDTF callbacks [#1555](https://github.com/apache/datafusion-python/pull/1555) (timsaucer) +- feat: accept distinct kwarg on sum and avg [#1556](https://github.com/apache/datafusion-python/pull/1556) (timsaucer) + +**Implemented enhancements:** + +- feat: add AI skill to find and improve the Pythonic interface to functions [#1484](https://github.com/apache/datafusion-python/pull/1484) (timsaucer) +- feat: enable pickling of most Expr except udaf and udwf [#1544](https://github.com/apache/datafusion-python/pull/1544) (timsaucer) +- feat: expose variety of features from DF54 update [#1554](https://github.com/apache/datafusion-python/pull/1554) (timsaucer) +- feat: Python UDFs: per-session inlining toggle and strict refusal setting [#1546](https://github.com/apache/datafusion-python/pull/1546) (timsaucer) +- feat: create free-threaded python wheels [#1553](https://github.com/apache/datafusion-python/pull/1553) (timsaucer) +- feat: expose lambda and higher-order array functions [#1561](https://github.com/apache/datafusion-python/pull/1561) (timsaucer) +- feat: import user-defined physical optimizer rules over FFI [#1557](https://github.com/apache/datafusion-python/pull/1557) (timsaucer) +- feat: expose SessionContext.copied_config and parse_capacity_limit [#1570](https://github.com/apache/datafusion-python/pull/1570) (timsaucer) +- feat: expose array_compact, array_normalize, cosine_distance, inner_product [#1567](https://github.com/apache/datafusion-python/pull/1567) (timsaucer) +- feat: expose arrow_field, arrow_try_cast, cast_to_type, with_metadata [#1568](https://github.com/apache/datafusion-python/pull/1568) (timsaucer) +- feat: expose spark-compatible functions [#1564](https://github.com/apache/datafusion-python/pull/1564) (timsaucer) +- feat: improve pythonic interface on date/time functions [#1563](https://github.com/apache/datafusion-python/pull/1563) (timsaucer) + +**Fixed bugs:** + +- fix: type scalar UDF returns as Arrow arrays [#1528](https://github.com/apache/datafusion-python/pull/1528) (BharatDeva) +- fix: Skip `fork` and `forkserver` on `win32` [#1566](https://github.com/apache/datafusion-python/pull/1566) (nuno-faria) + +**Documentation updates:** + +- docs: enrich module docstrings and add doctest examples [#1498](https://github.com/apache/datafusion-python/pull/1498) (timsaucer) +- docs: add README section for AI coding assistants [#1503](https://github.com/apache/datafusion-python/pull/1503) (timsaucer) +- docs: add upstream sync process documentation [#1524](https://github.com/apache/datafusion-python/pull/1524) (timsaucer) +- docs: document null-handling function arguments [#1527](https://github.com/apache/datafusion-python/pull/1527) (BharatDeva) +- docs: user guide + runnable examples for distributing expressions [#1547](https://github.com/apache/datafusion-python/pull/1547) (timsaucer) +- docs: convert reStructuredText sources to MyST markdown [#1579](https://github.com/apache/datafusion-python/pull/1579) (timsaucer) + +**Other:** + +- Release 53.0.0 [#1491](https://github.com/apache/datafusion-python/pull/1491) (timsaucer) +- ci: disable symbol export on Windows verification [#1486](https://github.com/apache/datafusion-python/pull/1486) (timsaucer) +- Add Python bindings for accessing ExecutionMetrics [#1381](https://github.com/apache/datafusion-python/pull/1381) (ShreyeshArangath) +- Support None comparisons for null expressions [#1489](https://github.com/apache/datafusion-python/pull/1489) (zeel2104) +- chore: update release documentation [#1494](https://github.com/apache/datafusion-python/pull/1494) (timsaucer) +- Fix error on show() with an explain plan [#1492](https://github.com/apache/datafusion-python/pull/1492) (timsaucer) +- Add SKILL.md and enrich package docstring [#1497](https://github.com/apache/datafusion-python/pull/1497) (timsaucer) +- minor: fix header on agent SKILL.md file [#1501](https://github.com/apache/datafusion-python/pull/1501) (timsaucer) +- tpch examples: rewrite queries idiomatically and embed reference SQL [#1504](https://github.com/apache/datafusion-python/pull/1504) (timsaucer) +- Move public skills to a directory to avoid downloading the whole repo [#1519](https://github.com/apache/datafusion-python/pull/1519) (ntjohnson1) +- Update user documentation for AI agent skill usage [#1505](https://github.com/apache/datafusion-python/pull/1505) (timsaucer) +- build(deps): combined dependabot bumps (Cargo + workflows) [#1534](https://github.com/apache/datafusion-python/pull/1534) (timsaucer) +- Add support for logical and physical codecs [#1541](https://github.com/apache/datafusion-python/pull/1541) (timsaucer) +- Add details on caching to skill [#1521](https://github.com/apache/datafusion-python/pull/1521) (ntjohnson1) +- Bump DataFusion to prepare for DF54 release candidate [#1562](https://github.com/apache/datafusion-python/pull/1562) (timsaucer) +- Export `to_datafusion_err` from the util crate root [#1487](https://github.com/apache/datafusion-python/pull/1487) (kosiew) +- chore: remove unused PyConfig [#1485](https://github.com/apache/datafusion-python/pull/1485) (timsaucer) +- refactor(context): deduplicate register/read option-building logic [#1479](https://github.com/apache/datafusion-python/pull/1479) (mesejo) +- Improve documentation site layout [#1578](https://github.com/apache/datafusion-python/pull/1578) (timsaucer) +- Allow `DataFrame.aggregate` to accept `None` for no grouping [#1581](https://github.com/apache/datafusion-python/pull/1581) (kosiew) +- Update to released DF 54.0.0 [#1588](https://github.com/apache/datafusion-python/pull/1588) (timsaucer) +- build(deps): batch dependabot dependency updates [#1589](https://github.com/apache/datafusion-python/pull/1589) (timsaucer) +- Deprecate `Expr` temporal part arguments in date extraction and truncation functions [#1587](https://github.com/apache/datafusion-python/pull/1587) (kosiew) +- chore: update rust dependencies [#1604](https://github.com/apache/datafusion-python/pull/1604) (timsaucer) +- Add deprecation warnings for Expr passed to confirmed literal-only function arguments [#1605](https://github.com/apache/datafusion-python/pull/1605) (kosiew) +- chore: resolve audits after DF54.0.0 update to skill [#1608](https://github.com/apache/datafusion-python/pull/1608) (timsaucer) +- chore: remove 3.13 freethreaded builds [#1609](https://github.com/apache/datafusion-python/pull/1609) (timsaucer) + +## Credits + +Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. + +``` + 39 Tim Saucer + 4 kosiew + 2 BharatDeva + 2 Nick + 1 Daniel Mesejo + 1 Nuno Faria + 1 Shreyesh + 1 Zeel Desai +``` + +Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. + diff --git a/dev/release/README.md b/dev/release/README.md index ed28f4aa6..384ef9210 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -26,13 +26,19 @@ required due to changes in DataFusion rather than having a large amount of work is available. When there is a new official release of DataFusion, we update the `main` branch to point to that, update the version -number, and create a new release branch, such as `branch-0.8`. Once this branch is created, we switch the `main` branch +number, and create a new release branch, such as `branch-53`. Once this branch is created, we switch the `main` branch back to using GitHub dependencies. The release activity (such as generating the changelog) can then happen on the release branch without blocking ongoing development in the `main` branch. -We can cherry-pick commits from the `main` branch into `branch-0.8` as needed and then create new patch releases +We can cherry-pick commits from the `main` branch into `branch-53` as needed and then create new patch releases from that branch. +## Upstream Sync + +Between releases the `main` branch is periodically synced to a newer upstream `apache/datafusion` version. This is +broken into a three-PR workflow (bump + fix breakage, consolidate transitive deps, fill API and documentation gaps). +See [`upstream-sync.md`](upstream-sync.md) for the full process. + ## Detailed Guide ### Pre-requisites @@ -53,8 +59,12 @@ You will also need access to the [datafusion](https://test.pypi.org/project/data Before creating a new release: - We need to ensure that the main branch does not have any GitHub dependencies +- Confirm the upstream sync workflow in [`upstream-sync.md`](upstream-sync.md) has been completed for this release cycle + (crate bump + breakage fixes, transitive dependency consolidation, and the `/check-upstream` and `/audit-skill-md` + passes). Any gaps surfaced by those skills should land before the release branch is cut. - a PR should be created and merged to update the major version number of the project -- A new release branch should be created, such as `branch-0.8` +- A new release branch should be created, such as `branch-53` +- It is best to push this branch to the apache repository rather than a personal fork in case patch releases are required. ## Preparing a Release Candidate @@ -65,14 +75,15 @@ We maintain a `CHANGELOG.md` so our users know what has been changed between rel The changelog is generated using a Python script: ```bash -$ GITHUB_TOKEN= ./dev/release/generate-changelog.py 24.0.0 HEAD 25.0.0 > dev/changelog/25.0.0.md +$ uv sync --group release +$ GITHUB_TOKEN= ./dev/release/generate-changelog.py 52.0.0 HEAD 53.0.0 > dev/changelog/53.0.0.md ``` This script creates a changelog from GitHub PRs based on the labels associated with them as well as looking for titles starting with `feat:`, `fix:`, or `docs:` . The script will produce output similar to: ``` -Fetching list of commits between 24.0.0 and HEAD +Fetching list of commits between 52.0.0 and HEAD Fetching pull requests Categorizing pull requests Generating changelog content @@ -81,6 +92,7 @@ Generating changelog content ### Update the version number The only place you should need to update the version is in the root `Cargo.toml`. +You will need to update this both in the workspace section and also in the dependencies. After updating the toml file, run `cargo update` to update the cargo lock file. If you do not want to update all the dependencies, you can instead run `cargo build` which should only update the version number for `datafusion-python`. @@ -94,14 +106,14 @@ you need to push a tag to start the CI process for release candidates. The follo the upstream repository is called `apache`. ```bash -git tag 0.8.0-rc1 -git push apache 0.8.0-rc1 +git tag 53.0.0-rc1 +git push apache 53.0.0-rc1 ``` ### Create a source release ```bash -./dev/release/create-tarball.sh 0.8.0 1 +./dev/release/create-tarball.sh 53.0.0 1 ``` This will also create the email template to send to the mailing list. @@ -124,10 +136,10 @@ Click on the action and scroll down to the bottom of the page titled "Artifacts" contain files such as: ```text -datafusion-22.0.0-cp37-abi3-macosx_10_7_x86_64.whl -datafusion-22.0.0-cp37-abi3-macosx_11_0_arm64.whl -datafusion-22.0.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl -datafusion-22.0.0-cp37-abi3-win_amd64.whl +datafusion-53.0.0-cp37-abi3-macosx_10_7_x86_64.whl +datafusion-53.0.0-cp37-abi3-macosx_11_0_arm64.whl +datafusion-53.0.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl +datafusion-53.0.0-cp37-abi3-win_amd64.whl ``` Upload the wheels to testpypi. @@ -135,23 +147,23 @@ Upload the wheels to testpypi. ```bash unzip dist.zip python3 -m pip install --upgrade setuptools twine build -python3 -m twine upload --repository testpypi datafusion-22.0.0-cp37-abi3-*.whl +python3 -m twine upload --repository testpypi datafusion-53.0.0-cp37-abi3-*.whl ``` When prompted for username, enter `__token__`. When prompted for a password, enter a valid GitHub Personal Access Token #### Publish Python Source Distribution to testpypi -Download the source tarball created in the previous step, untar it, and run: +Download the source tarball from the Apache server created in the previous step, untar it, and run: ```bash maturin sdist ``` -This will create a file named `dist/datafusion-0.7.0.tar.gz`. Upload this to testpypi: +This will create a file named `dist/datafusion-53.0.0.tar.gz`. Upload this to testpypi: ```bash -python3 -m twine upload --repository testpypi dist/datafusion-0.7.0.tar.gz +python3 -m twine upload --repository testpypi dist/datafusion-53.0.0.tar.gz ``` ### Run Verify Release Candidate Workflow @@ -162,8 +174,8 @@ Before sending the vote email, run the manually triggered GitHub Actions workflo 1. Go to https://github.com/apache/datafusion-python/actions/workflows/verify-release-candidate.yml 2. Click "Run workflow" -3. Set `version` to the release version (for example, `52.0.0`) -4. Set `rc_number` to the RC number (for example, `0`) +3. Set `version` to the release version (for example, `53.0.0`) +4. Set `rc_number` to the RC number (for example, `1`) 5. Wait for all jobs to complete successfully Include a short note in the vote email template that this workflow was run across all OS/architecture @@ -183,7 +195,7 @@ Releases may be verified using `verify-release-candidate.sh`: ```bash git clone https://github.com/apache/datafusion-python.git -dev/release/verify-release-candidate.sh 48.0.0 1 +dev/release/verify-release-candidate.sh 53.0.0 1 ``` Alternatively, one can run unit tests against a testpypi release candidate: @@ -195,7 +207,7 @@ cd datafusion-python # checkout the release commit git fetch --tags -git checkout 40.0.0-rc1 +git checkout 53.0.0-rc1 git submodule update --init --recursive # create the env @@ -203,7 +215,7 @@ python3 -m venv .venv source .venv/bin/activate # install release candidate -pip install --extra-index-url https://test.pypi.org/simple/ datafusion==40.0.0 +pip install --extra-index-url https://test.pypi.org/simple/ datafusion==53.0.0 # install test dependencies pip install pytest numpy pytest-asyncio @@ -224,7 +236,7 @@ Once the vote passes, we can publish the release. Create the source release tarball: ```bash -./dev/release/release-tarball.sh 0.8.0 1 +./dev/release/release-tarball.sh 53.0.0 1 ``` ### Publishing Rust Crate to crates.io @@ -232,7 +244,7 @@ Create the source release tarball: Some projects depend on the Rust crate directly, so we publish this to crates.io ```shell -cargo publish +cargo publish --workspace ``` ### Publishing Python Artifacts to PyPi @@ -252,15 +264,15 @@ Pypi packages auto upload to conda-forge via [datafusion feedstock](https://gith ### Push the Release Tag ```bash -git checkout 0.8.0-rc1 -git tag 0.8.0 -git push apache 0.8.0 +git checkout 53.0.0-rc1 +git tag 53.0.0 +git push apache 53.0.0 ``` ### Add the release to Apache Reporter Add the release to https://reporter.apache.org/addrelease.html?datafusion with a version name prefixed with `DATAFUSION-PYTHON`, -for example `DATAFUSION-PYTHON-31.0.0`. +for example `DATAFUSION-PYTHON-53.0.0`. The release information is used to generate a template for a board report (see example from Apache Arrow [here](https://github.com/apache/arrow/pull/14357)). @@ -283,7 +295,7 @@ svn ls https://dist.apache.org/repos/dist/dev/datafusion | grep datafusion-pytho Delete a release candidate: ```bash -svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-python-7.1.0-rc1/ +svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-python-53.0.0-rc1/ ``` #### Deleting old releases from `release` svn @@ -299,5 +311,5 @@ svn ls https://dist.apache.org/repos/dist/release/datafusion | grep datafusion-p Delete a release: ```bash -svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-python-7.0.0 +svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-python-52.0.0 ``` diff --git a/dev/release/rat_exclude_files.txt b/dev/release/rat_exclude_files.txt index dcd5d9aac..70f60168f 100644 --- a/dev/release/rat_exclude_files.txt +++ b/dev/release/rat_exclude_files.txt @@ -47,4 +47,9 @@ benchmarks/tpch/queries/q*.sql benchmarks/tpch/create_tables.sql .cargo/config.toml **/.cargo/config.toml -uv.lock \ No newline at end of file +uv.lock +examples/tpch/answers_sf1/*.tbl +**/SKILL.md +docs/source/llms.txt +CLAUDE.md +.claude/skills \ No newline at end of file diff --git a/dev/release/upstream-sync.md b/dev/release/upstream-sync.md new file mode 100644 index 000000000..dc05b4195 --- /dev/null +++ b/dev/release/upstream-sync.md @@ -0,0 +1,170 @@ + + +# Upstream Sync Process + +This document describes how to sync `datafusion-python` to a new version of the +upstream `apache/datafusion` Rust crates. This is a recurring task: between +official releases the `main` branch tracks DataFusion via crates.io or GitHub +dependencies, and we periodically bump those dependencies to pick up new +features and bug fixes. + +The work is broken into **three sequential PRs** rather than landing as one +large change. Splitting reviews along these lines keeps each PR focused, makes +breakage easier to bisect, and lets reviewers concentrate on one concern at a +time. + +## PR 1: Bump DataFusion crate dependencies and fix breakage + +**Goal:** update the upstream `datafusion` crate version and make the project +build, test, and lint cleanly against it. + +1. In the root `Cargo.toml`, update: + - `[workspace.package].version` to the new major (the `datafusion-python` + major tracks the upstream `datafusion` major, so a 53→54 bump moves + this from `53.0.0` to `54.0.0`), and + - every `datafusion` / `datafusion-*` entry in `[workspace.dependencies]` + to the same new major. + + Per-crate manifests under `crates/` inherit these pins via + `workspace = true` and need no edit. +2. Update `Cargo.lock` for the datafusion family only — leave unrelated + transitives at their current pins so PR 2 can address them deliberately. + List every `datafusion-*` workspace dependency with `-p`: + + ```bash + cargo update \ + -p datafusion \ + -p datafusion-substrait \ + -p datafusion-proto \ + -p datafusion-ffi \ + -p datafusion-catalog \ + -p datafusion-common \ + -p datafusion-functions-aggregate \ + -p datafusion-functions-window \ + -p datafusion-expr + ``` + + Or pin exact versions with `--precise`, one crate at a time: + + ```bash + cargo update -p datafusion --precise 54.0.0 + # repeat for each datafusion-* sibling + ``` + + A bare `cargo update` would refresh every transitive crate and blur the + diff between PR 1 and PR 2. +3. Run the standard build and test commands and address compilation errors, + API renames, signature changes, and behavior changes: + - `cargo build` + - `cargo test` + - `pytest` + - `pre-commit run --all-files` +4. Fix only what's needed to restore green CI. Resist the urge to bundle + unrelated cleanups — those belong in their own PR. +5. If a breaking change in upstream requires a user-facing API change in + `datafusion-python`, add the `api change` label and document the change + in the PR description so it surfaces in the changelog. + +**Reference PRs:** [#1311](https://github.com/apache/datafusion-python/pull/1311) +(DF51), [#1337](https://github.com/apache/datafusion-python/pull/1337) (DF52). + +## PR 2: Consolidate transitive dependencies + +**Goal:** after the upstream bump, the dependency tree may have multiple +versions of the same transitive crate (for example, two `arrow` versions, two +`object_store` versions). Reconcile these so we ship a single coherent set. + +1. Inspect the lockfile for duplicates: + ```bash + cargo tree --duplicates + ``` +2. For each duplicate that matters (Arrow, `object_store`, `parquet`, + `tokio`, `arrow-flight`, etc.), update our direct dependency declarations + in `Cargo.toml` to versions compatible with what upstream DataFusion now + pulls in. The goal is one version of each ecosystem-critical crate. +3. Re-run `cargo update` and re-run the full test matrix. Some duplicates are + benign (small leaf crates with no FFI surface) and can be left alone if + reconciliation would force a much larger change. Use judgment. +4. If consolidating forces a behavioral change visible to users (for example, + a newer `pyarrow`-compatible Arrow version), call it out in the PR + description. + +Keeping this work separate from PR 1 means PR 1 stays a "make it compile" +review and PR 2 stays a "tidy the dependency graph" review. + +## PR 3: Fill API and documentation gaps + +**Goal:** with the upstream version locked in, identify new APIs that landed +upstream and decide whether to expose them, and update agent-facing +documentation so it still matches the surface we ship. + +1. Run the `check-upstream` skill (`.ai/skills/check-upstream/SKILL.md`) to + diff the upstream Rust API against what's exposed in + `python/datafusion/`. The skill covers scalar/aggregate/window/table + functions, `DataFrame` methods, `SessionContext` methods, and FFI types. + Invoke it from the assistant with `/check-upstream` (optionally scoped to + one area, e.g. `/check-upstream scalar functions`). +2. For each gap, decide whether to: + - Expose it now (small, obvious additions can land in this PR). + - File a tracking issue (anything non-trivial — separate PR per feature + keeps reviews focused). + - Skip it (internal-only or already covered by an existing API; record + the decision in the "Evaluated and not requiring exposure" sections of + the skill so future runs don't re-flag it). +3. (Optional) Run the `make-pythonic` skill + (`.ai/skills/make-pythonic/SKILL.md`) over any newly exposed APIs to + align signatures with the project's Pythonic style (accepting plain + strings for column names, raw Python values where auto-wrapping + applies, etc.). Invoke it from the assistant with `/make-pythonic`. + Running this *before* the audit step means examples in `SKILL.md` get + updated to the final signature in one pass instead of churning twice. + Larger reshapes still belong in their own PR. +4. Run the `audit-skill-md` skill (`.ai/skills/audit-skill-md/SKILL.md`) to + cross-reference the user-facing skill at + [`skills/datafusion_python/SKILL.md`](../../skills/datafusion_python/SKILL.md) + against the current public API. The skill flags stale function names, + missing newly exposed APIs, examples that drifted from idiomatic style, + and missing version notes. Invoke it from the assistant with + `/audit-skill-md` (optionally scoped, e.g. `/audit-skill-md dataframe`). + Apply the resulting edits to `SKILL.md` and to the relevant RST pages + under `docs/source/user-guide/common-operations/`. +5. If new aggregate or window functions were exposed in step 2, also update: + - `docs/source/user-guide/common-operations/aggregations.rst` + - `docs/source/user-guide/common-operations/windows.rst` + +## Why three PRs + +- **Bisectable.** If a regression appears, `git bisect` lands on the + responsible PR (compile fix, dependency consolidation, or API addition) + rather than a single mega-commit. +- **Reviewable.** Each PR has a single concern. Reviewers reading PR 1 don't + need to also reason about whether new APIs are well-named. +- **Skippable.** Some upstream syncs are pure version bumps with no new APIs + worth exposing. PR 3 can be empty or merged as a no-op if the audit comes + back clean. + +## Related documents + +- [`README.md`](README.md) — the broader release process (this sync work + feeds into the next official release). +- [`.ai/skills/check-upstream/SKILL.md`](../../.ai/skills/check-upstream/SKILL.md) + — API coverage audit. +- [`skills/datafusion_python/SKILL.md`](../../skills/datafusion_python/SKILL.md) + — user-facing agent guide kept in sync via PR 3. diff --git a/dev/release/verify-release-candidate.sh b/dev/release/verify-release-candidate.sh index 9591e0335..42e3970fb 100755 --- a/dev/release/verify-release-candidate.sh +++ b/dev/release/verify-release-candidate.sh @@ -132,7 +132,7 @@ test_source_distribution() { # Clone testing repositories into the expected location git clone https://github.com/apache/arrow-testing.git testing - git clone https://github.com/apache/parquet-testing.git parquet-testing + git clone https://github.com/apache/parquet-testing.git parquet python3 -m venv .venv if [ -x ".venv/bin/python" ]; then @@ -153,7 +153,7 @@ test_source_distribution() { #TODO: we should really run tests here as well #python3 -m pytest - if ( find -iname 'Cargo.toml' | xargs grep SNAPSHOT ); then + if ( find . -iname 'Cargo.toml' | xargs grep SNAPSHOT ); then echo "Cargo.toml version should not contain SNAPSHOT for releases" exit 1 fi diff --git a/docs/build.sh b/docs/build.sh index f73330323..e8409ee72 100755 --- a/docs/build.sh +++ b/docs/build.sh @@ -36,6 +36,21 @@ rm -rf build 2> /dev/null rm -rf temp 2> /dev/null mkdir temp cp -rf source/* temp/ + +# myst-nb executes each page as a notebook from the directory that page +# lives in, so the example data files must sit alongside every page that +# loads them by relative name (e.g. `ctx.read_csv("pokemon.csv")`). Symlink +# them into each directory that has such a page rather than copying the +# 20 MB parquet repeatedly. +for d in temp temp/user-guide temp/user-guide/common-operations; do + ln -sf "$script_dir/pokemon.csv" "$d/pokemon.csv" + ln -sf "$script_dir/yellow_tripdata_2021-01.parquet" "$d/yellow_tripdata_2021-01.parquet" +done + +# myst-nb runs `{code-cell}` blocks against a Jupyter kernel named "python3". +# Register the active environment's interpreter as that kernel (idempotent). +python -m ipykernel install --sys-prefix --name python3 --display-name "Python 3" + make SOURCEDIR=`pwd`/temp html cd "$original_dir" || exit diff --git a/docs/source/_static/favicon.svg b/docs/source/_static/favicon.svg new file mode 100644 index 000000000..bf174719b --- /dev/null +++ b/docs/source/_static/favicon.svg @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/docs/source/_static/images/original_dark.svg b/docs/source/_static/images/original_dark.svg new file mode 100644 index 000000000..fbdf20ea7 --- /dev/null +++ b/docs/source/_static/images/original_dark.svg @@ -0,0 +1,31 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/source/_static/theme_overrides.css b/docs/source/_static/theme_overrides.css index aaa40fba2..9eab5b03c 100644 --- a/docs/source/_static/theme_overrides.css +++ b/docs/source/_static/theme_overrides.css @@ -21,62 +21,34 @@ /* Customizing with theme CSS variables */ :root { - --pst-color-active-navigation: 215, 70, 51; --pst-color-link-hover: 215, 70, 51; --pst-color-headerlink: 215, 70, 51; - /* Use normal text color (like h3, ..) instead of primary color */ - --pst-color-h1: var(--color-text-base); - --pst-color-h2: var(--color-text-base); - /* Use softer blue from bootstrap's default info color */ + /* Softer blue from bootstrap's default info color */ --pst-color-info: 23, 162, 184; - --pst-header-height: 0px; } code { color: rgb(215, 70, 51); } -.footer { - text-align: center; -} - -/* Ensure the logo is properly displayed */ - -.navbar-brand { - height: auto; - width: auto; +html[data-theme="dark"] code { + color: rgb(255, 138, 117); } -a.navbar-brand img { - height: auto; - width: auto; - max-height: 15vh; - max-width: 100%; +.footer { + text-align: center; } -/* This is the bootstrap CSS style for "table-striped". Since the theme does -not yet provide an easy way to configure this globally, it easier to simply -include this snippet here than updating each table in all rst files to -add ":class: table-striped" */ +/* Bootstrap "table-striped" applied globally so individual tables in + user-guide pages don't need ":class: table-striped" added one by one. */ .table tbody tr:nth-of-type(odd) { background-color: rgba(0, 0, 0, 0.05); } - -/* Limit the max height of the sidebar navigation section. Because in our -custimized template, there is more content above the navigation, i.e. -larger logo: if we don't decrease the max-height, it will overlap with -the footer. -Details: min(15vh, 110px) for the logo size, 8rem for search box etc*/ - -@media (min-width:720px) { - @supports (position:-webkit-sticky) or (position:sticky) { - .bd-links { - max-height: calc(100vh - min(15vh, 110px) - 8rem) - } - } +html[data-theme="dark"] .table tbody tr:nth-of-type(odd) { + background-color: rgba(255, 255, 255, 0.05); } @@ -91,3 +63,48 @@ Details: min(15vh, 110px) for the logo size, 8rem for search box etc*/ white-space: normal !important; } } + + +/* Hideable right-hand "On this page" sidebar. + * toc-toggle.js adds the button and toggles `pst-secondary-hidden` on ; + * hiding the sidebar lets the flex article container reclaim the width. */ + +body.pst-secondary-hidden .bd-sidebar-secondary { + display: none; +} + +/* Let the article use the freed space rather than just re-centering. */ +body.pst-secondary-hidden .bd-article-container { + max-width: none; +} + +/* Floating toggle button, pinned to the top-right under the navbar. */ +#pst-secondary-toggle { + position: fixed; + top: 4.5rem; + right: 0.75rem; + z-index: 1020; + display: flex; + align-items: center; + justify-content: center; + width: 2rem; + height: 2rem; + padding: 0; + border: 1px solid var(--pst-color-border, #ccc); + border-radius: 0.25rem; + background-color: var(--pst-color-surface, #fff); + color: var(--pst-color-text-base, #333); + cursor: pointer; +} + +#pst-secondary-toggle:hover { + color: rgb(var(--pst-color-link-hover)); +} + +/* The toggle is only meaningful where the sidebar is shown (wide screens); + * below the theme's lg breakpoint the sidebar is already collapsed away. */ +@media (max-width: 959.98px) { + #pst-secondary-toggle { + display: none; + } +} diff --git a/docs/source/_static/toc-toggle.js b/docs/source/_static/toc-toggle.js new file mode 100644 index 000000000..a5cc65e54 --- /dev/null +++ b/docs/source/_static/toc-toggle.js @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/* Adds a button that hides the right-hand "On this page" sidebar so the + * article can use the full page width. The choice is remembered across + * pages via localStorage. */ +(function () { + "use strict"; + var KEY = "pst-secondary-hidden"; + + function apply(hidden, btn) { + document.body.classList.toggle("pst-secondary-hidden", hidden); + if (btn) { + btn.setAttribute("aria-pressed", String(hidden)); + btn.title = hidden ? "Show page contents" : "Hide page contents"; + } + } + + document.addEventListener("DOMContentLoaded", function () { + // Only offer the toggle on pages that actually have the sidebar. + if (!document.querySelector(".bd-sidebar-secondary")) { + return; + } + + var btn = document.createElement("button"); + btn.id = "pst-secondary-toggle"; + btn.type = "button"; + btn.setAttribute("aria-label", "Toggle page contents sidebar"); + btn.innerHTML = ''; + document.body.appendChild(btn); + + btn.addEventListener("click", function () { + var hidden = !document.body.classList.contains("pst-secondary-hidden"); + try { + localStorage.setItem(KEY, hidden ? "1" : "0"); + } catch (e) { + /* localStorage may be unavailable; toggle still works for this page. */ + } + apply(hidden, btn); + }); + + var stored = null; + try { + stored = localStorage.getItem(KEY); + } catch (e) { + /* ignore */ + } + apply(stored === "1", btn); + }); +})(); diff --git a/docs/source/_templates/docs-sidebar.html b/docs/source/_templates/docs-sidebar.html deleted file mode 100644 index 44deeed25..000000000 --- a/docs/source/_templates/docs-sidebar.html +++ /dev/null @@ -1,19 +0,0 @@ - - - - - -
- - -
- - diff --git a/docs/source/_templates/layout.html b/docs/source/_templates/layout.html index 9f7880049..d83d283c7 100644 --- a/docs/source/_templates/layout.html +++ b/docs/source/_templates/layout.html @@ -1,9 +1,5 @@ {% extends "pydata_sphinx_theme/layout.html" %} -{# Silence the navbar #} -{% block docs_navbar %} -{% endblock %} - diff --git a/docs/source/_templates/sidebar-globaltoc.html b/docs/source/_templates/sidebar-globaltoc.html new file mode 100644 index 000000000..f4aa2051f --- /dev/null +++ b/docs/source/_templates/sidebar-globaltoc.html @@ -0,0 +1,30 @@ +{# Renders the global document toctree on every page (including the + landing page) with pydata-sphinx-theme's collapsible chevrons. + + The stock sidebar-nav-bs.html starts at the current section and is + stripped from the sidebar list by suppress_sidebar_toctree() on the + root page (no parent section). Using generate_toctree_html with + startdepth=0 renders the whole tree from root with the bootstrap + classes the theme's JS uses for expand/collapse toggles. Naming the + template "sidebar-globaltoc" sidesteps the suppress filter, which + matches on "sidebar-nav-bs.html" specifically. #} + diff --git a/docs/source/conf.py b/docs/source/conf.py index 01813b032..22bace809 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -35,8 +35,8 @@ # -- Project information ----------------------------------------------------- -project = "Apache Arrow DataFusion" -copyright = "2019-2024, Apache Software Foundation" +project = "Apache DataFusion in Python" +copyright = "2019-2026, Apache Software Foundation" author = "Apache Software Foundation" @@ -48,16 +48,40 @@ extensions = [ "sphinx.ext.mathjax", "sphinx.ext.napoleon", - "myst_parser", - "IPython.sphinxext.ipython_directive", + # myst_nb is a superset of myst_parser: it provides the MyST markdown + # parser plus executable `{code-cell}` notebook directives. Do NOT also + # list "myst_parser" — myst_nb activates it internally and listing both + # raises an extension conflict. + "myst_nb", "autoapi.extension", ] +# NOTE: .rst stays alongside .md because sphinx-autoapi generates RST +# under autoapi/ and Sphinx needs the suffix to parse it. The human- +# authored docs are all MyST .md now. ".md" is routed through myst-nb so +# pages carrying jupytext/kernelspec front matter execute their +# `{code-cell}` blocks; pages without that front matter render as plain +# MyST markdown. The ".rst" entry is only for the autoapi build artifacts. source_suffix = { ".rst": "restructuredtext", - ".md": "markdown", + ".md": "myst-nb", } +# Execute notebook code cells at build time and fail the build if any cell +# raises — this replaces the old IPython sphinx directive, whose executed +# examples are now `{code-cell}` blocks. "force" re-executes every build so +# stale cached output can never ship. +nb_execution_mode = "force" +nb_execution_timeout = 120 +nb_execution_raise_on_error = True + +# Prefer the plain-text repr of a cell's last expression over its rich +# `_repr_html_`. A DataFrame's HTML repr is a self-contained widget (inline +# styles + an injected