feat: expose SessionContext.write_csv, write_json, write_parquet#1569
Open
timsaucer wants to merge 1 commit into
Open
feat: expose SessionContext.write_csv, write_json, write_parquet#1569timsaucer wants to merge 1 commit into
timsaucer wants to merge 1 commit into
Conversation
Adds three plan-level writers on SessionContext that mirror the upstream datafusion::execution::context API. Each takes an ExecutionPlan and an output directory path; the plan is executed and its results are written one partition per file inside that directory. These complement the existing DataFrame.write_* methods, which are the right choice when callers need finer control (CSV header, Parquet compression, write options). The new SessionContext methods are the right choice when a caller already holds a physical ExecutionPlan (for example after custom physical optimizer rules or hand-built plans) and just wants the rows materialized. Related to apache#462. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #. No dedicated tracking issue; related to umbrella issue #462 (interface design / user stories).
Rationale for this change
DataFusion's
SessionContextexposeswrite_csv,write_json, andwrite_parquetmethods that take an already-builtArc<dyn ExecutionPlan>and a target path. These complement the existing per-DataFramewrite methods and are the right entry point when a caller already holds a physical plan -- for example after running custom physical optimizer rules (recently exposed via PR #1557) or after constructing a plan directly. The Python bindings did not surface them.What changes are included in this PR?
crates/core/src/context.rs: addwrite_csv,write_json, andwrite_parquetPyO3 methods onPySessionContext. Each accepts aPyExecutionPlanand a path, converts the plan toArc<dyn ExecutionPlan>, and delegates to the matching upstreamSessionContextmethod.write_parquetpassesNonefor theWriterPropertiesargument; per-partition Parquet tuning remains onDataFrame.write_parquet.python/datafusion/context.py: add Python wrappers with doctest examples that round-trip data through a temp directory. The docstrings flagDataFrame.write_*as the right entry point when callers need header control, compression, or other write options.Are there any user-facing changes?
Yes. Three new public methods on
datafusion.SessionContext:write_csv(plan, path)write_json(plan, path)write_parquet(plan, path)No breaking changes.