Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: djanand/datafusion-python
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 5ce35f8
Choose a base ref
...
head repository: apache/datafusion-python
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 2bd80ec
Choose a head ref
  • 11 commits
  • 70 files changed
  • 8 contributors

Commits on Jul 14, 2026

  1. fix: no panic on empty partition in register_record_batches (apache#1627

    )
    
    * fix: no panic on empty partition in register_record_batches
    
    `register_record_batches` read the schema via unchecked `partitions.0[0][0]`,
    which panics when a partition contains no record batches (e.g. the batches of
    an empty pyarrow table, whose `to_batches()` returns `[]`).
    
    Take the schema from the first available batch across all partitions and return
    a clear error when there is none, mirroring the empty-table handling added for
    `from_arrow_table` in apache#613. A non-empty later partition now also works instead
    of panicking on an empty leading one.
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
    
    * refactor: address review feedback
    
    - Use find_map(|p| p.first()) instead of flatten().next() to make the
      "first batch in any partition" intent clearer (per @kosiew).
    - Extend regression test to also assert the empty outer partition list
      case ([]) raises the same ValueError.
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
    
    ---------
    
    Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
    fangchenli and claude authored Jul 14, 2026
    Configuration menu
    Copy the full SHA
    699e00e View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2026

  1. Update pinned Rust toolchain action revision (apache#1635)

    * fix: replace rust-toolchain action with rustup commands for toolchain setup
    
    * fix: pin dtolnay/rust-toolchain sha
    
    * pin dtolnay/rust-toolchain sha
    kosiew authored Jul 28, 2026
    Configuration menu
    Copy the full SHA
    6d16ecd View commit details
    Browse the repository at this point in the history
  2. Use allowlisted Rust toolchain action revision (apache#1661)

    * Use allowlisted Rust toolchain action revision
    
    AI Disclosure: This code was written in part by an AI agent.:
    
    * Set rust toolchain so developer and CI get a consistent experience
    
    * minor clippy fixes
    timsaucer authored Jul 28, 2026
    Configuration menu
    Copy the full SHA
    5cc3230 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2026

  1. Add object_store parameter to register/read file methods for thread-s…

    …afe credentials (apache#1625)
    
    * Add object_store parameter to register/read file methods
    
    Add an optional object_store parameter to register_parquet, read_parquet,
    register_csv, read_csv, register_json, read_json, register_avro, read_avro,
    register_arrow, and read_arrow methods.
    
    When provided, the store is automatically registered for the URL scheme
    and host parsed from the path, removing the need to manually call
    register_object_store separately. This enables thread-safe cloud
    credential handling without os.environ mutation.
    
    Includes unit tests (mock-based) for URL parsing logic and integration
    tests using LocalFileSystem for end-to-end validation.
    
    Closes apache#1624
    
    * Fix file:// URL handling in _register_object_store_for_path
    
    - Allow empty netloc for file:// scheme (e.g. file:///tmp/data.parquet)
    - Require netloc (host/bucket) only for non-file schemes (s3, gs, az, https)
    - Add doctest +SKIP to register_parquet example referencing nonexistent file
    - Use Path.as_uri() in end-to-end tests for portable file:// URL construction
    - Add tests for file:// acceptance and scheme-without-host rejection
    
    * Refactor tests to use pytest.mark.parametrize per reviewer feedback
    
    Replace class-based test structure with flat parametrized functions
    to match the repository's testing conventions.
    
    ---------
    
    Co-authored-by: Tim Saucer <timsaucer@gmail.com>
    qzyu999 and timsaucer authored Jul 30, 2026
    Configuration menu
    Copy the full SHA
    4906d39 View commit details
    Browse the repository at this point in the history
  2. docs: fix bootstrap instructions in contributor guide (apache#1618)

    * docs: add missing cd step in contributor guide bootstrap
    
    * docs: fix maturin develop flag typo (-uv -> --uv)
    onestn authored Jul 30, 2026
    Configuration menu
    Copy the full SHA
    3d9583c View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2026

  1. fix: tighten scalar UDF return type hints (apache#1642)

    This PR tightens the scalar UDF annotations in python/datafusion/user_defined.py:
    
    the callable return type is bounded to pa.Array
    the scalar UDF constructor accepts return_field: pa.Field
    the decorator form accepts return_field: pa.DataType | pa.Field
    BharatDeva authored Aug 3, 2026
    Configuration menu
    Copy the full SHA
    7eaef92 View commit details
    Browse the repository at this point in the history
  2. Add is_nan aliases (apache#1643)

    Adds datafusion.functions.is_nan(...) as an alias for isnan(...) and adds Expr.is_nan() as the corresponding expression method. Existing isnan() behavior is unchanged.
    
    The PR also adds coverage for both the function-level alias and the expression method.
    BharatDeva authored Aug 3, 2026
    Configuration menu
    Copy the full SHA
    4702915 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2026

  1. chore: Update to latest main on upstream DataFusion (apache#1662)

    * Update to main on the upstream datafusion to prepare for DF55 release
    
    * Deprecate singular external table location
    
    AI Disclosure: This code was written in part by an AI agent.:
    
    * Fix tests for updated DataFusion behavior
    
    AI Disclosure: This code was written in part by an AI agent.:
    
    * create table requires a location
    
    * Remove unused Python token from Substrait deserialization
    
    AI Disclosure: This code was written in part by an AI agent.:
    
    * test: cover external table locations binding
    
    AI Disclosure: This code was written in part by an AI agent.:
    timsaucer authored Aug 6, 2026
    Configuration menu
    Copy the full SHA
    cc2ec5c View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2026

  1. Configuration menu
    Copy the full SHA
    c577c0f View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2026

  1. Configuration menu
    Copy the full SHA
    38422e4 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2026

  1. chore: Bump release candidate version to 55.0.0rc3 (apache#1686)

    * Bump release candidate version
    
    * spark slice function now respects inner field name
    timsaucer authored Aug 14, 2026
    Configuration menu
    Copy the full SHA
    2bd80ec View commit details
    Browse the repository at this point in the history
Loading