Skip to content

fix: hash NullType and ANSI interval values natively - #7084

Open
ump45nose wants to merge 1 commit into
apache:mainfrom
ump45nose:fix/hash-null-and-ansi-intervals
Open

ump45nose wants to merge 1 commit into
apache:mainfrom
ump45nose:fix/hash-null-and-ansi-intervals

Conversation

@ump45nose

Copy link
Copy Markdown

Which issue does this PR close?

Closes #7063.

Rationale for this change

The native Murmur3 and xxHash64 kernels had no arm for Null, Interval(YearMonth) or Duration(Microsecond), so hash(a, null), xxhash64(a, null) and native round-robin shuffle over those types failed with Unsupported data type in hasher.

What changes are included in this PR?

  • create_hashes_internal! hashes Interval(YearMonth) as an int and Duration(Microsecond) as a long, as Spark does for YearMonthIntervalType and DayTimeIntervalType, and leaves the hash unchanged for Null. This covers both the hash expressions and round-robin partitioning, since they share the kernel.
  • child_with_parent_nulls now returns a Null child as is. Without that, hashing a null struct that has a NullType field still panicked once the Null arm was in place, because a NullArray can't carry a null buffer (NullArray data should not contain a null buffer).

I left hash.scala alone, so hash/xxhash64 over the ANSI interval types still fall back to Spark. Enabling them there could be a follow-up now that the kernel handles them.

How are these changes tested?

New unit tests in murmur3.rs and xxhash64.rs: both interval types hash to the same values as the existing Int32/Int64 cases, and a Null column or a NullType struct field (including under a null struct) leaves the hash unchanged. A unit test in struct_nulls.rs covers the Null child. I ran the native tests for the two changed crates; I did not run the Spark/JVM suites.

AI disclosure: the change, the tests and this description were written with an AI coding assistant.

@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Oct 17, 2026

@rich7420 rich7420 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add expect_native tests for hash/xxhash64 with NULL and nullable structs containing a NullType field? Also cover round-robin with NULL and both ANSI intervals, positional placement disabled, a native exchange assertion, and full result comparison against Spark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native hashing of NullType and ANSI interval values fails with "Unsupported data type in hasher"

2 participants