Skip to content

fix: scope fallback reasons to the current conversion - #7095

Draft
sam-1112 wants to merge 2 commits into
apache:mainfrom
sam-1112:fix/5499-fallback-reason-provenance
Draft

sam-1112 wants to merge 2 commits into
apache:mainfrom
sam-1112:fix/5499-fallback-reason-provenance

Conversation

@sam-1112

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #5499.

Rationale for this change

Stale fallback reasons on shared expression nodes can be lifted onto unrelated expressions and operators. These reasons can also satisfy strict fallback checking, hiding a conversion that failed without recording an explanation.

What changes are included in this PR?

  • Track fallback reasons in a conversion-scoped journal keyed by node identity.
  • Use current conversion records for expression copy-back and operator roll-up, including Spark 4.x shim rewrites.
  • Preserve legitimate literal fallback reasons and operator reasons recorded by earlier planning rules.
  • Restore the previous journal after nested conversions and exceptions, without sharing it with child threads.
  • Add regression tests for stale and copied tags, decimal rewrites, shim failures, and journal isolation.

How are these changes tested?

  • Spark 4.1: all 104 tests in CometExecRuleSuite passed.
  • Spark 4.1: all 14 targeted tests passed across CometExecRuleSuite, CometWindowExecSuite, and CometDecimalPromotionSuite.
  • Spark 3.4 with Java 17: all 11 targeted tests passed.
  • Native build, Spotless, and Scalastyle passed.

@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Track provenance of fallback reasons copied through shared expression nodes

1 participant