feat(pregel): automatically skip second join when dst columns not needed#795

Merged

james-willis merged 13 commits into

graphframes:mainfrom

james-willis:feat/auto-skip-dst-join-790

Mar 12, 2026

Commits on Mar 4, 2026

feat(pregel): automatically skip second join when dst columns not needed
james-willis
committed
fix: only analyze message expressions for dst detection, provide dst.id when skipping join
james-willis
committed
style: run scalafmt and update SparkShims docs to be implementation-agnostic
james-willis
committed
feat: skip dst join when only dst.id is referenced
james-willis
committed
refactor: address reviewer feedback on dst join optimization
james-willis
committed
test: add comprehensive tests for extractColumnReferences and dst join optimization
james-willis
committed
refactor: remove Pregel references from SparkShims comments

authored and
james-willis
committed
perf: optimize caching and partitioning when skipping dst join

authored and
james-willis
committed
fix: correct repartition syntax for conditional partitioning

authored and
james-willis
committed
fix: correct repartition syntax for conditional partitioning

authored and
james-willis
committed
style: apply scalafmt formatting to Pregel.scala

authored and
james-willis
committed

Commits on Mar 11, 2026

address Sem's PR comments
james-willis
committed

Commits on Mar 12, 2026

push triplet filtering when no dst join
james-willis
committed