[SQL] Pull filters above map, integrals, aggregates, filters#6600
[SQL] Pull filters above map, integrals, aggregates, filters#6600mihaibudiu wants to merge 1 commit into
Conversation
494e10d to
39cb149
Compare
mythical-fred
left a comment
There was a problem hiding this comment.
Nice optimization. Filter pull-up above aggregates, maps, integrate/differentiate, noop, and filter-merge. Read carefully; the safety guards look right:
- Fanout=1 gate correctly prevents duplicating the source when it has other consumers.
- Aggregate case uses
NoExpressionin the value slot andContainsNoExpression.searchafter reduction to reject filters that actually depend on the aggregate's output — a clean way to prove key-only dependence. - Map/MapIndex relies on
shouldInlineComposition— same threshold used elsewhere, so complexity blow-up stays bounded. - Integrate/Differentiate/Noop commute with a stateless filter on Z-sets, so pulling through is sound.
- Filter+Filter merge into
ANDand calls.after(...)on the source'sRelNodeto preserve provenance — good.
DBSPCompiler.getFinalCircuit guard if (circuit != null) is a sensible fix for the failed-compilation path.
Persistent-id churn in metadataTests-generateDFRecursive.json is expected (circuit shape changed) and lives only in a compiler test fixture — no runtime checkpoint compatibility concern.
Two nitpicks, non-blocking:
- The filter-merge branch calls
.copyAnnotations(operator)on the combined filter but drops annotations from the upstream filter (source). Probably fine for now, but if either filter carried something meaningful (e.g. hoisted-from-view markers), it's gone. Utilities.showResultSetsignature reformat is unrelated to the optimization — trivial, keep it, just noting.
LGTM.
mythical-fred
left a comment
There was a problem hiding this comment.
Re-review at tip 67e0722 (delta since 39cb149).
Delta is cleanly additive:
PullFilterVisitor: new branch handlingDBSPConstantOperatorsources. For each(row, weight)in the constant Z-set, calls the filter closure,reduces andSimplifys the result; if every element folds to aDBSPBoolLiteralnon-null value, replacesfilter(const)with a filtered constant. Bails conservatively on NULL results or anything that doesn't collapse to a bool literal. Sound — filter is pure and inputs are literals.CircuitOptimizer: addsPropagateEmptySourcesafterPullFilterVisitor(natural follow-up now that filters can produce empty constants).SetOpTests.simplifyIntersect: verifiesINTERSECTwith an unmatchable row (5, NULLagainst aNOT NULLcolumn) collapses to a constant/source/sink — good end-to-end coverage of the new path.
Nits (still non-blocking, same shape as before):
- Filter annotations aren't copied onto the new
DBSPConstantOperator. Probably fine since it's a constant, but consistent with the earlier nit. - The "Should not happen" comment for
b.value == nullis a bit optimistic — WHERE semantics treat NULL as false, so folding null-bool to drop-the-row would recover a few more cases. Conservative bail is safe, just leaves optimization on the table.
Approving.
This is not optimistic, this is our IR, and all filter predicates return non-nullable boolean values |
| @Override | ||
| public void postorder(DBSPFlatMapIndexOperator node) { | ||
| // Source is input | ||
| this.filterFound = true; |
There was a problem hiding this comment.
this just checks that a filter is found, not that it is pulled above anything? my naive question would be why not push it to queue so you can assert something about the order
There was a problem hiding this comment.
The next assert says "the filter is after the input", which means it was pulled.
| @Override | ||
| public void postorder(DBSPFlatMapIndexOperator node) { | ||
| // Source is input | ||
| this.filterFound = true; |
| // This is simplified to an empty set | ||
| @Override | ||
| public void postorder(DBSPSimpleOperator operator) { | ||
| Assert.assertTrue( |
There was a problem hiding this comment.
This indicates that this program was shrunk to a constant output, there are no other operators. The INTERSECT was completely eliminated
| super.postorder(operator); | ||
| return; | ||
| } | ||
| if (source.node().is(DBSPAggregateLinearPostprocessOperator.class) |
There was a problem hiding this comment.
maybe this should be a helper isAggregateOperator(x)
| this.map(operator, result); | ||
| return; | ||
| } | ||
| } else if (source.node().is(DBSPMapOperator.class) |
| if (simplified.is(DBSPBoolLiteral.class)) { | ||
| DBSPBoolLiteral b = simplified.to(DBSPBoolLiteral.class); | ||
| if (b.value == null) { | ||
| // Should not happen |
There was a problem hiding this comment.
do we need to print a warning if it does?
| DBSPClosureExpression filter = operator.getClosureFunction(); | ||
| DBSPConstantOperator c = source.node().to(DBSPConstantOperator.class); | ||
| DBSPExpression value = c.getFunction(); | ||
| Simplify simplify = new Simplify(this.compiler); |
| import java.util.Map; | ||
|
|
||
| /** | ||
| * Move filters up in a plan, towards sources. |
There was a problem hiding this comment.
I didn't see tests that evaluate all these cases, maybe they already exist elsewhere
da30ae6 to
f76f08b
Compare
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
mythical-fred
left a comment
There was a problem hiding this comment.
Re-review at 82714218 — APPROVE.
Rebase onto main plus a small cleanup: the inline DBSPConstantOperator branch in PullFilterVisitor.postorder(DBSPFilterOperator) has been extracted into PropagateConstants.filterConstant(compiler, constOp, filterClosure). Same semantics as before (per-row reduce + Simplify, bail if any result is non-DBSPBoolLiteral or null), just with the loop lifted into a shared helper — less duplication and easier to reason about. PullFilterVisitor now only imports what it still uses; the HashMap/DBSPZSetExpression/DBSPBoolLiteral/Simplify imports are dropped, which is nice.
No behavioural change from the version I approved at 67e0722f.
Fixes #6565
A set of optimizations, both in Calcite and in our backend, to move filters before other operators when it looks profitable.
Describe Manual Test Plan
Ran all java tests
Checklist