Skip to content

[SQL] Pull filters above map, integrals, aggregates, filters#6600

Open
mihaibudiu wants to merge 1 commit into
feldera:mainfrom
mihaibudiu:issue6565
Open

[SQL] Pull filters above map, integrals, aggregates, filters#6600
mihaibudiu wants to merge 1 commit into
feldera:mainfrom
mihaibudiu:issue6565

Conversation

@mihaibudiu

Copy link
Copy Markdown
Contributor

Fixes #6565

A set of optimizations, both in Calcite and in our backend, to move filters before other operators when it looks profitable.

Describe Manual Test Plan

Ran all java tests

Checklist

  • Unit tests added/updated

@mihaibudiu
mihaibudiu force-pushed the issue6565 branch 2 times, most recently from 494e10d to 39cb149 Compare July 7, 2026 21:25

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice optimization. Filter pull-up above aggregates, maps, integrate/differentiate, noop, and filter-merge. Read carefully; the safety guards look right:

  • Fanout=1 gate correctly prevents duplicating the source when it has other consumers.
  • Aggregate case uses NoExpression in the value slot and ContainsNoExpression.search after reduction to reject filters that actually depend on the aggregate's output — a clean way to prove key-only dependence.
  • Map/MapIndex relies on shouldInlineComposition — same threshold used elsewhere, so complexity blow-up stays bounded.
  • Integrate/Differentiate/Noop commute with a stateless filter on Z-sets, so pulling through is sound.
  • Filter+Filter merge into AND and calls .after(...) on the source's RelNode to preserve provenance — good.

DBSPCompiler.getFinalCircuit guard if (circuit != null) is a sensible fix for the failed-compilation path.

Persistent-id churn in metadataTests-generateDFRecursive.json is expected (circuit shape changed) and lives only in a compiler test fixture — no runtime checkpoint compatibility concern.

Two nitpicks, non-blocking:

  • The filter-merge branch calls .copyAnnotations(operator) on the combined filter but drops annotations from the upstream filter (source). Probably fine for now, but if either filter carried something meaningful (e.g. hoisted-from-view markers), it's gone.
  • Utilities.showResultSet signature reformat is unrelated to the optimization — trivial, keep it, just noting.

LGTM.

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at tip 67e0722 (delta since 39cb149).

Delta is cleanly additive:

  • PullFilterVisitor: new branch handling DBSPConstantOperator sources. For each (row, weight) in the constant Z-set, calls the filter closure, reduces and Simplifys the result; if every element folds to a DBSPBoolLiteral non-null value, replaces filter(const) with a filtered constant. Bails conservatively on NULL results or anything that doesn't collapse to a bool literal. Sound — filter is pure and inputs are literals.
  • CircuitOptimizer: adds PropagateEmptySources after PullFilterVisitor (natural follow-up now that filters can produce empty constants).
  • SetOpTests.simplifyIntersect: verifies INTERSECT with an unmatchable row (5, NULL against a NOT NULL column) collapses to a constant/source/sink — good end-to-end coverage of the new path.

Nits (still non-blocking, same shape as before):

  • Filter annotations aren't copied onto the new DBSPConstantOperator. Probably fine since it's a constant, but consistent with the earlier nit.
  • The "Should not happen" comment for b.value == null is a bit optimistic — WHERE semantics treat NULL as false, so folding null-bool to drop-the-row would recover a few more cases. Conservative bail is safe, just leaves optimization on the table.

Approving.

@mihaibudiu

Copy link
Copy Markdown
Contributor Author

The "Should not happen" comment for b.value == null is a bit optimistic — WHERE semantics treat NULL as false, so folding null-bool to drop-the-row would recover a few more cases. Conservative bail is safe, just leaves optimization on the table.

This is not optimistic, this is our IR, and all filter predicates return non-nullable boolean values

@Override
public void postorder(DBSPFlatMapIndexOperator node) {
// Source is input
this.filterFound = true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this just checks that a filter is found, not that it is pulled above anything? my naive question would be why not push it to queue so you can assert something about the order

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The next assert says "the filter is after the input", which means it was pulled.

@Override
public void postorder(DBSPFlatMapIndexOperator node) {
// Source is input
this.filterFound = true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here?

// This is simplified to an empty set
@Override
public void postorder(DBSPSimpleOperator operator) {
Assert.assertTrue(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This indicates that this program was shrunk to a constant output, there are no other operators. The INTERSECT was completely eliminated

super.postorder(operator);
return;
}
if (source.node().is(DBSPAggregateLinearPostprocessOperator.class)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe this should be a helper isAggregateOperator(x)

this.map(operator, result);
return;
}
} else if (source.node().is(DBSPMapOperator.class)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isMappyThing(x)

if (simplified.is(DBSPBoolLiteral.class)) {
DBSPBoolLiteral b = simplified.to(DBSPBoolLiteral.class);
if (b.value == null) {
// Should not happen

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to print a warning if it does?

DBSPClosureExpression filter = operator.getClosureFunction();
DBSPConstantOperator c = source.node().to(DBSPConstantOperator.class);
DBSPExpression value = c.getFunction();
Simplify simplify = new Simplify(this.compiler);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SimplifyWhat

import java.util.Map;

/**
* Move filters up in a plan, towards sources.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't see tests that evaluate all these cases, maybe they already exist elsewhere

@mihaibudiu
mihaibudiu added this pull request to the merge queue Jul 20, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 20, 2026
@mihaibudiu
mihaibudiu enabled auto-merge July 20, 2026 21:15
@mihaibudiu
mihaibudiu added this pull request to the merge queue Jul 20, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 21, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Jul 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 21, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Jul 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 21, 2026
@mihaibudiu
mihaibudiu force-pushed the issue6565 branch 2 times, most recently from da30ae6 to f76f08b Compare July 21, 2026 23:48
@mihaibudiu
mihaibudiu enabled auto-merge July 21, 2026 23:48
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
@mihaibudiu
mihaibudiu added this pull request to the merge queue Jul 21, 2026

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 82714218 — APPROVE.

Rebase onto main plus a small cleanup: the inline DBSPConstantOperator branch in PullFilterVisitor.postorder(DBSPFilterOperator) has been extracted into PropagateConstants.filterConstant(compiler, constOp, filterClosure). Same semantics as before (per-row reduce + Simplify, bail if any result is non-DBSPBoolLiteral or null), just with the loop lifted into a shared helper — less duplication and easier to reason about. PullFilterVisitor now only imports what it still uses; the HashMap/DBSPZSetExpression/DBSPBoolLiteral/Simplify imports are dropped, which is nice.

No behavioural change from the version I approved at 67e0722f.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 22, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Jul 22, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 22, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Jul 22, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[SQL] Optimization to pull filters above aggregates

3 participants