[SQL] GC of tables with keys and LATENESS should never use retain_values, but can use retain_keys when keys have LATENESS#6693
Conversation
…ues, but can use retain_keys when keys have LATENESS Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
mythical-fred
left a comment
There was a problem hiding this comment.
APPROVE.
Fixes #6690 correctly. The previous input GC path installed accumulate_integrate_trace_retain_values on the upserted input trace, which is unsound: the input trace resolves upserts/deletions by key, so dropping values below the waterline breaks the "reject late writes to a frozen key" invariant — a subsequent insert on the same key looks brand-new, and there is no retraction to emit against.
The new logic is right in both directions:
- LATENESS inside the primary key: safe to GC by key, using
integrate_trace_retain_keys(non-accumulate variant, because inputs live outside the accumulate framework). A pruned key can never be legally referenced again — any later insert/update/delete on it carries the same key and is rejected as late. - LATENESS outside the primary key: no GC at all. Live rows are needed for retractions; frozen rows are needed as the "this key is late, reject writes" sentinel. The comment block in
InsertLimiters.processLatenessarticulates this well; worth keeping verbatim.
Nice touches:
DBSPIntegrateTraceRetainKeysOperator.accumulateis a first-class boolean that flows throughwith(...),create(...), andfromJson(...), so the two runtime variants stay in sync with the operator instance.StrayGC.checkAccumulateenforces the input↔non-accumulate / non-input↔accumulate invariant as anInternalCompilerError— exactly the kind of silent-no-op the old code invited. This is the right defense.- Removing
accumulatefromRetainValuesOperator(it was only everfalsefor inputs, which no longer use it) simplifies the surface. - Test coverage is thorough:
gcUpsertBoundary,gcDeleteOldRow,gcUpsertOldRowexercise on-waterline updatability, late-delete rejection, and late-upsert rejection with clear frozen/waterline diagrams in the comments.checkInputGCasserts the structural invariant (exactly N retain_keys on inputs, zero retain_values on inputs). - Stale
FIXME: see #2669comments indbsp/src/operator/{input,dynamic/input}.rsare cleaned up.
Trade-off is honestly acknowledged in the PR description: this GCs strictly fewer programs than before. That is the right call — correctness over completeness. Programs with LATENESS on a non-key column simply don't benefit from input-trace GC anymore, and users who want that should either add LATENESS to a key column or accept unbounded input state.
One micro-observation, not blocking:
keyWaterlineFieldsis indexed against the waterline tuple, which is built by iterating the same columns and appending only wherehasLateness. Order-preservation between the two loops is load-bearing here; a one-liner comment abovelatenessIndex++noting "index into the waterline tuple; must match the order used to build it" would save a future reader ten minutes. Fine to defer.
CI green, no other reviewers yet. Ship it.
Fixes #6690
The correct way to GC a collection with PRIMARY KEY and LATENESS on a field of the primary key is to use a
integrate_trace_retain_keysoperator. The previous implementation was usingintegrate_trace_retain_values, and was unsound: it could delete keys that should be permanently frozen.Unfortunately this implementation GCs fewer programs.
Checklist