Skip to content

[SQL] GC of tables with keys and LATENESS should never use retain_values, but can use retain_keys when keys have LATENESS#6693

Open
mihaibudiu wants to merge 1 commit into
feldera:mainfrom
mihaibudiu:issue6690
Open

[SQL] GC of tables with keys and LATENESS should never use retain_values, but can use retain_keys when keys have LATENESS#6693
mihaibudiu wants to merge 1 commit into
feldera:mainfrom
mihaibudiu:issue6690

Conversation

@mihaibudiu

Copy link
Copy Markdown
Contributor

Fixes #6690

The correct way to GC a collection with PRIMARY KEY and LATENESS on a field of the primary key is to use a integrate_trace_retain_keys operator. The previous implementation was using integrate_trace_retain_values, and was unsound: it could delete keys that should be permanently frozen.

Unfortunately this implementation GCs fewer programs.

Checklist

  • Unit tests added/updated

…ues, but can use retain_keys when keys have LATENESS

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE.

Fixes #6690 correctly. The previous input GC path installed accumulate_integrate_trace_retain_values on the upserted input trace, which is unsound: the input trace resolves upserts/deletions by key, so dropping values below the waterline breaks the "reject late writes to a frozen key" invariant — a subsequent insert on the same key looks brand-new, and there is no retraction to emit against.

The new logic is right in both directions:

  • LATENESS inside the primary key: safe to GC by key, using integrate_trace_retain_keys (non-accumulate variant, because inputs live outside the accumulate framework). A pruned key can never be legally referenced again — any later insert/update/delete on it carries the same key and is rejected as late.
  • LATENESS outside the primary key: no GC at all. Live rows are needed for retractions; frozen rows are needed as the "this key is late, reject writes" sentinel. The comment block in InsertLimiters.processLateness articulates this well; worth keeping verbatim.

Nice touches:

  • DBSPIntegrateTraceRetainKeysOperator.accumulate is a first-class boolean that flows through with(...), create(...), and fromJson(...), so the two runtime variants stay in sync with the operator instance.
  • StrayGC.checkAccumulate enforces the input↔non-accumulate / non-input↔accumulate invariant as an InternalCompilerError — exactly the kind of silent-no-op the old code invited. This is the right defense.
  • Removing accumulate from RetainValuesOperator (it was only ever false for inputs, which no longer use it) simplifies the surface.
  • Test coverage is thorough: gcUpsertBoundary, gcDeleteOldRow, gcUpsertOldRow exercise on-waterline updatability, late-delete rejection, and late-upsert rejection with clear frozen/waterline diagrams in the comments. checkInputGC asserts the structural invariant (exactly N retain_keys on inputs, zero retain_values on inputs).
  • Stale FIXME: see #2669 comments in dbsp/src/operator/{input,dynamic/input}.rs are cleaned up.

Trade-off is honestly acknowledged in the PR description: this GCs strictly fewer programs than before. That is the right call — correctness over completeness. Programs with LATENESS on a non-key column simply don't benefit from input-trace GC anymore, and users who want that should either add LATENESS to a key column or accept unbounded input state.

One micro-observation, not blocking:

  • keyWaterlineFields is indexed against the waterline tuple, which is built by iterating the same columns and appending only where hasLateness. Order-preservation between the two loops is load-bearing here; a one-liner comment above latenessIndex++ noting "index into the waterline tuple; must match the order used to build it" would save a future reader ten minutes. Fine to defer.

CI green, no other reviewers yet. Ship it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[SQL] Output of input_map_with_waterline should not be GC-ed using retain_values

2 participants