In follow and snapshot_and_follow mode, a deletion-vector delete of K rows from an N-row data file makes the connector ingest ~2N−K records instead of K. The Remove action is read as a full-file retraction (all N rows) and the same-path Remove action is read as a full-file retraction and the re-added file (with the new DV) as ~N−K inserts; DBSP then cancels the unchanged majority.
Result is correct, just wasted computation
Minimal repro: DV-enabled Delta table, one 100-row data file (v1). Delete a single row (v2, merge-on-read DV). Pipeline reads it in snapshot_and_follow from version 1.
- COUNT(*): 100 → 99 (correct, −1)
- records_input_total: 100 → 299 (+199 to delete 1 row)
Expected: the delete should ingest ≈ K records (the rows the DV newly masks), not ≈ 2N.
Root cause: process_log_entry (non-CDC branch) processes each Add/Remove independently over the full file; it does not recognize that a same-path Add+Remove pair differs only by deletion vector, so it never reads just the DV delta.
In follow and snapshot_and_follow mode, a deletion-vector delete of K rows from an N-row data file makes the connector ingest ~2N−K records instead of K. The Remove action is read as a full-file retraction (all N rows) and the same-path Remove action is read as a full-file retraction and the re-added file (with the new DV) as ~N−K inserts; DBSP then cancels the unchanged majority.
Result is correct, just wasted computation
Minimal repro: DV-enabled Delta table, one 100-row data file (v1). Delete a single row (v2, merge-on-read DV). Pipeline reads it in snapshot_and_follow from version 1.
Expected: the delete should ingest ≈ K records (the rows the DV newly masks), not ≈ 2N.
Root cause: process_log_entry (non-CDC branch) processes each Add/Remove independently over the full file; it does not recognize that a same-path Add+Remove pair differs only by deletion vector, so it never reads just the DV delta.