Skip to content

[adapters] Report batch write progress for Delta Lake output connector.#6008

Open
swanandx wants to merge 1 commit intomainfrom
issue5970
Open

[adapters] Report batch write progress for Delta Lake output connector.#6008
swanandx wants to merge 1 commit intomainfrom
issue5970

Conversation

@swanandx
Copy link
Copy Markdown
Contributor

@swanandx swanandx commented Apr 8, 2026

check commit message

Ref: #5970

Describe Manual Test Plan

ran locally and checked stats endpoint

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

adding metrics should not be breaking change

@swanandx swanandx requested a review from ryzhyk April 8, 2026 06:47
Copy link
Copy Markdown
Contributor

@ryzhyk ryzhyk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! I think this is the right direction, but maybe we can simplify the design.

@ryzhyk
Copy link
Copy Markdown
Contributor

ryzhyk commented Apr 8, 2026

It would be nice to add this to other connectors as well. But I don't think we will be able to do this in this PR. We should however be able to distinguish between the connector not supporting this counter and the counter being 0, so we can show the in different ways in the UI.

Copy link
Copy Markdown

@mythical-fred mythical-fred left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: commit message hygiene ("address review comments"), and docs for new user-visible metric. See inline for a functional blocker.

@swanandx swanandx force-pushed the issue5970 branch 2 times, most recently from b9a90c2 to 8c96cb3 Compare April 8, 2026 12:29
@swanandx swanandx requested review from mythical-fred and ryzhyk April 8, 2026 12:29
Copy link
Copy Markdown

@mythical-fred mythical-fred left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still missing docs for the new user-visible metric (batch_records_written). Please add a short note in docs.feldera.com (metrics/stats) before merge.

When a Delta Lake output connector writes a large batch, the write can take
a long time with no visible progress. Add a `batch_records_written` metric
that reports how many records have been written so far in the current batch,
giving users live feedback during long writes.

The metric is updated incrementally per chunk by each parallel write range
and resets to 0 after the batch is committed or on failure. Forward-progress
updates use `fetch_add` to atomically increment the shared counter, while
rollbacks on retry use `fetch_sub` to subtract exactly the failed attempt's
contribution. Batch boundaries and terminal failures use `store(0) for a
clean reset.

The counter is owned by encoder, e.g. DeltaTableWriter and registered with the
controller's OutputEndpointMetrics via register_batch_progress_counter()
during construction. To make this possible, add_output() is called before
encoder creation so the metrics slot exists when the connector is built.
Connectors that don't register a counter show batch_records_written: null.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Copy link
Copy Markdown

@mythical-fred mythical-fred left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants