pipeline-manager: use runtime status details for connector stats by snkas · Pull Request #6564 · feldera/feldera

snkas · 2026-06-30T09:28:18Z

Instead of the manager polling all pipelines every GET request for its list when using the status_with_connectors selector, instead incorporate the connector stats into the runtime status details. The details are regularly updated by the runner as it does its /status checks. As such, it will take some time for the latest connector stats to show up after this change (approximately between 1-15 seconds). At the cost of this slight staleness, the latency of the GET request for the list of pipelines with that selector will be significantly reduced (up to 10x faster is observed).

StorageStatusDetails is now the type shown in the API for the storage_status_details, instead of the generic JSON value type. The same for RuntimeStatusDetails.

Backward incompatibility notes:

Pipelines of older versions will continue to have their connector stats shown as before. After the user base has migrated past this change, the GET request fetching fallback can be removed.
Runtime status details were directly used for the approval diff. Now, if this is present (during the AwaitingApproval runtime status), it is located at: deployment_runtime_status_details.approval_diff. The API converts automatically the old JSON values into the JSON matching the strongly typed RuntimeStatusDetails.

The Web Console and fda clients are adjusted to read the approval diff from the new nested location.

PR information

Integration test added
Breaking changes: see above backward incompatibility notes.

Remaining tasks

Web Console changes accommodating for the difference in deployment_runtime_status_details (@Karakatiza666)
Check how blocking the connector stats retrieval function is (@blp / @ryzhyk)
Backward compatible grace period with the old mechanism still in place

snkas · 2026-06-30T09:33:33Z

CI run: https://github.com/feldera/feldera/actions/runs/28434522017

mythical-fred

LGTM — clean refactoring that eliminates per-request HTTP forwarding in favor of piggybacking on the runner's existing /status polling. Three inline notes.

mythical-fred · 2026-06-30T10:01:25Z

-
+/// Retrieves the error statistics across all endpoints.
+///
+/// TODO: how blocking is this? Will this block `/status` under high load?


You can close this TODO — retrieve_error_stats is non-blocking. It iterates in-memory InputEndpointStatus/OutputEndpointStatus maps and reads atomics (Ordering::Acquire). No I/O, no locks beyond the one already held by controller.status(). The cost is O(num_connectors) per /status call, which is negligible.

mythical-fred · 2026-06-30T10:01:25Z

+
+    /// Statistics across all connectors.
+    ///
+    /// Specifically useful for: `Paused`, `Running`.


Since this field is part of the OpenAPI spec (via ToSchema), the rustdoc should mention the staleness window. Something like: "Updated periodically by the runner's status polling (approximately every 1–15 seconds). Values may not reflect the most recent state."

mythical-fred

Re-approve after force-push. Nice improvements since the last tip:

RuntimeStatusDetails is now a proper ToSchema struct with reason / connector_stats / approval_diff, replacing the generic JSON value in the OpenAPI surface.
StorageStatusDetails gained ToSchema and switched VecDeque → Vec (right call for a serialized wire type).
The backward-compatibility comment on approval_diff (reserving program_diff as a discriminator) is a nice piece of forward thinking.
Changelog entry covers both the strong-typing break and the pipeline-upgrade requirement for "Runtime errors" on the pipeline list.

One nit inline.

Karakatiza666 · 2026-07-02T12:26:59Z

I pushed the necessary UI changes - using status instead of status_with_connectors, and deployment_runtime_status_details instead of connectors. Does it make sense to deprecate the PipelineSelectedInfo.connectors field?

mythical-fred · 2026-07-03T16:00:23Z

Follow-up on the "Backward compatibility for old pipelines" commit — approval still stands, but a couple of observations on the changelog entry:

The Unreleased block now duplicates the "deployment_runtime_status_details is now strongly typed…" bullet from v0.313.0 immediately below it. That bullet already shipped in 0.313.0; only the new "connector stats cached from this version onward" line belongs in Unreleased.
The indentation in Unreleased mixes tabs and spaces (the ## Unreleased heading has a leading tab plus spaces, the replay-journal bullet has a leading tab, the two new bullets are pure-space indented). Not a rendering bug in MDX but it stands out next to the v0.313.0 block which is consistently space-indented.

The migrator itself is nice — the parametrised test_backward_compatible_runtime_status_details table locks the wire behaviour for every JSON kind, including the program_diff → approval_diff restructuring path, which is exactly what I would want pinned before this hits older deployments.

snkas · 2026-07-09T10:09:21Z

CI run: https://github.com/feldera/feldera/actions/runs/29010644278

snkas · 2026-07-09T11:32:53Z

CI run: https://github.com/feldera/feldera/actions/runs/29015194824

Instead of the manager polling all pipelines every GET request for its list when using the `status_with_connectors` selector, instead incorporate the connector stats into the runtime status details. The details are regularly updated by the runner as it does its `/status` checks. As such, it will take some time for the latest connector stats to show up after this change (approximately between 1-15 seconds). At the cost of this slight staleness, the latency of the GET request for the list of pipelines with that selector will be significantly reduced (up to 10x faster is observed). `StorageStatusDetails` is now the type shown in the API for the `storage_status_details`, instead of the generic JSON value type. The same for `RuntimeStatusDetails`. Backward incompatibility notes: - Pipelines of older versions will continue to have their connector stats shown as before. After the user base has migrated past this change, the GET request fetching fallback can be removed. - Runtime status details were directly used for the approval diff. Now, if this is present (during the `AwaitingApproval` runtime status), it is located at: `deployment_runtime_status_details.approval_diff`. The API converts automatically the old JSON values into the JSON matching the strongly typed `RuntimeStatusDetails`. The Web Console and `fda` clients are adjusted to read the approval diff from the new nested location. Co-authored-by: Karakatiza666 <bulakh.96@gmail.com> Signed-off-by: Karakatiza666 <bulakh.96@gmail.com> Signed-off-by: Simon Kassing <simon.kassing@feldera.com>

mythical-fred

LGTM

snkas commented Jun 30, 2026

View reviewed changes

Comment thread crates/adapters/src/server.rs Outdated

snkas temporarily deployed to ci June 30, 2026 09:59 — with GitHub Actions Inactive

mythical-fred approved these changes Jun 30, 2026

View reviewed changes

mihaibudiu approved these changes Jun 30, 2026

View reviewed changes

snkas force-pushed the fix-list-retrieval branch 3 times, most recently from 234eeee to 6e6306e Compare July 1, 2026 16:50

mythical-fred approved these changes Jul 1, 2026

View reviewed changes

Comment thread docs.feldera.com/docs/changelog.md Outdated

snkas force-pushed the fix-list-retrieval branch from 6e6306e to 4f0b223 Compare July 2, 2026 09:59

Karakatiza666 force-pushed the fix-list-retrieval branch from 8dc7d0b to 4bcdc8b Compare July 8, 2026 16:22

snkas commented Jul 8, 2026

View reviewed changes

Comment thread js-packages/web-console/src/lib/functions/pipelines/pipelineDiff.spec.ts Outdated

Karakatiza666 force-pushed the fix-list-retrieval branch from 4bcdc8b to b2f088c Compare July 8, 2026 18:02

snkas force-pushed the fix-list-retrieval branch from b2f088c to 162730d Compare July 9, 2026 09:57

snkas temporarily deployed to ci July 9, 2026 10:32 — with GitHub Actions Inactive

snkas force-pushed the fix-list-retrieval branch 2 times, most recently from b9f1333 to cf029c2 Compare July 9, 2026 11:31

snkas temporarily deployed to ci July 9, 2026 11:50 — with GitHub Actions Inactive

gz approved these changes Jul 10, 2026

View reviewed changes

snkas force-pushed the fix-list-retrieval branch from cf029c2 to f90a6c2 Compare July 13, 2026 09:25

snkas force-pushed the fix-list-retrieval branch from f90a6c2 to c56c5c1 Compare July 15, 2026 08:55

snkas enabled auto-merge July 15, 2026 08:58

snkas added this pull request to the merge queue Jul 15, 2026

mythical-fred approved these changes Jul 15, 2026

View reviewed changes

Merged via the queue into main with commit 9751923 Jul 15, 2026
1 check passed

snkas deleted the fix-list-retrieval branch July 15, 2026 10:58

Uh oh!

Conversation

snkas commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

snkas commented Jun 30, 2026

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

mythical-fred Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

mythical-fred Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Karakatiza666 commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mythical-fred commented Jul 3, 2026

Uh oh!

Uh oh!

snkas commented Jul 9, 2026

Uh oh!

snkas commented Jul 9, 2026

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

snkas commented Jun 30, 2026 •

edited

Loading

Karakatiza666 commented Jul 2, 2026 •

edited

Loading