Tags: devdoshi/feldera
Tags
[dbsp] Merge the inputs in shard_pairs(). shard_pairs() takes a collection of sorted input vectors and shards them. The output vectors must also be sorted, which means that the code must merge them to avoid jumbling data in the output vectors. However, it didn't do that. This commit fixes the problem. Only multihost pipelines use this code. Signed-off-by: Ben Pfaff <blp@feldera.com>
[web-console] Implement Pipeline Health Monitoring UI Improve formatting of date ranges in shared Health UI Make inline drawers become overlays on smaller screens Refactor shared Health UI componetns to be more generic Signed-off-by: Karakatiza666 <bulakh.96@gmail.com>
pipeline-manager: pipeline monitoring events
Both at a regular interval and whenever the status of the pipeline
changes, store this as an event in the database. If there are no
changes, every 10 minutes it stores a duplicate event. If there are only
changes to the details, it only updates it every 10 seconds. However, if
there are changes to the status, it can update more frequently than
every 10 seconds.
A cleanup threads runs in the background, waking up every 30 seconds to
clean up for each pipeline events exceeding the retention limit of (by
default) 720. The retained events as such span approximately between 2
hours and 5 days generally. The number of retained events per pipeline
can be changed via CLI argument `--pipeline-monitor-events-retention`
or its corresponding environment variable
`FELDERA_PIPELINE_MONITOR_EVENTS_RETENTION`.
API endpoints:
- `GET /v0/pipelines/<pipeline>/events?selector=all/status`
- `GET /v0/pipelines/<pipeline>/event/latest?selector=all/status`
- `GET /v0/pipelines/<pipeline>/event/<event-id>?selector=all/status`
The clients have been updated with corresponding functionality:
- CLI:
- `fda events <pipeline> <selector>`
- `fda event <pipeline> latest <selector>`
- `fda event <pipeline> <event-id> <selector>`
- Python:
- `pipeline.events(selector=...)`
- `pipeline.event("latest", selector=...)`
- `pipeline.event(event_id, selector=...)`
Signed-off-by: Simon Kassing <simon.kassing@feldera.com>
PreviousNext