[adapters] checkpoint-suspend on SIGTERM#6482
Open
swanandx wants to merge 1 commit into
Open
Conversation
mythical-fred
left a comment
There was a problem hiding this comment.
Useful feature, and the refactor of trigger_suspend looks faithful to the original. A few blocking items before approval:
- New user-visible env var
FELDERA_CLEAN_SHUTDOWN_ON_SIGTERMwith no documentation. This belongs indocs.feldera.com/operations/(Kubernetes/deployment section), together with guidance onterminationGracePeriodSeconds. - New signal-shutdown behavior with no tests and no "Manual testing" section in the PR description. At minimum a brief description of how this was exercised (e.g.
kill -TERMon a running pipeline and observed checkpoint completing) is needed.
Non-blocking observations inline (K8s grace-period interaction, unbounded wait loop, SIGHUP/SIGQUIT no longer handled when disable_signals() is called, expect() defensiveness, macro-hygiene workaround).
Infra-initiated termination (node drain, eviction) doesn't involve the runner, so the pipeline does not stop gracefully. To solve this, we add an env var FELDERA_CLEAN_SHUTDOWN_ON_SIGTERM: when set, SIGTERM runs the same checkpoint-and-suspend as /suspend before stopping. disabled by default. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
cbae24b to
b00816c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Infra-initiated termination (node drain, eviction) doesn't involve the runner, so the pipeline does not stop gracefully.
To solve this, we add an env var FELDERA_CLEAN_SHUTDOWN_ON_SIGTERM: when set, SIGTERM runs the same checkpoint-and-suspend as /suspend before stopping.
disabled by default.
tested with local runner &
FELDERA_CLEAN_SHUTDOWN_ON_SIGTERM=1. Started pipeline and thenkill -TERM <pid>Breaking Changes?
not breaking change