Skip to content

[adapters] Fix test_clock_advance_survives_checkpoint.#6581

Merged
ryzhyk merged 1 commit into
mainfrom
fix-clock-checkpoint-journal-race
Jul 3, 2026
Merged

[adapters] Fix test_clock_advance_survives_checkpoint.#6581
ryzhyk merged 1 commit into
mainfrom
fix-clock-checkpoint-journal-race

Conversation

@ryzhyk

@ryzhyk ryzhyk commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

This fixes two bugs that caused the test to fail:

  1. A bug in the test itself, where the test did not wait for the journal to finish replay before probing pipeline's state
  2. A minor bug introduced by the recent fix to exactly-once FT where the last step before the pipeline terminated did not get recorded in the journal. I don't think this violates the exactly-once FT contract, but it does make it harder to write robust tests.

Describe Manual Test Plan

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

Describe Incompatible Changes

This fixes two bugs that caused the test to fail:

1. A bug in the test itself, where the test did not wait for the journal to
   finish replay before probing pipeline's state
2. A minor bug introduced by the recent fix to exactly-once FT where the last
   step before the pipeline terminated did not get recorded in the journal.
   I don't think this violates the exactly-once FT contract, but it does make
   it harder to write robust tests.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
@ryzhyk
ryzhyk requested a review from gz July 2, 2026 20:48
@ryzhyk ryzhyk added ft Fault tolerant, distributed, and scale-out implementation connectors Issues related to the adapters/connectors crate labels Jul 2, 2026
let reader = controller.get_input_endpoint("now").unwrap();
let clock_reader = reader.as_any().downcast::<super::ClockReader>().unwrap();

// Replay runs asynchronously on the circuit thread, feeding the journaled

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I guess it would be nice to hide this complexity somehow, I am pretty sure I wont remember this needs to be done next time I have to work with this API.

maybe get_input_endpoint could block until replay is complete

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I guess this test should have checked for controller status. Can't spend more time on this now.

@ryzhyk
ryzhyk added this pull request to the merge queue Jul 2, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 2, 2026

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ryzhyk
ryzhyk added this pull request to the merge queue Jul 2, 2026

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both fixes read cleanly.

The controller.rs change is the interesting one: keeping the journal write on shutdown while skipping only the endpoint-lifecycle diff is exactly right. The old early-return silently dropped the last step's already-computed input, which is precisely the kind of "silent loss of committed work" that FT is supposed to make impossible. The new comment spells out both halves — why we skip the diff, and why we must not skip the journal entry — which is what future-me would want to see when staring at this code again.

The test fix (waiting for is_replaying()) is a textbook async-test hazard and the 30s bounded wait with an explicit assertion is the right shape.

Nit: the regression-coverage doc comment on the test is helpful; consider tying it to a bug id or a short "before this fix, restart observed value X" example so it survives future refactors.

Merged via the queue into main with commit b7b342f Jul 3, 2026
1 check passed
@ryzhyk
ryzhyk deleted the fix-clock-checkpoint-journal-race branch July 3, 2026 02:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

connectors Issues related to the adapters/connectors crate ft Fault tolerant, distributed, and scale-out implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants