Repository navigation
[Bug]: Orphaned map/parallel branch can checkpoint after parent completion #640
Description
Expected Behavior
When a map or parallel operation completes early because its completion
criteria are met, an in-flight branch marked as orphaned must not checkpoint
its terminal SUCCEED or FAIL result after the parent context completes.
Checkpoint admission should have a deterministic order:
- If the branch checkpoint is admitted first, it may be sent before the parent
completion. - If the parent completion marks the branch orphaned first, the branch
checkpoint must be rejected and must not reach the checkpoint service.
Actual Behavior
ExecutionState.create_checkpoint() validates _parent_done while holding
_parent_done_lock, releases that lock, invokes the plugin hook, and only later
enqueues the update while holding _completion_lock.
This creates a check-to-enqueue race:
- An in-flight branch calls
create_checkpoint()with its terminal context
SUCCEEDupdate and passes the_parent_donecheck. - Before the branch update is enqueued, the map/parallel parent checkpoints
SUCCEEDand_mark_orphans()marks the branch operation ID as orphaned. - The branch resumes and enqueues its terminal
SUCCEEDupdate because orphan
status is not checked again. checkpoint_batches_forever()sends every queued update without
revalidating orphan status.
A deterministic reproduction recorded two successful checkpoint-service calls:
service call 1: parent SUCCEED with BatchResult branch status STARTED
service call 2: branch SUCCEED with payload "branch-result"
Both synchronous checkpoint callers were released after the mock service
returned a new checkpoint token.
The customer-visible BatchResult remains replay-consistent in the current
implementation because normal payloads replay the parent's serialized result
and ReplayChildren summaries record startedIndexes. However, the durable
operation hierarchy can contain a terminal branch result checkpointed after
its parent context completed.
Relevant code:
packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/state.py,
ExecutionState.create_checkpoint()around the_parent_done_lock
validation and later_checkpoint_queue.put()packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/state.py,
checkpoint_batches_forever()where queued updates are sent without orphan
revalidation
Steps to Reproduce
- Create an
ExecutionStatewith a branch operation registered as a child of
a map/parallel parent. - Start a branch terminal
CONTEXT/SUCCEEDcheckpoint on a worker thread. - Pause that thread after its initial orphan validation but before queue
insertion, for example using a blocking synchronous plugin hook. - Checkpoint the parent
CONTEXT/SUCCEED, causing_mark_orphans()to mark
the branch operation ID. - Allow the parent update to be sent successfully.
- Release the branch thread.
- Observe that the branch terminal update is sent in a subsequent checkpoint
API call and the synchronous branch caller completes successfully.
SDK Version
1.8.0, repository HEAD 7ac7acc6a7dae231f2abbb8e37f9780cc9b89af0
Python Version
3.14
Is this a regression?
No known regression.
Possible Fix
Make orphan validation and queue insertion atomic relative to parent
completion:
- Inside the final
_completion_lockblock, acquire_parent_done_lockusing
a consistent global lock order. - Recheck whether
operation_update.operation_idis in_parent_done
immediately before queue insertion. - Optionally also reject when
operation_update.parent_idis in
_parent_done, preventing new durable descendants from being created under
an already orphaned branch. - Insert the queued operation while still holding
_parent_done_lock.
This produces deterministic ordering: either the child is queued first, or
parent completion wins and the child is rejected.
Add deterministic unit tests for:
- A terminal branch checkpoint paused between initial validation and enqueue.
- A new operation created under an already orphaned branch.
- The valid child-first ordering.
- Replay consistency for normal and
ReplayChildrenmap/parallel results.
Additional Context
The existing core test suite passes (1504 passed, plus 5 subtests), but it
does not currently cover this interleaving.
Activity
warren830 commentedon Aug 30, 2026
I reproduced this deterministically on current
main(24c75363) with a no-sleep interleaving test: an already-known child passes the initial parent-done validation, the parent completes and marks it orphaned, and the child then resumes and is currently enqueued.I am working on a focused fix for #640 only: re-check the child orphan state in the atomic validation-and-enqueue section while keeping plugin callbacks outside the locks. I will not include the late-descendant behavior tracked separately in #641. I plan to open a draft PR with the red-to-green test and local gate results. If an implementation is already in flight, please let me know and I will step back.