Skip to content

feat(coderd): process on_success workspace build orchestration - #25759

Merged
geokat merged 18 commits into
george/plat-143/2-on-success-workspace-build-request-handlingfrom
george/plat-143/3-process-on-success-workspace-build-orchestration
Jul 6, 2026
Merged

feat(coderd): process on_success workspace build orchestration#25759
geokat merged 18 commits into
george/plat-143/2-on-success-workspace-build-request-handlingfrom
george/plat-143/3-process-on-success-workspace-build-orchestration

Conversation

@geokat

@geokat geokat commented May 27, 2026

Copy link
Copy Markdown
Contributor

This PR is part of a stack that adds durable server-side workspace restart support
to the API.

Add the coderd background worker that processes persisted on_success workspace
build orchestration rows. The worker claims ready rows, resolves terminal parent
build outcomes, and creates the child build when the parent completes
successfully.

This wires the worker into coderd startup and wake-ups from workspace build
completion paths. It also adds retry limiting, delayed retries, and a backup poll
so missed wake signals or transient child build failures do not leave
orchestration rows pending indefinitely.

Ref: https://linear.app/codercom/issue/PLAT-143/add-workspace-restart-functionality-to-api
Ref: #5800


This PR is part of a stack that merges into main:

  1. feat: add workspace build orchestration storage #25757
  2. feat: add on_success workspace build request handling #25758
  3. feat(coderd): process on_success workspace build orchestration #25759 👈
  4. chore(coderd/database): purge terminal workspace build orchestrations #25760
  5. feat(cli): server-side restart orchestration demo #25761

Created with stakk

@geokat
geokat force-pushed the george/plat-143/2-on-success-workspace-build-request-handling branch from 58700d9 to 3edff4e Compare May 28, 2026 04:27
@geokat
geokat force-pushed the george/plat-143/3-process-on-success-workspace-build-orchestration branch from 096cb57 to 998848a Compare May 28, 2026 04:27
@geokat geokat changed the title feat(coderd): process on_success workspace build orchestration feat(coderd): process on_success workspace build orchestrations May 28, 2026
@geokat geokat changed the title feat(coderd): process on_success workspace build orchestrations feat(coderd): process on_success workspace build orchestration May 28, 2026
@geokat
geokat marked this pull request as ready for review May 28, 2026 05:31
@github-actions github-actions Bot added the stale This issue is like stale bread. label Jun 10, 2026
@geokat geokat removed the stale This issue is like stale bread. label Jun 12, 2026
@geokat
geokat force-pushed the george/plat-143/2-on-success-workspace-build-request-handling branch from 3edff4e to 21fa1f4 Compare June 16, 2026 23:57
@geokat
geokat force-pushed the george/plat-143/3-process-on-success-workspace-build-orchestration branch from 998848a to cfa75f9 Compare June 16, 2026 23:57
@geokat
geokat force-pushed the george/plat-143/2-on-success-workspace-build-request-handling branch from 21fa1f4 to dca2e69 Compare June 17, 2026 19:24
@geokat
geokat force-pushed the george/plat-143/3-process-on-success-workspace-build-orchestration branch from cfa75f9 to 06af5be Compare June 17, 2026 19:31
@geokat
geokat force-pushed the george/plat-143/2-on-success-workspace-build-request-handling branch from dca2e69 to 2e97f53 Compare June 17, 2026 20:55
@geokat
geokat force-pushed the george/plat-143/3-process-on-success-workspace-build-orchestration branch from 06af5be to 0ddfc81 Compare June 17, 2026 20:55
@geokat
geokat force-pushed the george/plat-143/2-on-success-workspace-build-request-handling branch from 2e97f53 to fa1fd82 Compare June 17, 2026 23:23
@geokat
geokat force-pushed the george/plat-143/3-process-on-success-workspace-build-orchestration branch from 0ddfc81 to a1b168a Compare June 17, 2026 23:23
Add the background worker that claims pending workspace build
orchestration rows, resolves terminal parent build outcomes, and
creates child builds after successful parents.

Wire the worker into coderd startup and pubsub wake-ups from workspace
build completion paths. Add retry limiting, delayed retries, and a
backup poll so missed wake signals do not leave ready rows pending
indefinitely.

Add coverage for happy-path restart, parent failed/canceled outcomes,
child build failure handling, retry delays, and pubsub wake behavior.

Ref: https://linear.app/codercom/issue/PLAT-143/add-workspace-restart-functionality-to-api
Ref: #5800
@geokat
geokat force-pushed the george/plat-143/3-process-on-success-workspace-build-orchestration branch from a1b168a to 5c38677 Compare June 24, 2026 01:11
@Emyrk

Emyrk commented Jun 24, 2026

Copy link
Copy Markdown
Member

/coder-agents-review

@coder-agents-review

coder-agents-review Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Chat: Review in progress (6/6 reviewers complete) | View chat
Requested: 2026-06-30 19:29 UTC by @geokat
Spend: $116.12 / $100.00

Review history
  • R1 (2026-06-24): 17 reviewers, 4 Nit, 3 P2, 9 P3, COMMENT. Review
  • R2 (2026-06-30): 11 reviewers, 4 Nit, 4 P2, 10 P3, COMMENT. Review

deep-review v0.9.0 | Round 3 | 583b339..b127602

Last posted: Round 3, 18 findings (4 P2, 10 P3, 4 Nit), APPROVE. Review

Finding inventory

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P3 Author fixed (69d180f) workspacebuildorchestrator.go:118 time.NewTicker bypasses quartz Clock, untestable backup poll R1 Netero Yes
CRF-2 P2 Author fixed (d0bd7f8) workspacebuildorchestrator.go:262 Orchestrator can start build on soft-deleted workspace R1 Hisoka Yes
CRF-3 P2 Author fixed (cc95c8b) workspacebuildorchestrator.go:138 Non-child-build tx errors block entire queue (head-of-line blocking) R1 Kite P2, Hisoka P3, Mafuuu P3, Meruem P3 Yes
CRF-4 P2 Author fixed (bc26fb1) workspacebuilds_on_success_test.go:459 Cancel test exercises wrong path; ProvisionerJobStatusCanceled untested R1 Bisky P2, Mafuuu P3 Yes
CRF-5 P3 Author contested; panel closed R2 (6/6 accept) workspacebuilds_on_success_test.go:676 No test for retry exhaustion at max attempts R1 Bisky P3, Chopper P3 Yes
CRF-6 P3 Author fixed (cea1757) provisionerdserver.go:2550 Wake signal skipped when preceding workspace event publish fails R1 Meruem Yes
CRF-7 P3 Author fixed (b1c2229) workspacebuildorchestrator.go:57 Goroutine shutdown join missing; diverges from every other component R1 Takumi Yes
CRF-8 P3 Author fixed (1092007) workspacebuildorchestrator.go:420 Auth callback return true is wider than insert-time validation; autostart passes nil R1 Knov Yes
CRF-9 P3 Author fixed (67307a3) workspacebuildorchestrator.go:347 Post-commit log missing workspace_id and uses wrong logger R1 Chopper Yes
CRF-10 P3 Author fixed (f518df0) workspacebuildorchestrator.go:124 Bare "process orchestrations" log message names no failure R1 Leorio Yes
CRF-11 P3 Acknowledged by human reviewer @Emyrk R1 ("Ignore imo") workspacebuildorchestrator.go:62 Subscribe-with-retry duplicates acquirer.go pattern R1 Robin Yes
CRF-12 P3 Author contested; panel closed R2 (5/5 accept) workspacebuildorchestrator.go:441 BuildError classification diverges from prebuilds reconciler R1 Robin Yes
CRF-13 Nit Author fixed (178f69e) workspacebuildorchestrator.go:295 shouldFail naming misleads; nonRetryable is clearer R1 Gon Yes
CRF-14 Nit Author fixed (7767bf6) workspacebuildorchestrator.go:442 errors.As pre-1.26 pattern; errors.AsType is the replacement R1 Ging-Go Yes
CRF-15 Nit Author fixed (83d33be) workspacebuildorchestrator.go:248 Error variable reuse fragile under refactoring R1 Meruem Yes
CRF-16 Nit Author fixed (89071db) workspacebuildorchestrator.go:64 Comment verbosity pattern (12/26 comments have shorter minimum drafts) R1 Gon Yes
CRF-17 P2 Author fixed (4da51b2) wsbuildorchestrator.go:337 Dormant workspace not guarded; orchestrator starts build on dormant workspace R2 Pariston P2, Meruem P3 Yes
CRF-18 P3 Author fixed (b127602) workspacebuilds_on_success_test.go:186 Preset test omits child build success assertion R2 Bisky Yes

Contested and acknowledged

CRF-5 (P3, workspacebuilds_on_success_test.go:676) - No test for retry exhaustion at max attempts

  • Finding: No integration test verifies the SQL transition from pending to failed when attempt_count + 1 >= max_attempt_count.
  • Author defense: The SQL conditional is already tested by TestUpdateWorkspaceBuildOrchestrationRetryByIDMaxAttempts in querier_test.go (prior PR in stack).
  • Panel closure (R2, 6/6): Bisky, Hisoka, Mafu-san, Mafuuu, Pariston, and Kite confirmed the querier test covers the SQL boundary.

CRF-11 (P3, workspacebuildorchestrator.go:62) - Subscribe-with-retry duplicates acquirer.go

  • Finding: The subscribe-with-retry pattern is structurally identical to provisionerdserver/acquirer.go:291. A shared helper would eliminate the copy.
  • Author accepted R1: Thread resolved. Human reviewer @Emyrk said "Ignore imo." No code change.
  • Status: acknowledged by human reviewer. Do not re-evaluate.

CRF-12 (P3, workspacebuildorchestrator.go:441) - BuildError classification diverges from prebuilds reconciler

  • Finding: The orchestrator classifies 400/403/404 as permanent while prebuilds only checks 400.
  • Author defense: "Ignoring: the prebuilds logic is different and changing it would be out of scope anyway."
  • Panel closure (R2, 5/5): Hisoka, Mafu-san, Mafuuu, Pariston, and Kite confirmed the divergence is intentional. Different subsystems, different retry semantics.

Law analysis

Effective LOC: 1095 (+504 prod, +591 test). Head SHA: 5c38677. Verdict: Don't split. Enforcement: Advisory.

Round log

Round 1

Panel (17 reviewers). 3 P2, 9 P3, 4 Nit. Reviewed against 583b339..5c38677.

Round 2

Churn guard: 13 addressed, 2 contested, 1 acknowledged (human @Emyrk). Override BLOCKED->PROCEED (CRF-11 has explicit human decision). Panel (11 reviewers). Contested CRF-5 closed (6/6), CRF-12 closed (5/5). 1 P2, 1 P3 new. Reviewed against 583b339..69c59d1.

Round 3

Churn guard: 2 addressed, 0 contested, 0 silent. PROCEED. Panel (4 reviewers). 0 new findings. All 18 findings resolved. APPROVE. Reviewed against 583b339..b127602.

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well-structured background worker that mirrors established codebase patterns (lifecycle executor, provisioner acquirer). The problem-solution alignment is tight: orchestration rows need a consumer, this is the consumer. Test density is strong at 54%, covering happy paths, parent failures, cancellations, retryable vs non-retryable child errors, and cross-workspace isolation. Error classification in childBuildErrorShouldFailOrchestration is well-calibrated, and the post-transaction race handling with sql.ErrNoRows is correct and documented.

As one reviewer put it: "This is how you document a security-sensitive bypass: name the prior authorization, name the constraint, name what is still present. Every authorization bypass in this codebase should read like this."

Severity count: 3 P2, 9 P3, 4 Nit.

The three P2s are: (1) a missing workspace.Deleted guard that lets the orchestrator start builds on soft-deleted workspaces, (2) head-of-line blocking when a non-child-build transaction error permanently fails for one row, and (3) TestPostWorkspaceBuildsOnSuccessParentCanceled exercising the ProvisionerJobStatusFailed branch instead of the ProvisionerJobStatusCanceled branch it claims to test.

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/wsbuildorchestrator/wsbuildorchestrator.go
Comment thread coderd/workspacebuildorchestrator.go Outdated
Comment thread coderd/workspacebuilds_on_success_test.go Outdated
Comment thread coderd/workspacebuilds_on_success_test.go
Comment thread coderd/provisionerdserver/provisionerdserver.go Outdated
Comment thread coderd/workspacebuildorchestrator.go Outdated
Comment thread coderd/workspacebuildorchestrator.go Outdated
Comment thread coderd/workspacebuildorchestrator.go Outdated
Comment thread coderd/wsbuildorchestrator/wsbuildorchestrator.go Outdated
Comment thread coderd/workspacebuildorchestrator.go Outdated
Comment thread coderd/wspubsub/wspubsub.go
Comment thread coderd/workspacebuildorchestrator.go Outdated
Comment thread coderd/workspacebuildorchestrator.go Outdated
Comment thread coderd/workspacebuildorchestrator.go Outdated
@geokat

geokat commented Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 2. All 13 R1 fixes verified by multiple reviewers: deleted-workspace guard, head-of-line blocking fix, cancel test, wake signal decoupling, goroutine shutdown join, nil auth callback, log improvements, quartz clock, naming, errors.AsType, and comment cleanup. The package extraction into coderd/wsbuildorchestrator/ is clean and well-motivated.

Contested findings resolved by panel vote: CRF-5 closed (6/6, querier test covers the SQL boundary), CRF-12 closed (5/5, divergence from prebuilds is intentional).

Severity count for new findings: 1 P2, 1 P3.

As one reviewer noted: "The error handling architecture is well-considered. Carrying childBuildErr out of the transaction and making the fail-vs-retry decision outside the tx is a clean separation."

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/wsbuildorchestrator/wsbuildorchestrator.go
Comment thread coderd/workspacebuilds_on_success_test.go
@geokat

geokat commented Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 3. All 18 findings from R1-R2 are resolved: 15 fixed (with tests), 2 closed by panel vote (6/6 and 5/5), 1 acknowledged by human reviewer. The R2 fixes (dormancy guard, preset test assertion) are clean and verified. The markFailed dry-refactor reduced ~60 lines of repeated boilerplate.

Across three rounds, 32 reviewers examined this PR. The dormant-workspace and deleted-workspace guards, the head-of-line blocking fix, the goroutine shutdown join, the nil auth callback, and the wake signal decoupling all addressed real issues. The test suite (59% density, 933 lines) covers happy paths, parent failures, cancellations, deleted/dormant workspaces, retryable/non-retryable child errors, and concurrent worker behavior.

No new findings from the R3 panel (Bisky, Mafuuu, Hisoka, Kite).

🤖 This review was automatically generated with Coder Agents.

@geokat
geokat requested a review from Emyrk June 30, 2026 20:26
@geokat
geokat merged commit 59461aa into george/plat-143/2-on-success-workspace-build-request-handling Jul 6, 2026
20 of 23 checks passed
@geokat
geokat deleted the george/plat-143/3-process-on-success-workspace-build-orchestration branch July 6, 2026 20:45
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 6, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants