Repository navigation
ci(.github/workflows): report doc-check reviews that outlast the wait as running - #31707
nickvigilante wants to merge 4 commits into
Conversation
… as running Since doc-check was turned back on, 28 of 45 runs that started a review failed, every one because the job stopped waiting for the review chat after 40 minutes. The chat keeps running in Coder and posts its own results, so the red check reported a review that was still working. The chat step now may fail without failing the job, and a final step fails the job unless the failure is a wait timeout on a chat that is still running. The wait stays, so a review that finishes in time still stops its workspace, and the per-PR concurrency group still caps each PR at one running and one queued review. Refs DOCS-1478
|
CI note: Generated by Coder Agents on behalf of @nickvigilante. |
|
/coder-agents-review |
|
Chat: Review posted | View chat Review historydeep-review v0.13.0 | Round 4 | Last posted: Round 4, 17 findings (3 P2, 5 P3, 8 P4, 1 Nit), APPROVE. Review Finding inventoryFindings
Contested and acknowledgedRound logRound 1Netero first pass: 2 P4, 2 Note, no P0, so the panel ran. Law not run (effective +42).
Round 2Churn guard PROCEED: CRF-1, 2, 5 to 14 addressed in 1025355 (author claims; panel verified at head). Netero: no findings.
Round 3Churn guard PROCEED: CRF-26, 27, 28 fixed in 43a29e6; CRF-29 fixed by a PR description edit (no thread reply). Netero: no findings; actionlint exit 0; zizmor could not run (GLIBC_2.39 missing).
Round 4Churn guard PROCEED: CRF-38 fixed in d7f1ec2 (one comment sentence). Netero: no findings; actionlint exit 0. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
This PR makes a doc-check wait timeout on a running chat pass the job instead of failing it. Findings: 3 P2, 4 P3, 4 P4, 1 Nit.
Out of scope (needs a ticket or explicit acceptance by a human):
.github/workflows/doc-check.yaml:478: the action setsworkspace-idonly on success (setFailureOutputsinsrc/outputs.tsat the pinned SHA omits it), so the stop step never runs for a timed-out or errored chat, and those workspaces stay up until thecoder-workflow-bottemplate's 24-hour autostop..github/workflows/doc-check.yaml:482: the stop step's status list includescompleted, which the server no longer returns (migration 000543), so that entry never matches.coder/agents-chat-actionsrc/codersdk.gen.ts:100:ChatStatusSchemahas nointerrupting, so polling an interrupting chat fails schema parsing, and three such polls end the step asapi_error, which fails the job..github/workflows/doc-check.yaml:56: after a timed-out run ends, the merge-timedelete-workspacejob can delete the workspace while the chat is still working in it, because the concurrency group no longer covers the chat..github/workflows/triage-via-chat-api.yaml:296: the step summary prints a chat URL built fromsecrets.TRAIAGE_CODER_URL, so the URL renders masked.
Notes:
- A timeout on a chat in
pausedorrequires_actionfails the job. That is correct forrequires_action;pausedneeds a human edit on the bot's chat, so it should not occur in normal runs. - The new step has no automated test, and the repository has no harness for workflow scripts; the PR's bash table is the only coverage.
- The action reuses the PR's chat, so after a timed-out run ends the next push queues a follow-up on the still-running chat. The wait limits this to one queued message per 40 minutes per PR; it does not prevent it.
- The decision log's statement that doc-check is not a required check could not be checked here (branch rules API returned 401).
🤖 This review was automatically generated with Coder Agents.
…step Rename the step to "Fail unless the review is still running" and state its single pass case in the comments. Drop the pending status, which the server no longer returns. Name the chat by ID, because the chat URL is built from a secret and prints masked. Stop repeating the action's error message, and print the error kind, chat status, and chat ID instead. Say that the review posts only when it has findings, and that the workspace stays up until its autostop deadline. Restore the wait's coupling to timeout-minutes in its comment. Refs DOCS-1478
|
/coder-agents-review |
There was a problem hiding this comment.
Re-review of the round 1 fixes in 1025355. Every round 1 finding is fixed at this head, and the gate script gives the exit codes in the testing table. Findings: 1 P3, 3 P4.
Out of scope (needs a ticket or explicit acceptance by a human):
- doc-check review chats have no upper bound on runtime: a review that loops keeps its workspace and a share of the shared model for hours, and later runs queue messages behind it, so the PR gets no review in that time.
.claude/skills/doc-check/output.md:127: the skill says a later step reads thedoc-check-traceblock to set the commit status, but no step indoc-check.yamlreads it, so a review that ends withoutcome: errorafter the wait never changes the check.coder/agents-chat-actionsrc/index.ts:49: the action callscore.setFailedon a timeout on arunningchat and has no input to report that case as a non-failure, so each caller needscontinue-on-errorplus a gate script, and the run page keeps a failure annotation on a green job.
Notes:
- The gate passes any chat that is
runningat 40 minutes, so a stuck review now shows a green check and the still-running notice.chat-statuscannot tell a stuck review from a slow one, and the red check before this PR fired for both.
codersdk/chats.go:131
P4 [CRF-29] The PR description's testing table says it covers each status the server can return, but it omits paused and lists interrupting, which the action cannot report on a timeout. (Mafu-san, Leorio)
codersdk/chats.go:131 defines ChatStatusPaused, and the action's schema (src/codersdk.gen.ts:100-108 at the pinned SHA) accepts it and does not treat it as terminal, so a timeout on a paused chat reaches the gate step; the script exits 1 for it. The same schema has no interrupting, so an interrupting chat ends as api_error, not as timeout/interrupting. Add paused to the timeout exit-1 row and drop interrupting, or say that interrupting arrives as api_error.
🤖
🤖 This review was automatically generated with Coder Agents.
Name the chat step, not the chat, in the failure annotation, and label the status as the last one seen, because a polling failure does not mean the chat failed. Give the reason the wait is not longer, and say that a failure after the wait shows only in the chat. Refs DOCS-1478
|
/coder-agents-review |
1 similar comment
|
/coder-agents-review |
There was a problem hiding this comment.
Re-review of the round 2 fixes in 43a29e6 and the PR description edit. Every round 2 finding is fixed at this head. Findings: 1 P4, in text the wait-comment fix added.
🤖 This review was automatically generated with Coder Agents.
…eviews Refs DOCS-1478
|
/coder-agents-review |
There was a problem hiding this comment.
Re-review of d7f1ec2, which rewords the wait comment. The longer-wait sentence is fixed, and no in-scope findings remain open. The out-of-scope items listed in the earlier reviews still need a human to file a ticket or accept them.
🤖 This review was automatically generated with Coder Agents.
|
Closing this. It assumes a review that outlasts the wait still posts its results later, and the production chats show most of them can't: 159 of 244 review turns since Oct 30 hit a GitHub 401, because the review workspace's installation token expires an hour after the workspace starts. Generated by Coder Agents on behalf of @nickvigilante. |
Summary
Since doc-check was turned back on (Oct 30), 28 of the 45 runs that started a review failed.
Every one failed the same way: the job stopped waiting for the review chat after 40 minutes.
The chat keeps running in Coder after the job stops watching, so the red check reported a review that was still working.
This PR reports a review that outlasts the wait as still running, with a notice that names the chat, instead of failing the job.
Linear: DOCS-1478
Changes
continue-on-error: true.It passes in one case only: the action reports a
timeoutand the chat status isrunning.It then writes a notice and a step summary with the chat ID.
The summary says the review posts only if it has findings, and that the review workspace stays up until its autostop deadline.
That covers a chat error, a spend limit, an API failure, missing outputs, and a timeout on a chat that is not
running.wait-timeout-secondsnames both things the wait does and that it must stay belowtimeout-minutes.A review that times out and later ends in an error now shows a green check and posts nothing, so only the chat shows that result.
Before this PR its check was red, but only because of the timeout, the same as a review that later finished clean.
Testing
make lint/actions/actionlintandmake lint/actions/zizmorpass, with no zizmor findings ondoc-check.yaml.I ran the new step's script under bash with each status the action can report on a failure:
chat-error-kindchat-statustimeoutrunningtimeoutwaiting,requires_action,paused, orerrorapi_errorrunningspend_exceedederrorunknownThe action's status schema has no
interrupting, so an interrupting chat arrives asapi_error.I read
coder/agents-chat-actionat the pinned SHA. On a timeout it setschat-error-kind,chat-status, andchat-idbefore it fails the step.It does not set
workspace-id, so the workspace stop step skips a timed-out review, as it did before this PR.The step names the chat by ID because
chat-urlis built from theDOC_CHECK_CODER_URLsecret, and a recent run log prints it as***/agents/<id>.Not run end to end: that needs a real review longer than 40 minutes, which would post to a PR.
Decision log
Failure data (runs since Oct 30 17:54 UTC): 200 runs.
119 skipped every job (draft PRs, labels other than
doc-check, and closed PRs), 34 succeeded (17 of those ran a review), 28 failed, 17 were cancelled, and 2 were still running or awaiting approval.All 28 failures ended with
Timed out after 2400s waiting for chat <id> to reach a terminal status.Of the 10 timed-out chats I could trace, 7 were still running (some after 4.5 hours) and 3 had finished with a clean result after the job went red.
The 17 cancellations are pending runs that GitHub replaced with a newer run for the same PR, and this PR does not change them.
Why not
wait: none: it would end the job in a minute, but two things depend on the wait.The other 28 already leave their workspace up until the
coder-workflow-bottemplate's 24-hour autostop, before and after this PR.With
wait: none, all 45 would, which is the overload ci(.github/workflows): stop doc-check workspaces after each review #29851 fixed.Without the wait, every push would post another message to the PR's chat, and Coder queues messages to a busy chat, so reviews would pile up on the shared model.
After a timeout, the next run can still queue one message on a chat that is still running; the wait limits this to one per 40 minutes per PR.
Why not a longer wait: GitHub-hosted jobs stop at 6 hours, some reviews have run longer than 4.5 hours, and a longer wait holds the concurrency queue longer.
Left as is: an action annotation for the timeout still appears on the run page, because the action calls
core.setFailed.The job and the PR check are green.
The doc-check job is not a required check (
gh pr checks --requiredlists onlyrequired), so a green still-running review doesn't let anything merge that couldn't before.Review rounds (
coder-agents-review): accepted every in-scope finding in rounds 1 and 2.Out-of-scope findings for a follow-up, not changed here:
workspace-idonly on success.completed, which the server no longer returns.interrupting, so polling an interrupting chat ends asapi_error.delete-workspacejob on merge can delete a workspace that a still-running chat uses.triage-via-chat-api.yamlprints a chat URL built from a secret, so it renders masked.output.mdsays a later step reads thedoc-check-traceblock to set the commit status, but no step indoc-check.yamlreads it.runningchat as a non-failure, so each caller needscontinue-on-errorand a gate step.Tradeoff: the gate passes any chat that is
runningat 40 minutes, so a stuck review also shows green.chat-statuscannot tell a stuck review from a slow one; the red check before this PR fired for both.Related: #31649 makes reviews more thorough and slower, so it raises the share of reviews that outlast the wait.
This PR is what keeps those from failing the job.
CI flakes
test-go-pg (windows-2022)failed on the first run with no failing test (DONE 3922 tests, 134 skipped, thenmakeexit 1).This PR changes only a workflow file, and the job passed on rerun.
genfailed ond7f1ec2297before any step ran, with "The self-hosted runner lost communication with the server".It passed on rerun.
Generated by Coder Agents on behalf of @nickvigilante.