Skip to content

ci(.github): time out stalled Go module downloads - #30889

Open
ethanndickson wants to merge 3 commits into
mainfrom
ethan/codagt-878-go-mod-download-timeout
Open

ethanndickson wants to merge 3 commits into
mainfrom
ethan/codagt-878-go-mod-download-timeout

Conversation

@ethanndickson

Copy link
Copy Markdown
Member

Closes CODAGT-878

When proxy.golang.org is having a bad day, a module download can stall for minutes before the proxy resets the stream with INTERNAL_ERROR. go has no read timeout of its own, so retry.sh only gets to retry once the proxy gives up. In this test-e2e run the second attempt sat there for 10 minutes, and the job hit its 20 minute timeout during Playwright, even though the third attempt passed straight away.

To fix, retry.sh now takes a --timeout, and kills any attempt that runs longer than that. Killing go mod download is safe, and the next attempt picks up where the last one left off, as finished zips are already in the module cache. The timer is in bash as we can't rely on GNU timeout on the macOS runners, and a step timeout-minutes would kill the whole retry loop (it isn't supported in composite actions anyway).

I've gone with 120s per attempt. Across the last 32 runs on main, a healthy download took at most 41s on any platform, with macOS the slowest at 26 to 41s. During the incident on 2026-10-14, attempts that failed on their own took between 66s and 10 minutes, and three passed slowly at 113s, 133s and 151s. Those last two would now get killed, but that only costs a retry, which just needs whatever's left. Three attempts at 120s plus backoff comes to about 6 minutes, which still fits in test-e2e, whose slowest green run takes about 11 of its 20 minutes.

Separately, make fmt runs a bare go mod tidy, which needs modules that go mod download doesn't fetch (test dependencies of dependencies). So it was downloading 166 modules without any retries, and failing here. The fmt job now runs go mod tidy through retry.sh instead of go mod download.

proxy.golang.org can stall a module download for minutes before it
resets the stream, and go has no read timeout. Give retry.sh a
per-attempt --timeout and use 120s for every go mod download.

make fmt runs go mod tidy, which needs modules that go mod download
does not fetch, so the fmt job now runs a retried go mod tidy instead.
@linear-code

linear-code Bot commented Oct 16, 2026

Copy link
Copy Markdown

CODAGT-878

@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-16T06:21:00.044627Z 64bb4b2 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ce55ea33af

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/retry.sh Outdated
@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 64bb4b2ed7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ethanndickson
ethanndickson requested a review from BobbyHo October 16, 2026 06:18
@ethanndickson
ethanndickson marked this pull request as ready for review October 16, 2026 06:18

@BobbyHo BobbyHo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants