Repository navigation
ci(.github): time out stalled Go module downloads - #30889
ethanndickson wants to merge 3 commits into
Conversation
proxy.golang.org can stall a module download for minutes before it resets the stream, and go has no read timeout. Give retry.sh a per-attempt --timeout and use 120s for every go mod download. make fmt runs go mod tidy, which needs modules that go mod download does not fetch, so the fmt job now runs a retried go mod tidy instead.
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ce55ea33af
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Closes CODAGT-878
When proxy.golang.org is having a bad day, a module download can stall for minutes before the proxy resets the stream with
INTERNAL_ERROR.gohas no read timeout of its own, soretry.shonly gets to retry once the proxy gives up. In thistest-e2erun the second attempt sat there for 10 minutes, and the job hit its 20 minute timeout during Playwright, even though the third attempt passed straight away.To fix,
retry.shnow takes a--timeout, and kills any attempt that runs longer than that. Killinggo mod downloadis safe, and the next attempt picks up where the last one left off, as finished zips are already in the module cache. The timer is in bash as we can't rely on GNUtimeouton the macOS runners, and a steptimeout-minuteswould kill the whole retry loop (it isn't supported in composite actions anyway).I've gone with 120s per attempt. Across the last 32 runs on main, a healthy download took at most 41s on any platform, with macOS the slowest at 26 to 41s. During the incident on 2026-10-14, attempts that failed on their own took between 66s and 10 minutes, and three passed slowly at 113s, 133s and 151s. Those last two would now get killed, but that only costs a retry, which just needs whatever's left. Three attempts at 120s plus backoff comes to about 6 minutes, which still fits in
test-e2e, whose slowest green run takes about 11 of its 20 minutes.Separately,
make fmtruns a barego mod tidy, which needs modules thatgo mod downloaddoesn't fetch (test dependencies of dependencies). So it was downloading 166 modules without any retries, and failing here. Thefmtjob now runsgo mod tidythroughretry.shinstead ofgo mod download.