Skip to content

chore(ci): retry desktop packaging once after a transient failure - #56073

Open
neriousy wants to merge 1 commit into
desktop-early-startfrom
desktop-package-retry
Open

neriousy wants to merge 1 commit into
desktop-early-startfrom
desktop-package-retry

Conversation

@neriousy

Copy link
Copy Markdown
Member

Stacked on #56072 (desktop-early-start) ("chore(ci): build the desktop app while the CLI builds"). Below that are #56071 (desktop-release-files), #56070 (cli-target-jobs) and #55987 (release-runner-moves). Only the last commit belongs to this PR. It does not depend on the PRs below it: the commit also cherry-picks cleanly onto origin/dev.

Summary

One failed desktop leg (of six) fails a dev release until someone re-runs it. Most leg failures since mid-August were short network or signing errors in Package that a second try would have fixed.

  1. Package retries once (publish.yml, build-electron).
    • Package gets id: package and continue-on-error: true.
    • Clean up failed Package (if steps.package.outcome == 'failure', shell: bash): warns with the target name, deletes only packages/desktop/dist, detaches leftover /Volumes/OpenCode* DMGs on macOS, sleeps 90 s.
    • Package (retry): same if and same command, env, shell and 60-minute timeout, without continue-on-error, so a second failure fails the leg as today.
    • Package (no publish) is unchanged.
  2. script/sign-windows.ps1 retries the TrustedSigning module install up to 3 times (30 s, then 60 s apart), re-registering PSGallery if missing. It only helps if the gallery is back within about 90 s. Invoke-TrustedSigning is not retried.

Expected effect

Release dispatches since 2026-08-15 (67 runs) had 9 failed build-electron legs. Estimates come from step timestamps, not measured runs.

  • 504 on Electron download (job 116597252358, run 38785024495): very likely absorbed; about 49 min sooner (est.), mostly unwatched time.
  • Apple timestamp errors (job 106275288038, run 35580854706): about 14 min sooner (est.) if Apple's server was up at retry time. Job 100912449525 (run 33836944369) was never re-run and would most likely have shipped.
  • electronjs.org timeout (job 105256240999, run 35236268439): plausibly absorbed.
  • PSGallery missing in Windows Prepare (job 109246925614, run 36518254848): about 14 min sooner (est.), only via the script retry and only if the gallery was back within 90 s.
  • Not absorbed: Azure login (two legs), setup-bun, lost runner.
  • Cost: none when Package passes; a real break shows up about 3 to 13 min later.

Evidence

  • steps.package.outcome is failure (also on timeout) while the job stays success. Later steps use only the retry's output; a double or cleanup failure fails the job with nothing stale shipped.
  • Package (retry) matches Package (compared via Bun.YAML). electron-builder copies into dist, so wiping it is safe. GH_TOKEN expiry cannot fail the retry.
  • The cleanup script was tested locally for all three OSes; it has never run on a real runner. sign-windows.ps1 has never been executed; it was reviewed by hand.
  • actionlint, prettier --check and bun run lint:changed are clean.
  • Beta merge simulation (against origin/beta 2fbdff9):
    • This commit on origin/dev alone: 0 conflicts.
    • Whole stack: the parent's same 7 conflict regions in publish.yml, all from lower PRs. The new steps and sign-windows.ps1 merge cleanly.
    • This diff alone on beta: 1 conflict at the comment above Package (a lower PR removes Build); it does not occur in either real merge.

Things to know

  • One retry, for real errors too (afterPack checks, broken config, notarization rejection).
  • How a retry looks. A green job with a red first Package step and the warning "Package failed for <target>; retrying once in 90 seconds".
  • Windows signing token. Package uses the az token cached at Azure login, which cannot refresh. After a normal failure the retry signs 7 to 10 min after login, which already works (job 116597252325). After a 60-minute timeout it most likely fails about 2 h in, same outcome as today but later. Re-running Azure login is left out.
  • More signing calls. A retry re-notarizes on macOS and re-signs everything on Windows.
  • PSGallery. The failure was on a GitHub-hosted runner; Windows legs now run on Blacksmith (chore(ci): build macOS x64 and Windows arm64 desktop apps on faster runners #55987), with no PSGallery failure found there. On beta the install happens inside Package, so warnings are not annotations and the Package retry covers a final failure.
  • Not covered: CLI wait/download, other Prepare failures (including Invoke-TrustedSigning's nuget.org downloads), build-cli and sign-cli-*.
  • Beta. Merging adds no new conflict; beta's uploads read the retry's clean dist, and the glob matches "OpenCode Beta" volumes.

Verification

publish.yml cannot run on a PR. On the first dev release dispatch after merge:

  • All six legs: both new steps skipped; durations match the PR below.
  • If a leg retries: the retry warning; artifact holds only the retry's files; .app.tar.gz (macOS) or signature verification (Windows) passes; publish runs.
  • If a Windows Prepare install retries: a "TrustedSigning module install attempt N failed" annotation, then a normal signing log.

Rollback: revert this commit.

@neriousy
neriousy added this pull request to stack #56077 October 16, 2026 13:20
@neriousy
neriousy force-pushed the desktop-package-retry branch from 64763e6 to 0d6b71f Compare October 16, 2026 16:08
@neriousy
neriousy force-pushed the desktop-package-retry branch from 0d6b71f to 177b3e5 Compare October 16, 2026 16:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant