ci: split db-backup into preflight/dump/release jobs and switch to zstd -19 --long=31#561
Merged
Conversation
…td -19 --long=31
Two changes, bundled:
1. Split the single mega-job into three phases so failures are legible and
no job holds both powers:
- preflight (ubuntu-latest): computes the dump filename and release tag
ONCE as job outputs (per-job date stamping could straddle midnight UTC
and split a run across two dates); fails fast if the tag already exists.
- dump (blacksmith-32vcpu): holds only the DB credential, no repo write.
Keeps pg_dump | zstd | split as one streaming step -- a job boundary
between dump and compress would force uploading and re-downloading the
full uncompressed database (tens of GB) as an artifact. Adds a
post-compress `zstd -t` integrity pass, a SHA256SUMS, and a forensics
header (nproc/free/df).
- release (ubuntu-latest): holds only contents:write, no DB credential.
Verifies checksums, creates the release with SHA256SUMS attached.
2. Fix the failing weekly run (#29166233752) and pick the compressor from a
benchmark on a real 3 GiB dump sample (ratios conservative -- the sample
loses cross-region matches the contiguous stream has):
zstd -12 --long=27 24.4x @ 377 MiB/s (too weak)
zstd -19 --long=31 36.7x @ 27 MiB/s <- chosen
xz -9e dict=192MiB 35.8x @ 5 MiB/s (old prod: worse ratio, 5x slower)
zstd -22 --ultra 38.9x @ 8 MiB/s (+6% only, ~130 min -> times out)
The old `xz -9e --dict=192MiB` ran near single-threaded (~5 MiB/s: 9e
can't fill its 576 MiB blocks fast enough to parallelize on a pipe), so
the dump ran for hours and the job was cancelled/timed out. zstd -19
--long=31 beats it on ratio, runs ~5x faster (~40 min on ~60 GB, well
under the 120 min timeout), and uses a bounded window (no runaway memory).
Restore docs updated to `zstd -d --long=31`.
A failed release is now re-runnable without repeating the dump.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The 32-vCPU/128-GB runner was sized for xz -9e's memory/thread appetite. zstd -19 --long=31 uses a bounded ~2 GiB window (a few GiB total across threads) and the pipeline is likely source-bound (pg_dump over the network), so the monster runner is unnecessary. Drop to blacksmith-8vcpu; the forensics header (nproc/free/df) and zstd -v rate on the first real run confirm sizing and timeout headroom before downsizing further.
The smoke test (#29176304089) OOM-killed zstd on the 8 vCPU/30 GiB runner when the large eval_samples table started streaming: broken pipe, exit 143, ~8.5 min in (not the timeout). Root cause: zstd's multithreaded memory scales with windowSize x thread count, and --long=31's 2 GiB window across 8 -T0 workers exceeded 30 GiB at steady state. The ratio benchmark used a 3 GiB sample that never reached that high-water, so it measured ratio correctly but missed the memory cost. --long=27 (128 MiB window) keeps memory to a few GiB regardless of thread count and still compresses ~32.4x (vs 36.7x for --long=31, 24.4x for the old -12), while decompressing with plain `zstd -d` (<=27 is within zstd's default decompressor limit). Reclaiming --long=31's ratio later means capping threads or a high-memory runner; documented inline.
…p VM kill Two smoke runs died mid-dump on a big-blob table (eval_samples, benchmark_server_logs) with no OOM line and the VM terminated; --long=27 ruled out zstd as the memory hog. Add a 3s sampler logging total memory + top-RSS processes so the next run's log shows the memory curve and the owning process (likely pg_dump/libpq buffering a giant row). Temporary -- revert once the cause is confirmed.
…expand mem sampler The instrumented run proved the memory hog is blacksmithd (the Blacksmith runner daemon) at ~28 GiB, not the pipeline (pg_dump 75 MiB, zstd 1.5 GiB). The 30 GiB 8-vCPU tier OOM-killed on the daemon; the historical 32-vCPU/ 128 GiB runner had headroom, so this only broke after the downsize. Revert to 32 vCPU. Blacksmith has no workflow toggle to disable the cache (support ticket only), and the ~28 GiB daemon looks like a bug worth reporting. zstd kept at --long=27 to isolate the runner fix (restore --long=31 for the better ratio once green). Sampler expanded: mem+swap, tmpfs usage, top-RSS processes with argv, plus a one-time blacksmithd dump.
…lls any VM) Instrumentation is conclusive: blacksmithd (the Blacksmith runner daemon) grows to consume the whole VM regardless of size -- ~28 GiB on a 30 GiB runner, ~110 GiB on a 115 GiB runner -- then OOMs the box mid-dump. The pipeline is tiny throughout (pg_dump 75 MiB, zstd 1.5 GiB). A bigger Blacksmith VM can't fix it and the cache has no workflow-level disable, so move the dump job to a github-hosted ubuntu-latest (no blacksmithd; the <2 GiB job fits 16 GiB). timeout 120 -> 180 for the 4-core compress on a growing DB. Instrumentation kept until this passes green.
Diagnosis complete (blacksmithd was the OOM cause; fixed by moving to ubuntu-latest). Drop the forensics step and the per-3s memory sampler; keep a trimmed comment recording why the job runs on a GitHub-hosted runner and stays at zstd --long=27.
Remove the prose comments accumulated during debugging. Keep the inline permission-scope notes (contents: read/write), the two pinned-action version tags, and a shellcheck disable directive (which also clears actionlint's last SC2086 warning). Rationale lives in git history.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9d4cf6e. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Splits the weekly
Database Dumpworkflow from one mega-job into three phases, and fixes the failing weekly run (#29166233752) by replacing the xz compressor with a benchmark-chosen zstd setting.1. Job split —
preflight→dump→releaseubuntu-latest): computes the dump filename and release tag once as job outputs — with per-job date stamping, a run straddling midnight UTC would dump under one date and release under another — and fails fast if the release tag already exists.blacksmith-32vcpu): holds only the DB credential, no repo write. Keepspg_dump | zstd | splitas one streaming step — a job boundary between dump and compress would force uploading and re-downloading the full uncompressed database (tens of GB) as an artifact. Adds a post-compresszstd -tintegrity pass,SHA256SUMS, and annproc/free/dfforensics header.ubuntu-latest): holds onlycontents:write, no DB credential. Verifies checksums and creates the release withSHA256SUMSattached.No single job holds both the database credential and repo write, each phase fails visibly on its own, and a failed release is re-runnable without repeating the dump.
2. Compressor fix — xz → zstd
The previous
xz -T0 -9e --dict=192MiB --memlimit-compress=99%compressed at only ~5 MiB/s — near single-threaded, because9ecan't fill its 576 MiB blocks fast enough to parallelize on a pipe — so a multi-GB dump ran for hours and the job was cancelled / timed out.Setting chosen from a benchmark on a real 3 GiB dump sample (ratios are conservative — the composite sample loses cross-region matches the contiguous stream has):
zstd -12 --long=27zstd -19 --long=31xz -9e dict=192MiB(old prod)zstd -22 --ultra --long=31zstd -19 --long=31beats the old xz ratio, runs ~5× faster (~40 min on ~60 GB, well under the 120-min timeout), uses a bounded window (a few GiB across all threads — no runaway memory), and-T0costs no ratio versus single-thread on this data. Restore docs updated tozstd -d --long=31.Test plan
actionlintclean (only the pre-existingblacksmith-*runner-label note and the intentional unquoted release glob).workflow_dispatchon this branch before relying on the Monday schedule.Note
Medium Risk
Changes production backup timing, compression format, and credential/permission boundaries; a bad dump or release step could block weekly restores until fixed, but scope is CI-only with integrity checks.
Overview
Refactors the weekly Database Dump workflow from one job into preflight → dump → release, so dump naming/tags are computed once, duplicate release tags fail early, and DB credentials (
DATABASE_WRITE_URL) are only on the dump job while release creation stays on a job withcontents: write.The dump step still streams
pg_dump | compress | split, but compression switches from xz tozstd -T0 -19 --long=27, with filenames/tags using a.dump.zstsuffix. It adds post-compresszstd -t,SHA256SUMS, and a short-lived artifact handoff to release, which re-verifies checksums and publishes parts plusSHA256SUMSwith updated restore instructions (zstd -d --long=27).Reviewed by Cursor Bugbot for commit c9ed7cb. Bugbot is set up for automated code reviews on this repo. Configure here.