Skip to content

ci: split db-backup into preflight/dump/release jobs and switch to zstd -19 --long=31#561

Merged
adibarra merged 9 commits into
masterfrom
ci/db-backup-split-jobs
Jul 12, 2026
Merged

ci: split db-backup into preflight/dump/release jobs and switch to zstd -19 --long=31#561
adibarra merged 9 commits into
masterfrom
ci/db-backup-split-jobs

Conversation

@adibarra

@adibarra adibarra commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Splits the weekly Database Dump workflow from one mega-job into three phases, and fixes the failing weekly run (#29166233752) by replacing the xz compressor with a benchmark-chosen zstd setting.

1. Job split — preflightdumprelease

  • preflight (ubuntu-latest): computes the dump filename and release tag once as job outputs — with per-job date stamping, a run straddling midnight UTC would dump under one date and release under another — and fails fast if the release tag already exists.
  • dump (blacksmith-32vcpu): holds only the DB credential, no repo write. Keeps pg_dump | zstd | split as one streaming step — a job boundary between dump and compress would force uploading and re-downloading the full uncompressed database (tens of GB) as an artifact. Adds a post-compress zstd -t integrity pass, SHA256SUMS, and an nproc/free/df forensics header.
  • release (ubuntu-latest): holds only contents:write, no DB credential. Verifies checksums and creates the release with SHA256SUMS attached.

No single job holds both the database credential and repo write, each phase fails visibly on its own, and a failed release is re-runnable without repeating the dump.

2. Compressor fix — xz → zstd

The previous xz -T0 -9e --dict=192MiB --memlimit-compress=99% compressed at only ~5 MiB/s — near single-threaded, because 9e can't fill its 576 MiB blocks fast enough to parallelize on a pipe — so a multi-GB dump ran for hours and the job was cancelled / timed out.

Setting chosen from a benchmark on a real 3 GiB dump sample (ratios are conservative — the composite sample loses cross-region matches the contiguous stream has):

setting ratio speed verdict
zstd -12 --long=27 24.4× 377 MiB/s too weak
zstd -19 --long=31 36.7× 27 MiB/s chosen — the knee
xz -9e dict=192MiB (old prod) 35.8× 5 MiB/s worse ratio and 5× slower
zstd -22 --ultra --long=31 38.9× 8 MiB/s +6% only; ~130 min on the current DB → would time out

zstd -19 --long=31 beats the old xz ratio, runs ~5× faster (~40 min on ~60 GB, well under the 120-min timeout), uses a bounded window (a few GiB across all threads — no runaway memory), and -T0 costs no ratio versus single-thread on this data. Restore docs updated to zstd -d --long=31.

Test plan

  • actionlint clean (only the pre-existing blacksmith-* runner-label note and the intentional unquoted release glob).
  • Compressor settings validated against a real dump: the old-prod xz row reproduced 35.8× on the sample vs the known ~38.5× full-file ratio, confirming the sample is representative.
  • Verify end-to-end via workflow_dispatch on this branch before relying on the Monday schedule.

Note

Medium Risk
Changes production backup timing, compression format, and credential/permission boundaries; a bad dump or release step could block weekly restores until fixed, but scope is CI-only with integrity checks.

Overview
Refactors the weekly Database Dump workflow from one job into preflight → dump → release, so dump naming/tags are computed once, duplicate release tags fail early, and DB credentials (DATABASE_WRITE_URL) are only on the dump job while release creation stays on a job with contents: write.

The dump step still streams pg_dump | compress | split, but compression switches from xz to zstd -T0 -19 --long=27, with filenames/tags using a .dump.zst suffix. It adds post-compress zstd -t, SHA256SUMS, and a short-lived artifact handoff to release, which re-verifies checksums and publishes parts plus SHA256SUMS with updated restore instructions (zstd -d --long=27).

Reviewed by Cursor Bugbot for commit c9ed7cb. Bugbot is set up for automated code reviews on this repo. Configure here.

…td -19 --long=31

Two changes, bundled:

1. Split the single mega-job into three phases so failures are legible and
   no job holds both powers:
   - preflight (ubuntu-latest): computes the dump filename and release tag
     ONCE as job outputs (per-job date stamping could straddle midnight UTC
     and split a run across two dates); fails fast if the tag already exists.
   - dump (blacksmith-32vcpu): holds only the DB credential, no repo write.
     Keeps pg_dump | zstd | split as one streaming step -- a job boundary
     between dump and compress would force uploading and re-downloading the
     full uncompressed database (tens of GB) as an artifact. Adds a
     post-compress `zstd -t` integrity pass, a SHA256SUMS, and a forensics
     header (nproc/free/df).
   - release (ubuntu-latest): holds only contents:write, no DB credential.
     Verifies checksums, creates the release with SHA256SUMS attached.

2. Fix the failing weekly run (#29166233752) and pick the compressor from a
   benchmark on a real 3 GiB dump sample (ratios conservative -- the sample
   loses cross-region matches the contiguous stream has):
     zstd -12 --long=27  24.4x @ 377 MiB/s   (too weak)
     zstd -19 --long=31  36.7x @  27 MiB/s   <- chosen
     xz -9e dict=192MiB  35.8x @   5 MiB/s   (old prod: worse ratio, 5x slower)
     zstd -22 --ultra    38.9x @   8 MiB/s   (+6% only, ~130 min -> times out)
   The old `xz -9e --dict=192MiB` ran near single-threaded (~5 MiB/s: 9e
   can't fill its 576 MiB blocks fast enough to parallelize on a pipe), so
   the dump ran for hours and the job was cancelled/timed out. zstd -19
   --long=31 beats it on ratio, runs ~5x faster (~40 min on ~60 GB, well
   under the 120 min timeout), and uses a bounded window (no runaway memory).
   Restore docs updated to `zstd -d --long=31`.

A failed release is now re-runnable without repeating the dump.
@vercel

vercel Bot commented Jul 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
inferencemax-app Ready Ready Preview, Comment Jul 12, 2026 5:37am

Request Review

Comment thread .github/workflows/db-backup.yml Outdated
@adibarra adibarra changed the title ci: split db-backup into preflight/dump/release jobs and switch to zstd -19 --long=31 / 将 db-backup 拆分为 preflight/dump/release 任务并切换到 zstd -19 --long=31 ci: split db-backup into preflight/dump/release jobs and switch to zstd -19 --long=31 Jul 12, 2026
The 32-vCPU/128-GB runner was sized for xz -9e's memory/thread appetite.
zstd -19 --long=31 uses a bounded ~2 GiB window (a few GiB total across
threads) and the pipeline is likely source-bound (pg_dump over the
network), so the monster runner is unnecessary. Drop to blacksmith-8vcpu;
the forensics header (nproc/free/df) and zstd -v rate on the first real
run confirm sizing and timeout headroom before downsizing further.
The smoke test (#29176304089) OOM-killed zstd on the 8 vCPU/30 GiB runner
when the large eval_samples table started streaming: broken pipe, exit 143,
~8.5 min in (not the timeout). Root cause: zstd's multithreaded memory
scales with windowSize x thread count, and --long=31's 2 GiB window across
8 -T0 workers exceeded 30 GiB at steady state. The ratio benchmark used a
3 GiB sample that never reached that high-water, so it measured ratio
correctly but missed the memory cost.

--long=27 (128 MiB window) keeps memory to a few GiB regardless of thread
count and still compresses ~32.4x (vs 36.7x for --long=31, 24.4x for the
old -12), while decompressing with plain `zstd -d` (<=27 is within zstd's
default decompressor limit). Reclaiming --long=31's ratio later means
capping threads or a high-memory runner; documented inline.
…p VM kill

Two smoke runs died mid-dump on a big-blob table (eval_samples,
benchmark_server_logs) with no OOM line and the VM terminated; --long=27
ruled out zstd as the memory hog. Add a 3s sampler logging total memory +
top-RSS processes so the next run's log shows the memory curve and the
owning process (likely pg_dump/libpq buffering a giant row). Temporary --
revert once the cause is confirmed.
…expand mem sampler

The instrumented run proved the memory hog is blacksmithd (the Blacksmith
runner daemon) at ~28 GiB, not the pipeline (pg_dump 75 MiB, zstd 1.5 GiB).
The 30 GiB 8-vCPU tier OOM-killed on the daemon; the historical 32-vCPU/
128 GiB runner had headroom, so this only broke after the downsize. Revert
to 32 vCPU. Blacksmith has no workflow toggle to disable the cache (support
ticket only), and the ~28 GiB daemon looks like a bug worth reporting.

zstd kept at --long=27 to isolate the runner fix (restore --long=31 for the
better ratio once green). Sampler expanded: mem+swap, tmpfs usage, top-RSS
processes with argv, plus a one-time blacksmithd dump.
…lls any VM)

Instrumentation is conclusive: blacksmithd (the Blacksmith runner daemon)
grows to consume the whole VM regardless of size -- ~28 GiB on a 30 GiB
runner, ~110 GiB on a 115 GiB runner -- then OOMs the box mid-dump. The
pipeline is tiny throughout (pg_dump 75 MiB, zstd 1.5 GiB). A bigger
Blacksmith VM can't fix it and the cache has no workflow-level disable, so
move the dump job to a github-hosted ubuntu-latest (no blacksmithd; the
<2 GiB job fits 16 GiB). timeout 120 -> 180 for the 4-core compress on a
growing DB. Instrumentation kept until this passes green.
Diagnosis complete (blacksmithd was the OOM cause; fixed by moving to
ubuntu-latest). Drop the forensics step and the per-3s memory sampler;
keep a trimmed comment recording why the job runs on a GitHub-hosted
runner and stays at zstd --long=27.
Remove the prose comments accumulated during debugging. Keep the inline
permission-scope notes (contents: read/write), the two pinned-action
version tags, and a shellcheck disable directive (which also clears
actionlint's last SC2086 warning). Rationale lives in git history.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9d4cf6e. Configure here.

Comment thread .github/workflows/db-backup.yml
Comment thread .github/workflows/db-backup.yml
@adibarra adibarra merged commit 7780426 into master Jul 12, 2026
14 checks passed
@adibarra adibarra deleted the ci/db-backup-split-jobs branch July 12, 2026 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant