Add a weekly newest-dependencies canary - #3316
Closed
maxisbey wants to merge 5 commits into
Closed
Conversation
Users installing `mcp` get the newest release of every dependency the day it ships, but PR CI only tests uv.lock and the floors, so an upstream release that breaks the SDK is currently noticed by users first (median 11 days across past incidents). Running "highest" on every PR was tried and removed (#1869) because a half-uploaded release turns unrelated PRs red; the weekly lock-bump PR that replaced it (#1874) never got merged. This adds a scheduled workflow instead. Every Monday it re-resolves the runtime closure of mcp[cli,rich] to the newest versions the specifiers allow (test tooling stays at uv.lock, releases younger than a day are ignored), runs the suite on ubuntu 3.10/3.14 and windows 3.14, and keeps a single tracking issue in sync: opened and assigned when newest-allowed breaks, refreshed while it stays broken, closed once it passes again. Each cell re-runs failures serially and once more with deprecation warnings demoted, so the issue says whether users are actually broken or a dependency merely deprecated something. The report lists what changed since the last green run, what is held below latest by someone else's cap, and the exact command to reproduce the resolution. It never runs on pull requests and never opens a PR adding a ceiling; the issue carries the runbook and a maintainer decides. No-Verification-Needed: CI-only change (workflow + scripts/ci); scripts exercised locally, workflow verified by a branch run
The canary relies on `--exclude-newer` leaving locked versions in place for packages it does not explicitly upgrade, which uv only guarantees from 0.10 (astral-sh/uv#17721), and on relative cutoffs. Move every workflow to the same pin so the canary and PR CI cannot disagree about resolution semantics. 0.10/0.11 carry no breaking changes that touch how this repo uses uv (frozen syncs, lowest-direct resolution, lock --check). No-Verification-Needed: CI configuration only; exercised by PR CI itself
The workflow ran green end to end on the branch (run 31947729778), so remove the push trigger that exercised it. Also read pyright's error count rather than its last output line for the informational summary, and silence its new-version nag. No-Verification-Needed: CI-only change (workflow trigger + report cosmetics)
Review feedback, taken as one structural change rather than patches: the report classified whatever artifacts happened to arrive, so a cell that timed out or lost its runner simply vanished (two passing cells read as green and would have closed a live incident), a cancelled run could still write to the issue, and any resolve-job failure was reported as "cannot be resolved". Now the matrix is defined once and published by the resolve job, the report checks every planned cell against `needs.test.result`, a cell writes a provisional status before doing anything, `uv lock` records whether it was the thing that failed, and incomplete runs are their own class that comments on an open incident instead of rewriting or closing it. Artifacts carry their own top-level directory and are merged on download, so the layout no longer depends on how many of them exist. Smaller corrections from the same review: per-package cutoffs instead of a global --exclude-newer (a freshly bumped exact pin elsewhere could otherwise fail the resolution); the since-last-green diff uses the lock the last green run actually uploaded rather than re-resolving today's tree at an old cutoff; the issue's reproduce block is the literal commands the job ran (group strip included) under `uvx uv@<version>`; per-cell output is size-bounded at the source and the full report goes to the step summary before any truncation; the flake/deprecation re-runs key on whether pytest recorded failures rather than on its exit code; titles say "nothing changed since last green" when that is the case; bash runs with pipefail everywhere; the group-strip generator follows include-group and default-groups = "all"; wording says tooling is preferred at uv.lock, not frozen. The temporary branch trigger is back for one more end-to-end run. No-Verification-Needed: CI-only change (workflow + scripts/ci); scripts exercised locally, workflow verified by a branch run
Second end-to-end branch run (31949021363) is green with the revised layout. No-Verification-Needed: CI-only change (workflow trigger)
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a scheduled "dependency canary": once a week, re-resolve the runtime dependencies of
mcp[cli,rich]to the newest versions our specifiers allow, run the test suite against them, and keep a single tracking issue in sync with the result. Also moves CI to uv 0.11.33, which the canary's resolution step relies on.Motivation and Context
pyproject.tomldeliberately carries floors only, so anyone runningpip install mcpgets the newest release of every dependency the day it ships. PR CI never sees that combination: it testsuv.lock(locked) and the floors (lowest-direct). Looking back over the repo's history there have been roughly six "a new upstream release broke the SDK" events a year (click 8.2, pydantic 2.11/2.12/2.13, starlette 0.52 typing, httpx 1.0 pre-releases, pytest 9.1, pyjwt 2.11, ...), with a median of 11 days before anyone here noticed and a third of them reported by users first.We have tried two other shapes already. A real
highestleg on every PR (#1609) was removed in #1869 after a half-uploaded ruff release turned every open PR red — an upstream release should not be able to do that. The weeklyuv lock --upgradePR that replaced it (#1874) produced four PRs, none merged (CI did not even trigger on the bot's pushes), and was deleted in #2919 in favour of Dependabot, whose PRs tend to get closed in bulk. The common thread is that a PR nobody particularly wants to merge is not an alert.What demonstrably works elsewhere is a scheduled run that files one deduplicated issue and closes it again when things go green (FastMCP's upgrade checks close each incident within days; xarray, dask and scikit-learn run the same pattern). Scheduled runs that only go red, or only ping Slack, rot: pydantic's
dependencies-checkworkflow has failed 100+ runs in a row unnoticed, and logfire's weekly job sat red for 16 weeks.What the workflow does
.github/workflows/dependency-canary.yml, Mondays 05:23 UTC plusworkflow_dispatch:mcp[cli,rich]from the lock (39 packages today), strip the dependency groupsuv syncdoes not install (translate,codegen— otherwiseanthropicwould hold pydantic below a new major forever), anduv lock -P <each> --exclude-newer-package <each>=<now − 24h>. Everything outside the closure keeps its locked version as a preference (it moves only if a floated dependency forces it, and the report tags such rows tooling), so a pytest or ruff release cannot masquerade as an SDK break; releases younger than a day are invisible, which makes the ci: replace highest resolution with locked in test matrix #1869 partial-upload class inexpressible. It also downloads theuv.lockthe last green scheduled run uploaded and diffs against it, so the report can show only what moved since then (usually one to three packages — that table is the suspect list). The resolveduv.lock, the literal commands used, and the logs are uploaded as an artifact; the lock is the exact repro.pyright src/mcp), windows 3.14:uv sync --frozenfrom that lock, run pytest, re-run failures serially (flakes drop out), and if they persist re-run once more withDeprecationWarning/PendingDeprecationWarning/FutureWarningdemoted. Each cell ends up aspass,flaky,warnings-only,errororinstall-failed.uv.lock, the literal commands to reproduce underuvx uv@<same version>, a short runbook) and sync the tracking issue: open it (labelsdependency-canary+dependencies, assigned to @maxisbey and @Kludex,P0only when every cell hard-fails) / overwrite the body and leave a one-line comment while it stays red / comment and close when green. Runs where the canary itself had a problem (a cell without a result,uv lockfailing for non-resolution reasons, a cancelled test job) are a separate incomplete class that only comments on an open incident and never rewrites or closes it. A human-closed issue is never reopened; the next red run opens a fresh one linking the previous. Only scheduled runs (or a dispatch withfile-issue: true) touch issues;prerelease: truedispatches are investigative and never do.Everything is
ghCLI plus two small scripts underscripts/ci/; no third-party actions,permissions: {}at the top withissues: writeonly on the report job, which installs nothing.What it deliberately does not do
maindaily, so a scheduled pre-release lane is left as a possible follow-up rather than day-one noise.workflow_dispatchwithprerelease: truecovers ad-hoc checks.uv.lock. That is a separate concern (and a separate PR); the canary reports against whatever the lock is.uv 0.11.33
Combining
-P <closure>with an upload-time cutoff only keeps the non-upgraded packages at their locked versions from uv 0.10 (astral-sh/uv#17721); on 0.9.5 the cutoff would silently float everything. Rather than run the canary on a different uv than CI, the second commit moves every workflow to 0.11.33 (0.10 and 0.11 have no breaking changes touching frozen syncs,lowest-direct, orlock --check).How Has This Been Tested?
uv lockfailure; resolve-job infrastructure failure; cancelled test job; nothing changed since last green).pushtrigger (removed again in the last commit): run 31947729778 and, after the review revisions, run 31949021363. The resolve job floated exactly the runtime closure to current PyPI (starlette 1.6.0, uvicorn 0.52.3, pydantic 2.13.4, pyjwt 2.13.0, rich 15, typer 0.27.1, cryptography 50, ...) while pytest/ruff/pyright/coverage stayed at their locked versions, all three cells passed (5600-odd tests each, plus the examples smoke tests on ubuntu 3.14), and the report job rendered the would-be issue body into the run summary without touching issues.zizmor --persona regularandactionlintare clean on the new workflow.Newest-allowed is green today, so merging this should not open an issue on the first Monday.
Breaking Changes
None; CI only.
Types of changes
Checklist
Additional context
cron:line also gets GitHub's own failure e-mail for red scheduled runs; everyone else only sees the issue.translate/codegenout of the project resolution (PEP 723 script metadata) so the runner no longer has to strip them; a scheduled--prerelease-package <direct deps>=allowlane reporting to a separate, never-P0 issue; pointingDEPENDENCY_POLICY.md(Publish versioning, roadmap, and dependency policies for v2 #3215) at the issue runbook.AI Disclaimer