Structure-review batch: sweep fixes, module taxonomy, npm-family dedup, coverage gaps, CI honesty - #150
Conversation
Bug fixes - scan --prune now exempts manifest entries whose ecosystem this run never crawled: an unknown `pkg:<type>/` (a newer CLI's ecosystem in a committed, shared manifest) and the runtime-gated maven/nuget crawlers with their gate off. Absence from a crawl that never looked is not evidence of removal, so pruning them silently deleted a teammate's patch plus its blobs. - get: thread --api-token/--api-url/--org/--proxy-url into the nested apply. They were dropped, so a token supplied purely as a flag fell through to the token-less public proxy — against the wrong host, with the wrong client. - redirect: write the managed `[registries.…]` block into the legacy extensionless `.cargo/config` when a project carries it. Cargo reads that spelling in preference to `config.toml`, so the block was landing in a file cargo ignores while the run still reported the dep redirected. - rollback: fall back to dropping the go.mod `replace` + `.socket/go-patches/` copy for a local-mode go patch whose module the crawler cannot find. - crawlers: composer reads `installed.json` through `open_regular_file` (a planted FIFO no longer hangs scan/apply forever); maven treats an empty MAVEN_REPO_LOCAL/M2_HOME as unset; nuget matches `nuget.config` / `packages.config` case-insensitively; python scans the macOS `osx_framework_user` user-install root; ruby's local-mode fallback returns every gem home `gem env` reports (not just `gemdir`) and accepts the alternate `gems.rb`/`gems.locked` Bundler spelling. - setup: `--exclude` trims CSV whitespace and covers the excluded directory's whole subtree; `finalize_gem` forwards an absolutized --manifest-path. - update: non-fatal advisories ride the envelope's run-level `warnings[]`, so a `--json` run no longer silently swallows a managed-install override. - vex: git-config parsing matches git itself (case-insensitive names, BOM tolerance, whitespace before a quoted subsection). - lib: `--update=<VERSION>` (inline `=` spelling) is recognized, and a `--update` after `--` is correctly left as an escaped operand. Test harness / CI - New [profile.ci-release] (release minus the full-LTO link) for test-release, and dependency opt-levels raised in the dev profile — the self_update fixtures gzip and sha256 a multi-MB binary per test at opt-level 0. Workspace members stay at opt-level 0, so llvm-cov line fidelity and debug experience are unchanged. - CI: per-job timeout-minutes, and a concurrency group that supersedes stale runs while never cancelling main (main is the only rust-cache writer). - The two wall-bound real-package-manager redirect capstones are #[ignore]-gated out of the serial `test` job and relocated to the parallel e2e matrix, which runs `-- --ignored` on all three OSes. Same coverage, off the critical path. - Drop the unused testcontainers dev-dependency (-1001 Cargo.lock lines). - Many new invariant/e2e suites: scan, apply, remove, setup, vendor, vex, crawlers, in-process redirect, CLI config fallback. Known-RED tests, gated with #[ignore] and a reason Each of these is a correct test for a real bug whose production fix is NOT in this change. They are gated rather than deleted so the finding is not lost: apply_lock waiter/orphaned-inode (two simultaneous holders of the "exclusive" apply lock), apply's manifest_unreadable fail-closed arm, the telemetry --api-url/--proxy-url env mirror (on-prem token egress), composer.json setup mode preservation (one-line fix: use atomic_write_bytes_preserving_mode), pnpm `node-linker=pnp` detection, pnpm workspace flow-sequence parsing, apply's cached-package-archive fallback, remove's ledger-generation match, and the yarn-PnP refusal scoping (its counter-guard stays live). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Bugbot Autofix is ON. A cloud agent has been kicked off to fix the reported issues.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit f41b2d6. Configure here.
…treat empty CARGO_HOME as unset
Two more fixes the sweep's own RED regression tests were pinning:
* scan --vendor --json: reconcile_dropped mutates the on-disk ledger
BEFORE staging, but a staging failure returned Err without the
envelope — the JSON consumer saw only the error object and never
learned entries had been reverted on disk. The step error now carries
the envelope built so far and the JSON fold attaches it as `vendor`.
(Human mode prints no per-event lines even on success; unchanged.)
* cargo crawler: CARGO_HOME="" hit PathBuf::from("") and resolved
registry/src against the CWD, silently crawling nothing. Empty now
means unset (env_non_empty convention), falling back to ~/.cargo.
Pinned by scan_vendor_staging_error_still_reports_the_reconcile and
empty_cargo_home_falls_back_to_home_dot_cargo.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pure move, no logic changes. patch/ was 63% of the crate but only ~11% patch engine; the vendoring + lockfile-rewriting subsystem it contained (34 files, ~47% of core) is the crate's real center of mass and now lives at crate::vendor. The npm-family strays move with it (bun_lock_text, go_mod_edit), and the project-local Go replace-redirect backend joins the other rewiring code as patch::redirect::golang_local. Old `patch::*` paths keep compiling through re-export shims for external consumers of the published crate; internal references are repointed in the follow-up commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e paths Mechanical: patch::vendor → vendor, patch::go_mod_edit → vendor::go_mod_edit, patch::bun_lock_text → vendor::bun_lock_text, patch::go_redirect → patch::redirect::golang_local, across both crates and tests. The patch::* re-export shims stay for external consumers of the published core crate, but #[deprecated] on a pub use emits no warnings (rust-lang/rust#30827), so a CI grep now rejects new internal uses of the alias paths. The bun_lock_text shim is dropped outright: it was pub(crate) before the move, so no external consumer could name it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ensure_table / has_table were defined inside the pypi setup backend (pth_hook) but consumed by vendor::cargo_config, vendor::pypi and vendor::pypi_uv — the pypi hook module owned the crate's generic structured-TOML seam. Move both (verbatim) to utils::toml_edit_ext and repoint the five callers. Unblocks folding pth_hook under a future setup/ umbrella without dragging vendor dependencies along. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mechanical moves with compat re-exports left in utils/ for external consumers of the published crate (internal paths repointed, CI grep-guarded like the vendor promotion): * utils/telemetry.rs -> src/telemetry.rs — a 1k-LOC subsystem, not a leaf helper * utils/cleanup_blobs.rs -> manifest/cleanup_blobs.rs — imports manifest::operations/schema; it is manifest-domain blob GC * utils/date.rs -> api/date.rs — parses the API's RFC-2822 wire dates * utils/fuzzy_match.rs -> crawlers/fuzzy_match.rs — depends on crawlers::types utils/ keeps the genuine leaves: fs, env_compat, http, process, purl, serde, socket_cli_config, toml_edit_ext, uri. Also: vendor/state.rs flavor docstring gains the missing yarn-berry (npm_flavor emits and revert-routes it; the doc list had drifted). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four naming schemes for one concept, now one umbrella: gem_setup -> setup::gem, composer_setup -> setup::composer, pth_hook -> setup::pypi, plus a thin setup::npm alias re-exporting package_json's setup-facing surface. package_json itself stays top-level: it doubles as the crate-wide shared npm-manifest library (crawlers and vendor parse package.json through it), which is exactly why it never fit under a setup umbrella wholesale. Pure git-mv moves — the pypi backend's generic TOML helpers were already extracted to utils::toml_edit_ext, so nothing vendor-shaped rides along. Old top-level paths keep compiling through lib.rs aliases for external consumers of the published crate; internal references are repointed and the CI alias-path grep now rejects the three old paths. The shared per-backend Status enum the four modules' docs describe informally is deliberately NOT introduced here: their status semantics differ subtly (gem template regeneration), and unification is a behavior decision, not motion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…drift-guard tests The npm-family package managers spell their shared knowledge across four subsystems, each with its own list — and those lists accept INTENTIONALLY divergent subsets (hosted redirect deliberately omits bun.lockb; the pnpm-lock.yml spelling is setup-detection-only), so a flat shared list cannot serve them. Instead constants::npm_family now holds a structured row table (name x per-consumer role flags) plus the genuinely shared literals, and each consumer keeps its own shape guarded by an equality test against its role: * vendor::npm_flavor: probe families == rows flagged vendor_probe; local PNP_MARKERS deduped onto the shared const (same set the crawler probes — a past-divergence risk, now one definition) * scan::hosted: REDIRECT_CANDIDATE_FILES' npm-family subset == rows flagged redirect_candidate, both directions, so bun.lockb's deliberate absence is pinned as deliberate * package_json::find: detection iterates rows flagged detects_pnpm (behavioral pin per spelling, incl. pnpm-lock.yml) * crawlers::pkg_managers: PnP probe uses the shared PNP_MARKERS * Rush's common/config/rush/pnpm-lock.yaml literal (3 code sites) is now RUSH_COMMON_LOCK_REL Also: apply's package-manager match is exhaustive (a 7th layout must make an explicit appearance instead of falling into the wildcard), and deno.lock's absence from the npm-family lists is recorded as a decision in the table and the hosted candidate list. Not attempted here, deliberately: unifying the three PM enums (they answer different questions), merging redirect's pnpm regex grammar onto vendor's parser (intentionally different version envelopes), and the setup PackageManager Yarn/Bun widening (a product decision on hook commands, its own PR). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The test and test-release jobs ran `cargo test --workspace --all-features`, which also RUNS the feature-gated docker-e2e and setup-e2e suites. Those tests soft-skip as "ok" when no socket-patch-test images exist — which is always true in these jobs (no images are built there; macOS/Windows have no Docker at all). Every OS leg therefore reported dozens of fake green tests, and a broken skip-guard would disable a whole suite while CI stayed green. Split build from run: --all-features --no-run keeps the compile-rot coverage for the gated suites, the run step uses default features only. The dedicated e2e-docker and setup-matrix jobs remain the places where the gated suites actually execute. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…every hosted failure; fetch_stage unit tests Three coverage gaps from the 2026-08-10 structure review, each with the test that pins it: * setup --exclude persistence clobbered a corrupt manifest: persist_setup_excludes flattened a read/parse error to "no manifest yet" and rewrote the file as a bare setup block, destroying every patch record a merely-corrupt manifest still held. Now fails closed (skip persistence, warn on stderr, manifest bytes untouched). RED test: exclude_persistence_fails_closed_on_corrupt_manifest. * scan --redirect --json emitted empty stdout on every failure exit (discovery-detail failure, reference-resolve failure, file/ledger write failure) — exit 1 with nothing to parse. All four bail-outs now emit the machine-readable error envelope (status/error mirror the success envelope's error fold). RED test: redirect_json_mode_failures_emit_error_envelope. * fetch_stage.rs (the offline-guard-critical download planner) had zero direct tests. In-src unit tests now pin: the read-only-.socket contract, in-place staging when fully cached, the diff-archive disk-vs-vendor staging asymmetry both module docs describe, overlay promotion for late downloads, overlay_dir semantics, and the bad --download-mode hard failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ken by the setup/ move
Two CI reds from the consolidated branch's first run, plus doc-path rot:
* e2e_composer's scan tests were designed offline ("the package count is
derived from the local crawl") but implicitly called the LIVE public
proxy — their exit-0 assumption dated from before the
all-batches-failed fix made total API failure exit non-zero, so any
production hiccup (like today's patches-api 503 "over capacity"
incident) failed the test, coverage and test-release jobs on every OS.
Now pinned to an in-test wiremock proxy with the same harness shape as
e2e_nuget/e2e_gem (empty no-patch result, env scrub, spawn_blocking,
request-count hermeticity guard). e2e_embedded_vex audited too: its
scans find zero packages, so no API call fires — left as is.
* lint-ecosystems ruby-checked the gem templates at their pre-move path;
the setup/ umbrella relocated them to src/setup/gem/templates/ and the
Rust-path CI grep could not catch a filesystem path in a workflow.
Also: release.yml's gem_setup comment and CLI_CONTRACT.md's
src/patch/vendor/ references updated to the new module homes (the only
stale non-Rust references a repo-wide sweep found).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
macOS debug builds use unpacked split-debuginfo: every linked binary — including each of the ~90 e2e test executables — pins its per-codegen-unit .o files in target/debug/deps, and cargo never garbage-collects the generations superseded by lockfile/toolchain bumps; this grew a single worktree's target/ to 99 GB. Line tables keep panic backtraces readable while dropping the bulk of the retained DWARF; for a full-fidelity debugger session, override with CARGO_PROFILE_DEV_DEBUG=full. Applies to the test/bench profiles via inheritance. (Authored during the 2026-08-10 disk-space cleanup; folded into this branch at the owner's request.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Two commits appended addressing this PR's CI reds plus the disk-space work:
Remaining known red: |
… status, dead Err plumbing removed — with the pins the review demanded Review follow-ups (2026-08-11 ULTRACODE pass over #150): * detect_updates: the qualifier-stripped fallback did HashMap iter().find(), so qualifier TWINS (one package under two artifact-pinned manifest keys, e.g. a pypi wheel+sdist pair) resolved to a per-process-random record. Now: any stale twin means an update; twins scan in sorted-key order and the first differing one names old_uuid — deterministic across runs. New pins: the qualifier-bridge leg (previously only percent-encoding was tested) and the twin cases (stale twin wins over 16 iterations; all-twins- current stays quiet). * scan --vendor --json: the envelope carried into the staging-error fold is now demoted to partialFailure before the carry — a consumer reading .vendor.status inside a "status":"error" result saw the fresh-envelope default "success". Pinned in scan_vendor_step_error_e2e. * stage_vendor_sources_in_memory returns MemStageOutcome directly: it never constructed Err, so the Result wrapper bred statically-dead stage_failed arms in three callers (scan flow, vendor, repair_vendor) — all removed. * Hosted --json write-failure bail-outs (legs 3-4: unwritable lockfile, directory squatting on the revert-ledger path) get the envelope test the 882cdb7 commit message claimed — driven with real filesystem obstructions, cross-platform via set_readonly. * get nested apply: a token-less --proxy-url-only leg. The two authenticated legs could not catch a dropped proxy_url (the client consults it only on the token-less branch); this leg goes red for exactly that regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fail-closed skip on a corrupt manifest only warned on stderr behind
!silent: a --json consumer saw a fully-successful setup whose excludes
silently evaporate on the next flag-less run, and --silent left no trace
at all. persist_setup_excludes now returns the warning and run_setup
folds it into the run's warnings channel — human summary line and the
--json envelope's warnings array. --silent stays quiet by contract
("errors only") and is now PINNED as a decision: the silent leg asserts
suppression AND that fail-closed still holds byte-identically.
Known edge left as-is: a corrupt manifest in a project with zero hook
files exits through report_no_files before warnings assemble.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…k the pnpm-pin self-reference, truth the table doc * BUN_MIGRATE_CMD had zero call sites while its doc claimed it was spliced into every message — deleted; the four messages keep their literals (the executed bun argv lives separately anyway, so the const single-sourced nothing). * npm_flavor's vendor_rush_unsupported message now formats RUSH_COMMON_LOCK_REL instead of hardcoding the path twice — the third code site the original commit claimed but did not wire. * find.rs gains a hardcoded pnpm-spelling pin: production code and the guard test iterated the identical names_with(detects_pnpm) expression, so a deleted table row shrank both together while .yml detection silently vanished. * The npm_family module doc now states exactly which consumers are guard-tested and which are behaviorally pinned instead (pkg_managers' own lockfile literals, the probe's decision literals) — it previously promised per-consumer guards it did not have. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pt removal preserves quoted && bytes * vex's three track_vex_* sites passed the raw --api-token/--org flags, so a `socket login`-only user attributed vex telemetry anonymously — the exact gap list/setup closed in the sweep (the long-standing "vex.rs telemetry raw-flags twin"). All three now resolve through list::telemetry_credentials (flag / env / socket-cli config.json). * remove_socket_patch_from_script kept survivors via trim+canonical " && " rejoin, which rewrote a && INSIDE a quoted argument of a surviving user command: `grep "a&&b"` came back as `grep "a && b"` — a different pattern, not the "cosmetic" respacing the docstring claimed. Survivors are now spliced out of the original text verbatim (inner spacing included); only seams next to removed segments collapse. All eleven existing removal pins hold unchanged; two new tests pin the quoted-&& and inner-spacing preservation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Four follow-up commits from the adversarial review of this PR's own diff (every finding independently verified against the code before fixing):
Verified: clippy |

Summary
The complete 2026-08-10 structure-review implementation as one PR (consolidates and replaces #151–#155, which are closed in its favor). Ten commits, reviewable in order — each was independently verified before folding in:
1–2. The 2026-07 review-sweep, landed at last (
fix:×2): the 66-file bug-fix + test-harness sweep rebased onto main, plus the four fixes its own RED regression tests were pinning without:scan --redirecthuman mode printed JSON-quoted values;detect_updateswas blind to percent-encoded/artifact-qualified manifest keys (scoped packages never reachedupdates[]); the vendor reconcile's on-disk ledger mutation was invisible to JSON consumers when staging failed afterward (the step error now carries the envelope);CARGO_HOME=""resolvedregistry/srcagainst the CWD and silently crawled nothing. Also in the sweep: the nested-apply fix (getno longer drops--api-token/--api-url/--org/--proxy-urlon its internal apply).3–6. Module taxonomy (
refactor(core):×4):patch/vendor/(34 files, ~47% of core) promoted to top-levelvendor/; the strays rehomed (bun_lock_text,go_mod_edit→vendor/;go_redirect→patch::redirect::golang_local); the generic TOML helpers extracted from the pypi hook intoutils::toml_edit_ext; theutils/misfiles dissolved (telemetry→ top level,cleanup_blobs→manifest/,date→api/,fuzzy_match→crawlers/). Old paths keep compiling viapub useshims for external consumers of the published core crate; internal references are repointed, and a CI grep step rejects new internal uses (#[deprecated]on re-exports emits no warnings — rust-lang/rust#30827). Commit 3 is a pure-rename commit: if squash-merging, please add the squash SHA to.git-blame-ignore-revs.7. Setup umbrella (
refactor(core):):gem_setup/composer_setup/pth_hook— four naming schemes for one concept — unified assetup/{gem,composer,pypi}plus a thinsetup::npmalias overpackage_json's setup surface (package_jsonstays top-level: it doubles as the shared npm-manifest library).8. npm-family knowledge dedup (
refactor(npm):): a structured role-flag table inconstants::npm_family(sites accept intentionally divergent subsets, so a flat list can't serve them), equality drift-guard tests beside each consumer, PnP-marker and Rush-lock-path dedup, an exhaustive package-manager match inapply, anddeno.lock's absence recorded as a decision.9. CI honesty (
ci:): thetest/test-releasejobs build--all-features --no-runbut run default features, so the docker-e2e/setup-e2e suites stop reporting dozens of soft-skip fake greens per OS leg (they keep executing for real ine2e-docker/setup-matrix).10. Coverage gaps (
fix(setup,scan):):setup --excludefails closed on a corrupt manifest instead of rewriting it down to a bare setup block (bytes-unchanged, RED-verified);scan --redirect --jsonemits a parseable error envelope on all four failure exits instead of empty stdout (RED-verified); 7 unit tests for the previously-untestedfetch_stage.rsdownload planner.Verification
patches-api.socket.dev503 "Service temporarily over capacity" production incident (same tests were green this morning; same 503 failed this PR's earlier CI runs — infra, not code).cargo clippy --workspace --all-targets -- -D warningsclean,cargo fmt --checkclean, alias-path grep guard clean.🤖 Generated with Claude Code