fix(gem): bundler vendored+hosted audit — 13 confirmed bugs fixed, tests hardened - #175

…ll scan arms

Addresses both Bugbot findings on #175:

- The grant-agnostic Gemfile idempotency guard required LF (`do\n`), so a
  core.autocrlf checkout of a previously-redirected Gemfile was not
  recognized and the indented gem line inside the block got wrapped in a
  second, nested source block on re-run. The recognizer now accepts
  `do\r?\n`; pinned by gemfile_rerun_on_crlf_checkout_never_nests
  (verified red without the fix).

- The dry-run VEX skip only covered the vendor JSON arm; the interactive
  `scan --vendor --dry-run --vex` path (embed_vex_human) and the JSON
  `scan --apply --dry-run --vex` path (embed_vex_into_json at the apply
  fall-through) still generated the document — exiting 1 on a
  not-yet-vendored project or writing the attestation during --dry-run.
  The guard now lives at the top of both embed helpers, covering every
  scan arm; pinned by scan_vendor_dry_run_with_vex_interactive_* and
  scan_apply_json_dry_run_with_vex_* (both verified red without it).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…dor coverage; gems.rb + CRLF redirect fixes (#177)

* fix(gem): bundler vendored+hosted sweep — 13 confirmed bugs fixed, tests hardened

ULTRACODE review + full test matrix over the gem/bundler ecosystem
(vendored and hosted modes, every configuration). 24 findings survived
adversarial verification; the 13 well-scoped ones are fixed here, each
pinned by a test that fails without its fix.

Hosted redirect (patch/redirect/mod.rs, gem section):
- splice the Gemfile edit by regex byte range: a commented-out duplicate
  of the gem line no longer gets rewritten instead of the live line
- grant-agnostic idempotency guard: re-running scan --mode hosted with a
  rotated grant token refreshes the source URL in place instead of
  nesting a second source block (new edit kind redirect_gemfile_source_url)
- fail closed on gem-level git:/github:/path:/source: options (they
  override the enclosing source block, making the redirect an attested
  no-op); warn redirect_gem_source_option and skip the dep
- fail closed on platform-suffixed CHECKSUMS siblings
  (redirect_gem_platform_unsupported) instead of inserting a duplicate
  bare-coordinate pin
- recognize paren/tab/multi-space gem declarations; gate the
  append-branch on the gem being genuinely undeclared (no more duplicate
  declarations bundler rejects)
- never pin the lock CHECKSUMS when the Gemfile source redirect did not
  land (mixed state guaranteed a checksum failure)
- warn that a redirected pair breaks frozen/deployment installs
- record the upstream sha256 line as original on the CHECKSUMS edit so a
  future revert can restore it (golden fixture updated)

Vendored backend (vendor/gem.rs):
- insert new PATH sections at bundler's sorted position (identifier
  order, verified against real bundler 4.0.15 bundle lock) — two or more
  vendored gems no longer churn the committed lock
- fail closed on platform-suffixed GEM-specs siblings on no-CHECKSUMS
  locks (mirrors the existing CHECKSUMS guard)
- re-vendor on a patch UPDATE (new uuid, same purl): recognize our own
  path: wiring and rewire in place instead of refusing with
  gemfile_declaration_not_editable — the documented automatic re-vendor
  contract now actually works for gem

Auto-fetch (vendor/registry_fetch.rs):
- stage fetched gems into the canonical <name>-<version> leaf instead of
  a dir literally named "gem", which vendor_gem refused as
  platform_gem_unsupported — lockfile auto-fetch for gems was dead

Crawler (crawlers/ruby_crawler.rs):
- parse_dir_name_version prefers the last dotted-version boundary, so
  http-2-1.0.1 parses as (http-2, 1.0.1) instead of the ghost (http, 2)
- vendor/bundle discovery enumerates engine dirs (jruby, truffleruby)
  instead of hardcoding ruby/

Scan/get plumbing:
- run_nested_apply now threads --ecosystems: scan --ecosystems gem
  --sync no longer applies (or mutates) other ecosystems' patches
- scan --vendor --dry-run --vex no longer writes the VEX file nor exits
  1 on not-yet-vendored state

Test hardening:
- e2e_gem lifecycle harness: BUNDLE_PATH replaces bundle install --path
  (removed in bundler 3+; all 3 lifecycle tests green under 4.0.15)
- e2e_hosted_production gem leg now asserts the CHECKSUMS digest CHANGED
  after redirect (client-verifiable today) and, in the success arm,
  verifies installed content against the published afterHashes — an
  inert gem patch can no longer stay green (the npm minimist blindspot)
- docker_e2e_gem serves the true git-blob beforeHash so the chain
  exercises the default non-forced apply path, not just --force
- setup_matrix_gem module doc: the with-setup Docker cases ARE still a
  baseline gap (bootstrap deadlock: plugin registration evaluates
  plugins.rb before any gems land; exit-semantics twin), doc corrected

Verified: core 2460/0, cli lib 350/0, clippy+fmt clean; e2e_gem 11/11
(incl. live lifecycle under bundler 4.0.15), e2e_vendor_gem_build 6/6
(incl. real-bundler capstone), docker_e2e_gem 2/2, docker_e2e_vendor_gem
1/1, hosted production gem leg green (redirect verified; install still
blocked by the known depscan#23630 compact-index 404 — server-side).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(review): CRLF-tolerant source-block guard; dry-run VEX skip for all scan arms

Addresses both Bugbot findings on #175:

- The grant-agnostic Gemfile idempotency guard required LF (`do\n`), so a
  core.autocrlf checkout of a previously-redirected Gemfile was not
  recognized and the indented gem line inside the block got wrapped in a
  second, nested source block on re-run. The recognizer now accepts
  `do\r?\n`; pinned by gemfile_rerun_on_crlf_checkout_never_nests
  (verified red without the fix).

- The dry-run VEX skip only covered the vendor JSON arm; the interactive
  `scan --vendor --dry-run --vex` path (embed_vex_human) and the JSON
  `scan --apply --dry-run --vex` path (embed_vex_into_json at the apply
  fall-through) still generated the document — exiting 1 on a
  not-yet-vendored project or writing the attestation during --dry-run.
  The guard now lives at the top of both embed helpers, covering every
  scan arm; pinned by scan_vendor_dry_run_with_vex_interactive_* and
  scan_apply_json_dry_run_with_vex_* (both verified red without it).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(gem): hermetic hosted capstone + CHECKSUMS/transitive/scan--vendor coverage; gems.rb + CRLF redirect fixes

Closes the gem audit's top coverage gaps with hardened, independent-oracle
tests, plus the two small redirect fixes the new tests exercise.

Redirect fixes (patch/redirect/mod.rs gem section + hosted.rs candidates):
- gems.rb/gems.locked support end-to-end: the candidate list now reads the
  modern pair and the gem rewriter keys on whichever pair bundler reads
  (gems.rb wins when both spellings exist — verified on bundler 4.0.15;
  same order as setup::gem). Diverging spellings fail closed with
  redirect_gem_gemfile_spellings_diverge. Before: a gems.rb project was a
  silent hosted-mode no-op.
- CRLF Gemfile.lock tolerance: the CHECKSUMS matchers accept \r-terminated
  lines and edits preserve the file's CRLF endings byte-for-byte. Before:
  a CRLF lock (legal to bundler — verified via bundle check/frozen install)
  was misdiagnosed as bundler <2.6 (redirect_gem_no_checksums_section).

New coverage:
- e2e_redirect_gem_build.rs — FULLY HERMETIC hosted gem capstone: authored
  gems built with real `gem build`, one wiremock playing the upstream
  compact index, the Socket patch-registry compact index (production's
  /patch-registry/gem/<token>/<uuid>/ shape, real /versions md5s and
  /info checksums), and the patches API. scan --mode hosted, then a REAL
  fresh-checkout `bundle install` (host bundler) pulls the patched .gem:
  bytes match afterHash, the runtime dep installs because /info declares
  it, require probe + post-install verified VEX. A gems.rb twin pins the
  modern spelling end-to-end. A deps red-arm pins the compact-index
  dependency contract the production server currently violates: a
  deps-less /info breaks the install with bundler's
  APIResponseMismatchError (the live-CI signature). A KNOWN-LIMITATION
  canary pins that on a CHECKSUMS lock (bundler >= 4 default) the current
  rewrite makes the prescribed unfrozen install fail with "mismatched
  checksums" (empirically verified; the converged-lock fix shape that a
  frozen install accepts is documented in the test).
- docker_e2e_vendor_gem.rs — lockfile_checksums twin (bundler 2.7 in the
  image, `bundle lock --add-checksums`): vendor swaps the registry sha256=
  CHECKSUMS line for bundler's bare path-gem form, a frozen --network none
  install accepts the rewritten lock byte-stably, and revert restores the
  registry sha256= line VERBATIM (the exit-16 hazard).
- e2e_vendor_gem_build.rs — TRANSITIVE-dep capstone: vendoring rack via a
  rack-test project appends the managed block + sorted `rack (= v)!`
  DEPENDENCIES pin, a real frozen fresh-checkout install accepts the pair
  byte-stably, require probe through the dependent, revert round-trip.
- in_process_vendor.rs — gem through `scan --vendor` (mock-proxy API,
  hermetic bundler layout, no ruby needed): JSON-arm end-to-end (pair edit
  + artifact + stub gemspec + already_vendored rerun), manifest-drop
  reconcile byte-restore, and the --detached variant (no manifest,
  embedded record, vendor --revert exit path).

Every rewriter behavior change is pinned by unit tests that fail on the
base commit (gems.rb pair routing, diverging-spelling fail-closed, CRLF
pin-in-place/insert byte-preservation, CRLF rerun no-op); the gems.rb e2e
leg fails without the candidate-list entry. Bundler behaviors (gems.rb
preference, lockfile naming, compact-index formats, checksum enforcement,
CRLF acceptance) were validated against real bundler 4.0.15 and the
image's 2.7.2 before being baked into fixtures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(redirect): judge gems.rb/Gemfile divergence on the redirect footprint residue, not raw bytes

Run 1 on byte-identical gems.rb/Gemfile twins edits only gems.rb (the
file bundler reads), so the raw-byte divergence guard trapped every
later run — the rotated-grant URL refresh included — behind
redirect_gem_gemfile_spellings_diverge, a divergence the rewriter
itself created. Compare redirect-footprint residues instead: erase the
managed Socket source block (rotating grant segments wildcarded) and
the dep's own gem declaration line from both spellings before judging.

Rebased onto d063cb1 (CRLF-tolerant block recognizer): the residue
eraser now accepts `do\r?\n … end\r?` too, mirroring the recognizer —
a core.autocrlf checkout rewrites run 1's LF block to CRLF, and a
block the recognizer accepts must also be erased here or the re-run is
trapped behind the divergence warning before it can reach the
recognizer. Pinned by
gems_rb_crlf_twins_rerun_is_no_op_and_rotated_grant_refreshes
(verified red with an LF-only residue eraser).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…on gem/pypi/maven variants; variant-scope the rollback before-blob gate (#176)

* fix(gem): bundler vendored+hosted sweep — 13 confirmed bugs fixed, tests hardened

ULTRACODE review + full test matrix over the gem/bundler ecosystem
(vendored and hosted modes, every configuration). 24 findings survived
adversarial verification; the 13 well-scoped ones are fixed here, each
pinned by a test that fails without its fix.

Hosted redirect (patch/redirect/mod.rs, gem section):
- splice the Gemfile edit by regex byte range: a commented-out duplicate
  of the gem line no longer gets rewritten instead of the live line
- grant-agnostic idempotency guard: re-running scan --mode hosted with a
  rotated grant token refreshes the source URL in place instead of
  nesting a second source block (new edit kind redirect_gemfile_source_url)
- fail closed on gem-level git:/github:/path:/source: options (they
  override the enclosing source block, making the redirect an attested
  no-op); warn redirect_gem_source_option and skip the dep
- fail closed on platform-suffixed CHECKSUMS siblings
  (redirect_gem_platform_unsupported) instead of inserting a duplicate
  bare-coordinate pin
- recognize paren/tab/multi-space gem declarations; gate the
  append-branch on the gem being genuinely undeclared (no more duplicate
  declarations bundler rejects)
- never pin the lock CHECKSUMS when the Gemfile source redirect did not
  land (mixed state guaranteed a checksum failure)
- warn that a redirected pair breaks frozen/deployment installs
- record the upstream sha256 line as original on the CHECKSUMS edit so a
  future revert can restore it (golden fixture updated)

Vendored backend (vendor/gem.rs):
- insert new PATH sections at bundler's sorted position (identifier
  order, verified against real bundler 4.0.15 bundle lock) — two or more
  vendored gems no longer churn the committed lock
- fail closed on platform-suffixed GEM-specs siblings on no-CHECKSUMS
  locks (mirrors the existing CHECKSUMS guard)
- re-vendor on a patch UPDATE (new uuid, same purl): recognize our own
  path: wiring and rewire in place instead of refusing with
  gemfile_declaration_not_editable — the documented automatic re-vendor
  contract now actually works for gem

Auto-fetch (vendor/registry_fetch.rs):
- stage fetched gems into the canonical <name>-<version> leaf instead of
  a dir literally named "gem", which vendor_gem refused as
  platform_gem_unsupported — lockfile auto-fetch for gems was dead

Crawler (crawlers/ruby_crawler.rs):
- parse_dir_name_version prefers the last dotted-version boundary, so
  http-2-1.0.1 parses as (http-2, 1.0.1) instead of the ghost (http, 2)
- vendor/bundle discovery enumerates engine dirs (jruby, truffleruby)
  instead of hardcoding ruby/

Scan/get plumbing:
- run_nested_apply now threads --ecosystems: scan --ecosystems gem
  --sync no longer applies (or mutates) other ecosystems' patches
- scan --vendor --dry-run --vex no longer writes the VEX file nor exits
  1 on not-yet-vendored state

Test hardening:
- e2e_gem lifecycle harness: BUNDLE_PATH replaces bundle install --path
  (removed in bundler 3+; all 3 lifecycle tests green under 4.0.15)
- e2e_hosted_production gem leg now asserts the CHECKSUMS digest CHANGED
  after redirect (client-verifiable today) and, in the success arm,
  verifies installed content against the published afterHashes — an
  inert gem patch can no longer stay green (the npm minimist blindspot)
- docker_e2e_gem serves the true git-blob beforeHash so the chain
  exercises the default non-forced apply path, not just --force
- setup_matrix_gem module doc: the with-setup Docker cases ARE still a
  baseline gap (bootstrap deadlock: plugin registration evaluates
  plugins.rb before any gems land; exit-semantics twin), doc corrected

Verified: core 2460/0, cli lib 350/0, clippy+fmt clean; e2e_gem 11/11
(incl. live lifecycle under bundler 4.0.15), e2e_vendor_gem_build 6/6
(incl. real-bundler capstone), docker_e2e_gem 2/2, docker_e2e_vendor_gem
1/1, hosted production gem leg green (redirect verified; install still
blocked by the known depscan#23630 compact-index 404 — server-side).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(review): CRLF-tolerant source-block guard; dry-run VEX skip for all scan arms

Addresses both Bugbot findings on #175:

- The grant-agnostic Gemfile idempotency guard required LF (`do\n`), so a
  core.autocrlf checkout of a previously-redirected Gemfile was not
  recognized and the indented gem line inside the block got wrapped in a
  second, nested source block on re-run. The recognizer now accepts
  `do\r?\n`; pinned by gemfile_rerun_on_crlf_checkout_never_nests
  (verified red without the fix).

- The dry-run VEX skip only covered the vendor JSON arm; the interactive
  `scan --vendor --dry-run --vex` path (embed_vex_human) and the JSON
  `scan --apply --dry-run --vex` path (embed_vex_into_json at the apply
  fall-through) still generated the document — exiting 1 on a
  not-yet-vendored project or writing the attestation during --dry-run.
  The guard now lives at the top of both embed helpers, covering every
  scan arm; pinned by scan_vendor_dry_run_with_vex_interactive_* and
  scan_apply_json_dry_run_with_vex_* (both verified red without it).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(apply,rollback): singleton-variant mismatch policy + variant-scoped rollback blob gate

apply (P1): for release-variant ecosystems (gem/pypi/maven) the variant
loop's installed-distribution gate skipped ANY variant whose
representative file mismatched — for a SINGLETON base (the common bare
pkg:gem/name@ver record) that made the documented default mismatch
policy (npm parity: warn content_mismatch_overwritten + apply the full
verified patched content, --strict refuses, --force unchanged)
unreachable: a locally-modified file failed the run with "no matching
variant found" and left the file untouched. The gate now applies to
multi-variant bases only — a singleton has no sibling to make a
mismatch mean "different distribution", so it falls through to
apply_package_patch's MismatchPolicy exactly like the npm branch.
mismatch_blob_gaps mirrors the same singleton exemption so the Warn
fall-through's afterHash blobs are prefetched under --download-mode
diff. Multi-variant behavior is UNCHANGED (sibling mismatch = skip;
no-variant-matches = fail closed), pinned by two guards.

rollback/remove (P2): the before-blob gate ran over the WHOLE
ecosystem-scoped manifest BEFORE select_installed_variants narrowed a
broad multi-platform gem manifest to the installed platform, so an
uninstalled sibling ?platform= variant's unfetchable before-blob
aborted the entire run (--offline: wholesale; online: on any download
failure) — and remove then refused with rollback_failed. Discovery +
variant narrowing now run first and the gate covers only the (purl,
path) pairs that will actually be attempted, plus in-scope purls the
crawler could not resolve at all — those keep the pre-existing
fail-closed gate (pinned by rollback_invariants / cli_rollback_silent /
remove_network), and local-redirect purls stay excluded as before.

Tests (each fails on the base commit without its fix):
- cli_gem_variant_mismatch_policy.rs (new): singleton default
  warn+apply (human + JSON envelope), singleton --strict refusal;
  guards: --force unchanged, multi-variant sibling never overwritten,
  multi-variant no-match still fails closed.
- in_process_gem_multi_platform.rs: rollback with unfetchable sibling
  before-blob succeeds online (404 mock, blob never requested) and
  --offline (installed variant's blobs only); remove ditto and clears
  both records.
- apply.rs unit: mismatch_blob_gaps_singleton_mismatch_queued (RED vs
  the old gate); the sibling-skip test now uses a genuine multi-variant
  group.

Verified: core 2460/0, cli lib 351/0, docker_e2e_gem 2/2, e2e_gem 8/8
non-ignored, gem/pypi variant + rollback/remove/apply suites green,
clippy+fmt clean. Platform-gem dir layout re-validated against real
RubyGems (ruby 3.4.10): gems/<name>-<version>-<platform> leaf.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(apply): keep qualified singletons and vendor-owned bases out of the mismatch fall-through

Review fixes for the singleton-variant mismatch policy:

Major: the singleton exemption covered QUALIFIED singletons too. A lone
`pkg:gem/name@ver?platform=x86_64-linux` record names one specific
distribution, and the representative-hash gate is the ONLY platform
resolution (the ruby crawler drops the gem dir's platform suffix
precisely because "the installed platform is resolved later" by that
hash check) — falling through let a linux-only record silently
overwrite a darwin install in the Bundler plugin's `--silent`
auto-apply, where the warn half of warn-and-apply is invisible, with a
later rollback restoring the LINUX before-bytes. The exemption now
applies to UNQUALIFIED singletons only (`variants[0] == base_purl`);
qualified singletons keep the pre-existing fail-closed "no matching
variant found". `mismatch_blob_gaps` mirrors the same condition (its
records now retain their manifest keys so the gate can see qualifiers).

Minor: `mismatch_blob_gaps` now skips vendor-owned bases. The apply
loop unconditionally skips them (their results are synthesized up
front), so a drifted vendored release-variant file queued a spurious
"Downloading N full patched blob(s)" fetch online and a spurious "will
fail to apply" warning under `--offline`.

Minor: refreshed the rollback_invariants comment made stale by the
gate reorder — the offline missing-blob bail now aborts after
discovery, before the rollback loop produces per-package results; the
pinned contentless partial_failure shape is unchanged.

Tests (each fails without its fix):
- cli_gem_variant_mismatch_policy::qualified_singleton_wrong_platform_fails_closed:
  lone ?platform= record + wrong-platform install → exit 1,
  "no matching variant found", file byte-identical, no overwrite warning.
- apply.rs unit mismatch_blob_gaps_qualified_singleton_gated_unless_forced:
  qualified singleton mismatch queues nothing by default, queues under
  --force.
- apply.rs unit mismatch_blob_gaps_vendored_base_never_queued: vendored
  base queues nothing (even under --force), same fixture queues without
  the vendor claim.

Verified: cli lib 353/0, cli_gem_variant_mismatch_policy 6/6,
in_process_gem_multi_platform 9/9 non-ignored, apply/rollback/remove/
vendor/variant suites green, docker_e2e_gem 2/2, e2e_gem 8/8
non-ignored, core green, clippy+fmt clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>