Add native memory retrieval and BM25 evaluation tooling - #3258

Summary

• Add a chunking-bench CLI for Qasper/QMSum corpus preparation, retrieval diagnostics, and native answer evaluation.
• Add immutable, paired native/BM25 experiments, reusable BM25 statistics, source-coordinate evidence mapping, and opt-in native block/window policies.
• Honor named embedding selections when legacy settings remain, warning instead of rejecting an otherwise valid configuration.

Motivation

Compare retrieval and actual native answers on common datasets, then measure whether lexical reranking or additional lexical candidates improve native evidence selection. Retrieval span metrics are diagnostics, not answer-quality scores. This does not enable BM25 in production.

Behavior and configuration

• Production retrieval defaults are unchanged. Intervention hooks run only when explicitly supplied through the benchmark runtime. Returned blocks are canonicalized against active, authorized corpus/source/revision identities; window hooks must permute existing windows without changing their geometry.
• Native evaluation preserves existing DocParts and reader limits. The separate article-style retrieval diagnostic can use a packed context budget; that budget is not imposed on native answering.
• Paired policies include identity control, candidate reranking, window reranking, native-score/BM25 RRF, and an explicitly gated experimental controller. Corpus statistics and compatible frozen snapshots are pinned rather than silently rebuilding stores.
• Output fingerprints, single-executor locks, pre-call attempt markers, saved answers, and judge caches prevent accidental regeneration or incompatible resumes. Failed/interrupted answer attempts are terminal in that output namespace; saved answers can resume grading.
• Fingerprints include effective embeddings and the native reader connection independently of the judge. Reusing an older snapshot with a changed query fingerprint requires explicit fresh query lineage and a new control.
• Named embedding selectors, including none, take precedence over legacy variables with a warning. Legacy-only behavior and invalid-selector validation remain.
• Datasets, archives, local configuration, provider credentials, and experimental outputs are not included.

Reviewer guide

1. Runbook and operator commands: distinct retrieval/native/paired paths and limitations.
2. Benchmark orchestration: datasets.ts, runner.ts, nativeAnswers.ts, and incrementalRunner.ts; compatibility, snapshot, receipt, and policy modules separate the integrity checks from scoring.
3. Native selection hooks and benchmark bridge: absent-hook parity, active-revision/source isolation, and model/configuration identity.
4. Embedding precedence and its catalog regression tests.

Regression tests cover source-coordinate mapping, raw-source integrity, BM25 legacy parity, canonical policy outputs, real no-hit recovery, failed-answer replay prevention, and native-reader endpoint drift with a separate judge.

Validation

Tested integrated head: b1bc84b0af232e19bcea883bdc0aa3545caa9585, including latest main be033bc58, the formatting bot's script-order correction, and generated documentation updates.
Environment: Windows, Node 24.22.1, standalone pnpm 11.9.0.

Focused benchmark selector: incrementalBm25.spec.js|nativeAnswers.spec.js|chunkingBench.spec.js|memoryBench.spec.js.
Memory-service selector: indexedFragmentPolicy.spec.js|indexedFragmentPolicyIntegration.spec.js|indexedFragments.spec.js|answerKnowPro.spec.js|benchmarkBridge.spec.js|documentChunking.spec.js|retainedIndexService.spec.js|topicKnowPro.spec.js|viewSynthesisEvidence.spec.js|evidenceWindowUpgrade.spec.js.

CI follow-through: Linux/Windows test jobs exposed a module-link failure in the
historical evidence-upgrade fixture: the frozen selector module did not expose
the current host's new applyIndexedBlockPolicy adapter. Reproduced locally
before the fix; the fixture now supplies that host helper alongside its
existing query-term shim. Frozen selector contents/checksum and both
unchanged-index upgrade assertions remain intact. All 40 focused
upgrade/block-policy/window regression cases pass after the correction.

These are local results; CI and human review are separate. One existing
view-publication archive-cleanup assertion passed the initial integrated run,
failed a later repetition, and passed its focused retry. No assertion was
weakened or disabled; the initial failure and retry are retained in validation
evidence. The 239-case total includes that passed-on-retry case.

Risks, limitations and rollout

• Experimental controllers are opt-in, not an activation recommendation. No production global lexical index, smaller DocParts, additional native context cap, or provider-call benchmark is introduced.
• Existing pre-PR pilot grades are small, inspected, model-judged evidence, not proof of quality gains. The v1 N3 prototype fused storage positions rather than native-score ranks; those historical results do not evaluate the corrected raw-native-bm25-v2 policy. Do not relabel or mutate old receipts. No new paid experiments were run during PR preparation.
• Native internal model-call/token totals remain unavailable rather than reported as zero. Official threshold passes do not necessarily mean complete answers.
• Held-out evaluation, calibrated/independent judging, larger datasets and any production activation remain follow-up work.