feat(v4): n_bootstrap validation sweep + fail-closed inference selector (2(d) PR-B, M-081/M-096) - #754
feat(v4): n_bootstrap validation sweep + fail-closed inference selector (2(d) PR-B, M-081/M-096)#754igerber wants to merge 1 commit into
Conversation
…or (2(d) PR-B, M-081/M-096)
M-081: a shared utils.validate_n_bootstrap (promoted verbatim from
ChangesInChanges' local validator - non-negative int, numpy integers
accepted, bool/None/float/negative rejected) now runs at __init__ for
CallawaySantAnna, SunAbraham, EfficientDiD, ImputationDiD, TwoStageDiD,
WooldridgeDiD, ContinuousDiD, StaggeredTripleDifference and the DiD
family (DiD's __init__; MPD/TWFE inherit); CiC/QDiD re-point to the
shared helper with a byte-identical message. 0 stays legal and still
means bootstrap off on every > 0-gated analytical lane; no numeric
defaults change. Named exception (ledger + design doc): HAD keeps its
>= 1 floor - its n_bootstrap powers only the optional sup-t band whose
off-switch is fit(cband=False).
M-096: the inference selector fails closed. The accepted set is exactly
{"analytical", "wild_bootstrap"}, string-typed (an isinstance guard
rejects the one-element-ndarray hole in bare tuple membership), at
__init__ and transactional set_params. At fit - placed AFTER the survey
and Conley front doors so their NotImplementedError rejections keep
precedence - DiD with wild_bootstrap and no cluster= raises ValueError
(previously a SILENT analytical fallback; the pinned test
test_did_wild_bootstrap_requires_cluster flips BY DESIGN), and DiD/TWFE
with n_bootstrap < 2 raise (n_bootstrap in {0, 1} ran WCR with too few
draws and returned a wild-labeled all-NaN inference tuple; the < 2
floor amends the locked < 1, recorded in v4-design section 7 + the
M-096 notes in this diff). TWFE's unit auto-cluster stays; MPD's
warn-and-analytical-fallback stays (n_bootstrap-independent).
Fixed: DiD/TWFE never cleared per-fit bootstrap state, so a wild fit
followed by set_params(inference="analytical") + refit reported stale
inference_method="wild_bootstrap" + n_bootstrap/n_clusters/p_val_type
metadata. Both now reset _bootstrap_results at the top of fit().
Ledger: M-081 + M-096 planned -> done with test_ref
tests/test_v4_inference_policy.py (126 tests: the validation sweep with
rollback atomicity, the selector value-set incl. a non-string probe,
cluster/floor coherence with full-message pins, boundary n=2 acceptance
with a finite quintet, front-door precedence at sub-floor counts on
both estimators, the MPD carve-out at n=0/1/999, refit transitions, and
the dynamic roster guard pinning inference exposure to exactly
{DiD, MPD, TWFE}). ContinuousDiD enrolled in the BAD_VALUES rollback
lane (pre-existing catalog gap). Docs: REGISTRY WCR fail-closed Note +
MPD wild rows corrected (SE-summary table + section bullet);
troubleshooting/choosing_estimator examples gain the cluster=
prerequisite (plus the adjacent weight_type= -> bootstrap_weights=
kwarg fix); README/llms.txt/llms-practitioner normative wild
recommendations qualified DiD-vs-TWFE and the two CS-adjacent advice
strings reworded; llms-full DiD block annotated; a systematic
wild-mention disposition sweep recorded. The wild-bootstrap
cluster-count guidance is harmonized to the single 50-cluster
convention (choosing_estimator.rst was the <30 outlier). doc-deps gains
troubleshooting.rst + llms-practitioner.txt under estimators.py; TODO
row for the type-blind n_bootstrap holes in already-validated
estimators (HAD/dCDH/TROP/SyntheticDiD); DEFERRED MPD anchor corrected.
Overall Assessment✅ Looks good — no unmitigated P0 or P1 findings. Executive Summary
Methodology
Code QualityNo findings. The shared validator and guards are consistently implemented in PerformanceNo findings. Constructor validation and fit-time guards have negligible cost. MaintainabilityNo findings. Centralizing Tech Debt
SecurityNo findings or apparent secrets/PII. Documentation/TestsNo findings. Tests cover invalid types, rollback, selector values, DiD/TWFE prerequisites, survey/Conley precedence, MPD behavior, boundary counts, and refit transitions. Static AST parsing passed; runtime execution was unavailable due missing |
Summary
done): promote a sharedvalidate_n_bootstraptodiff_diff/utils.py(verbatim from ChangesInChanges' local validator: non-negative integer, numpy integers accepted, bool/None/float/negative rejected) and apply it at__init__across the previously-unvalidated roster - CallawaySantAnna, SunAbraham, EfficientDiD, ImputationDiD, TwoStageDiD, WooldridgeDiD, ContinuousDiD, StaggeredTripleDifference, and the DiD family (DiD's__init__; MultiPeriodDiD/TwoWayFixedEffects inherit); CiC/QDiD re-point to the shared helper with a byte-identical message.0stays legal and still means bootstrap off on every> 0-gated analytical lane; no numeric defaults change. Named exception recorded in the ledger + design doc: HAD keeps its>= 1floor (itsn_bootstrappowers only the optional sup-t band, whose off-switch isfit(cband=False)).done): theinferenceselector fails closed. Accepted set is exactly{"analytical", "wild_bootstrap"}, string-typed (anisinstanceguard closes the one-element-ndarray hole in bare tuple membership), at__init__and transactionalset_params. At fit - placed after the survey and Conley front doors so theirNotImplementedErrorrejections keep precedence - DiD withwild_bootstrapand nocluster=raisesValueError(previously a silent analytical fallback;test_did_wild_bootstrap_requires_clusterflips BY DESIGN, disclosed in CHANGELOG), and DiD/TWFE withn_bootstrap < 2raise (n_bootstrap ∈ {0, 1}ran WCR with too few draws and returned a wild-labeled all-NaN inference tuple; the< 2floor amends the originally-locked< 1, recorded indocs/v4-design.md§7 + the M-096 ledger notes in this diff). TWFE's unit auto-cluster stays; MPD's warn-and-analytical-fallback stays (n_bootstrap-independent).set_params(inference="analytical")+ refit reported staleinference_method="wild_bootstrap"+n_bootstrap/n_clusters/p_val_typemetadata. Both now reset_bootstrap_resultsat the top offit().troubleshooting.rst/choosing_estimator.rstexamples gain thecluster=prerequisite (plus the adjacentweight_type=→bootstrap_weights=kwarg fix); README/llms.txt/llms-practitioner.txtnormative wild recommendations qualified DiD-vs-TWFE and the two CS-adjacent advice strings reworded;llms-full.txtDiD block annotated; the wild-bootstrap cluster-count guidance harmonized to the single 50-cluster convention (choosing_estimator.rstwas the<30outlier). A systematic repo-wide wild-mention disposition sweep was run: normative recommendations qualified, capability/method prose untouched.docs/doc-deps.yamlgainstroubleshooting.rst+llms-practitioner.txtunderestimators.py(dependencies this diff proves); TODO row for the type-blindn_bootstrapholes in the already-validated estimators (HAD bool, dCDH bool+float, TROP float, SyntheticDiD float/jackknife); DEFERRED MPD anchor corrected; ContinuousDiD enrolled in theBAD_VALUESrollback lane (pre-existing catalog gap).Methodology references (required if estimator / math changes)
fwildclusterboot::boottest- seedocs/methodology/REGISTRY.md"Wild cluster bootstrap (WCR)".n_bootstrap >= 2under wild) is documented as a labeled Note in the REGISTRY WCR section; the< 2floor amendment to the locked v4-design decision is recorded indocs/v4-design.md§7 and the M-096 ledger notes in this same diff.Validation
tests/test_v4_inference_policy.py(new, 126 tests - both rows'test_ref: validation sweep with set_params rollback atomicity, selector value-set incl. a non-string probe, cluster/floor coherence with full-message pins, boundaryn=2acceptance with a finite inference quintet, survey/Conley precedence at sub-floor counts on both estimators, MPD carve-out atn=0/1/999, refit metadata transitions, dynamic roster guard pinninginferenceexposure to exactly {DiD, MPD, TWFE});tests/test_wild_bootstrap.py(the by-design flip);tests/test_estimators_vcov_type.py(cluster added to the wild/vcov coexistence pin);tests/test_base_estimator.py(ContinuousDiDBAD_VALUESenrollment).cluster=. Sphinx-Wbuild + the four post-build HTML guards pass;test_doc_snippets.pyexecutes the corrected RST examples.Security / privacy