--not_fast silently ignores --start_date/--end_date and --release_version; other scenarios ignore the date flags too

Affected

28fef95 (main, 2026-09-02)

• lcb_runner/benchmarks/code_generation.py:139-143 (load_code_generation_dataset_not_fast)
• lcb_runner/benchmarks/code_generation.py:124-136 (load_code_generation_dataset, the only loader that honors the flags)
• lcb_runner/runner/scenario_router.py:50-77 (build_prompt_benchmark)
• lcb_runner/benchmarks/test_output_prediction.py:62-66 and code_execution.py:59-63 (release_version accepted, never used)
• lcb_runner/runner/parser.py:121-133 (the flags are accepted for every scenario)

Summary

--start_date and --end_date select the contamination window and --release_version pins the dataset revision. All three are honored only on the default code_generation_lite code path. When --not_fast is passed (the README's documented way to run the original benchmark with the full private test suites), build_prompt_benchmark calls load_code_generation_dataset_not_fast(args.release_version), which:

1. never forwards release_version to load_dataset (load_dataset("livecodebench/code_generation", split="test") has no version_tag), so the request silently resolves to whatever the Hub repo's default config currently is, and
2. has no date parameters at all, so args.start_date/args.end_date are silently dropped and every problem in the dataset is loaded, including problems after the requested end date.

No warning, error, or log line mentions either drop. The same silent drop applies to --scenario selfrepair (dates dropped, release version forwarded) and to --scenario testoutputprediction/codeexecution (dates and release version both dropped).

Impact

The contamination-free property of a run is decided by the union of the loaded problems, not by the flags on the command line. A user replicating the paper's windowed evaluation (for example "only problems released after August 2023" as documented in the README) with --not_fast gets the full dataset, and if they then compute scores without re-filtering (because they believe generation was already windowed), contaminated problems silently enter the score. A user pinning --release_version release_v2 with --not_fast silently benchmarks against a different, drifting dataset revision. Cross-model comparisons built this way compare different problem sets without any signal.

Reproduction

PoC: poc/poc_f1_dropped_window_flags.py (runs via poc/run_all.sh; evidence in poc/out/f1/run{1,2}/evidence.json, double-run byte-identical).

It calls the real build_prompt_benchmark at the pin with a stubbed HuggingFace load_dataset boundary that records its arguments and returns four synthetic problems (two inside the 2024-01-01 to 2024-06-01 window, one before, one after):

• fast path (control): 2 problems loaded, version_tag="release_v5" forwarded
• --not_fast with identical flags: 4 problems loaded (both out-of-window problems included), no version_tag in the load_dataset call
• --scenario selfrepair with identical flags: 4 problems loaded, dates dropped
• --scenario testoutputprediction with identical flags: out-of-window rows included, dates and version dropped

Suggested fix

Honor or reject: either thread start_date/end_date (and version_tag) through every loader, or raise/warn when selection flags are provided that the chosen path does not implement. For --not_fast, at minimum forward the release version and apply the same date filter used by the fast path.