Description and results

Use Python 3.12+ and run commands from the repository root. The repository includes the frozen task metadata. Images download automatically from S3 using anonymous requests; no AWS account or Hugging Face login is needed for the dataset.

With uv:

git clone https://github.com/qforge-dev/wildlife-CSI-bench.git
cd wildlife-CSI-bench
uv sync
test -f .env || cp .env.example .env
uv run csi models

Alternatively, use a virtual environment and pip:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
csi models

With pip, omit uv run from the commands below. The CLI loads .env from the repository root; existing shell variables take precedence. Keep credentials in .env, which Git ignores.

Edit .env.example's copied values in .env. You only need credentials for the selected model and the answer extractor used for scoring.

For OpenAI, use https://api.openai.com/v1 as the base URL. For Azure, use https://YOUR-RESOURCE.openai.azure.com/openai/v1 and set the model override to your deployment name. Bedrock authentication applies only to model calls; image downloads stay anonymous. Muse's Contributor tier requires accepting the provider's data-use terms.

The default answer extractor uses GPT6_SOL_BASE_URL, GPT6_SOL_API_KEY, and GPT6_SOL_MODEL. Configure those even when benchmarking another model. It only reads model replies the parser cannot resolve. To use your own compatible text model, copy that YAML, change its model and environment-variable fields, and pass --extractor-config path/to/answer.yaml to csi benchmark or csi score. Changing the extractor can affect comparability with the published scores.

uv run csi models lists model IDs and missing configuration without printing keys. Each model's settings, reasoning effort, token limit, and estimated prices live in configs/models. Verify rates for your provider before a paid run.

OpenRouter configs set cost_source: provider: recorded USD charges from usage.cost take precedence over token-rate estimates, including when enforcing a spending limit or resuming a run. Missing or invalid charges fall back to the YAML rates. The estimated_cost_usd output field keeps its existing name; provider.cost_source identifies reported charges versus estimates. AWS and Azure costs remain token-rate estimates. Published raw runs retain their original estimates; the charts use their saved OpenRouter charges where available.

uv run csi validate

This checks task, source, location, and selection hashes, then downloads and verifies all 2,000 images. Subsequent runs reuse data/work/csi-s3-cache/. Corrupt cached images are downloaded again; a remote checksum mismatch stops the run.

For a complete run with one model:

uv run csi benchmark --models deepseek --max-workers 10

Select multiple models with --models deepseek,glm. --models all explicitly selects every configured model and requires their credentials. The runner checks credentials before making model calls and scores each model once all predictions are recorded.

To check your setup with a spending limit:

uv run csi benchmark --models deepseek --max-workers 2 --max-cost-per-model 1

The limit stops new requests after recorded estimated spending reaches $1; in-flight calls can exceed it. A limited run can be incomplete and will not be automatically scored. Repeat the command with a higher limit, or without the limit, to finish it.

Outputs default to data/benchmarks/wildlife-csi-country-v1-2000/runs/<model>/. Repeating the same command resumes recorded predictions. Use a new --out, such as --out data/work/my-experiment, for a fresh experiment or changed model configuration. Provider errors already recorded as predictions remain part of that run.

Generation needs only the selected vision model's credentials:

uv run csi run \
  data/benchmarks/wildlife-csi-country-v1-2000/tasks.jsonl \
  data/work/my-run/predictions.jsonl \
  --model deepseek --max-workers 10

uv run csi score \
  data/benchmarks/wildlife-csi-country-v1-2000/tasks.jsonl \
  data/work/my-run/predictions.jsonl

Add --offline to csi score once the run's taxonomy-resolution and extraction caches are complete. Keep the same extractor configuration and endpoint when replaying its cache. The scorer rejects caches belonging to different task, prediction, or extractor snapshots.

Use uv run csi --help or a command's --help for all options. csi run --images /path/to/images accepts a local directory containing <image_sha256>.jpg files instead of downloading from S3.

Copy a YAML from configs/models to a new file in that directory. Give it a unique id, set the provider's model name, and point api_key_env and base_url_env at variables in your .env. Adjust token limits, reasoning options, and prices for that provider. Then select the new ID with --models your-model.

The runner supports OpenAI-compatible chat-completion vision endpoints and Bedrock Converse. It records the resolved settings with each run. Keep the frozen photos and prompts unchanged when comparing against the published results.

Your generated runs, caches, and credentials are excluded from Git. The eight published runs live separately in runs, with full predictions compressed as predictions.jsonl.gz. Each publication.json records original and published file hashes. Private Azure resource hostnames are replaced with azure-endpoint.invalid; model answers, usage, costs, and grades are preserved.

Scoring writes score/confusion_matrix.json automatically. To regenerate existing runs offline:

uv run csi confusion runs/*/score

Rows are actual species; columns are predictions. overall.counts[actual][predicted] holds photo counts (omitted cells are zero), row_totals gives support, and per_clue_type splits the counts by trace type. labels maps keys to taxonomy IDs and names. Species use species: keys, higher-rank answers use taxon:, and failures use status:. Endpoint blocks retain the runner's transport_error status. Unknown parent-species names are null, with recorded names in observed_names. Source hashes and scoring-review flags are included.

Render the README chart with its separate Matplotlib dependency:

uv run scripts/plot_confusion.py runs/opus/score/confusion_matrix.json \
  --title "Opus 5.5 · Species confusion"

The script exports light and dark SVGs. It selects the 15 most frequent species (ties by scientific name) and includes all remaining predictions in “Other predictions”.

The explorer combines the published confusion matrices into a zoomable network. Use Node.js 22.13+, Python 3.12+, and uv:

cd explorer
npm ci
npm run dev

Open the local URL printed by Vite. The startup script generates graph data from all published matrices; it needs no model credentials or API calls. Search species, inspect per-model counts, filter shared mistakes, or isolate a neighborhood. The mutual k-nearest view connects species with similar outgoing error profiles across all eight models. Communities are exploratory; repeated model answers concern the same photos, and low-support species can be unstable.

npm test
npm run build
npm run preview

The build writes a static site to explorer/dist/, using relative URLs compatible with GitHub Pages project sites. It has no backend. The Explorer workflow tests and builds pull requests, then publishes pushes to main to GitHub Pages. Generated data, builds, and dependencies are excluded from Git.

The initial view highlights mistakes shared by at least four models among species with at least five photos. Use the finding presets or adjust the filters, then Copy link to share the current view, selection, neighborhood, and camera position. The address updates automatically.

Query examples (append to the explorer URL):

?preset=shared
?preset=raccoon
?preset=similar
?preset=all
?view=direct&models=opus,sonnet&shared=2&support=5

view accepts direct or profile; models uses run IDs (none selects none). min and shared set mistake and model-count thresholds; support accepts 0, 3, 5, or 10 photos. k accepts 3, 5, 10, or 20. group uses the displayed community number; predicted=0 hides predicted-only species. species, focus, and the from/to pair accept stable taxon IDs or scientific names. camera is generated by the app. Preset parameters can be overridden by other parameters. Use the public site when copying links to share with others.

The run index links each published result to its evidence. Predictions are gzip-compressed to keep the repository manageable and stay below GitHub's file-size limit. Manifests, summaries, and per-photo scores are ordinary JSON/JSONL files.

To reproduce Opus's score without provider credentials or network calls:

mkdir -p data/work/replay-opus
gzip -dc runs/opus/predictions.jsonl.gz > data/work/replay-opus/predictions.jsonl
cp -R runs/opus/score data/work/replay-opus/score
uv run csi score \
  data/benchmarks/wildlife-csi-country-v1-2000/tasks.jsonl \
  data/work/replay-opus/predictions.jsonl \
  --offline --extractor-config runs/opus/extractor.yaml

Replace opus with another directory from runs/ to replay that model. Use the run's extractor.yaml: it preserves the published cache identity and uses a non-routable endpoint. It is only for offline replay; new model runs use your own provider configuration.

The chart snapshot retains the original prediction hashes. Each run's publication.json maps those hashes to the published copies after endpoint redaction and cache rebinding. Recorded per-photo scores are unchanged. Local logs, the taxonomy database, and the interrupted Opus outage attempt are not part of the published results.

There are 400 photos each of bones, eggs, feathers, droppings, and footprints. Every photo uses this system message:

Identify animal traces from photos. Give one best species guess, preferably a scientific name. Reply on one line: ANIMAL: <name>. No explanation or alternatives.

The user message varies only by country:

I found this animal trace in the field. The photo's country is South Africa. Which species most likely left it?

The parser extracts one animal name and the scorer resolves it against iNaturalist taxonomy IDs. Credit is cumulative: species matches also count toward genus and family. The text extractor sees only the model's reply. Failed calls, empty replies, and invalid answers remain in the denominator. Inspect score_complete, provisional_due_to_extractor, and the review queue before comparing scores.

The official suite has task hash:

b1d0ee38219c2f63542ef6d39249ab2735b976287bb775101f5eba73cd877b42

Tasks, prompts, source snapshots, location records, and the selection ledger are frozen. Changes to them require a new suite identity and fresh runs. Image access settings in the manifest describe hosting and do not change the task hash.

Countries come from public iNaturalist place records. Four private observations use documented species-range examples and are marked location.is_observation_location: false; these are answer-derived hints. One public-place country conflicts with free-text locality, and one Puerto Rico observation is represented as United States. The scorer reports results by location basis.

Images were normalized to metadata-free RGB JPEGs, with a minimum short side of 320 pixels, a maximum long side of 1,536 pixels, and a 3 MB size cap. Source revisions, photo attribution, and individual licenses are retained in the task records. Some photos have noncommercial, no-derivatives, or missing license information; public download access does not replace those terms.

Use the exact image_s3_uri and image_sha256 in each task. The HTTPS equivalent is:

https://wildlife-csi-dataset-088543363904.s3.us-east-1.amazonaws.com/media/sha256/<first-two-hash-characters>/<sha256>.jpg

Individual image GETs and HEADs are public. Bucket contents and object versions cannot be listed anonymously. The runtime only fetches known keys and verifies their SHA-256; it never lists the bucket. Task metadata is distributed with this repository.

The bucket's reproducible settings are in infra/s3-policy.json and infra/s3-public-access-block.json. Maintainers with bucket-management credentials can apply them with:

aws s3api put-public-access-block \
  --bucket wildlife-csi-dataset-088543363904 \
  --public-access-block-configuration file://infra/s3-public-access-block.json
aws s3api put-bucket-policy \
  --bucket wildlife-csi-dataset-088543363904 \
  --policy file://infra/s3-policy.json

The policy grants only s3:GetObject under media/sha256/. Public ACLs remain blocked and ignored; there are no public listing or write grants.

uv sync --extra dev
uv run pytest -q
uv run ruff check .

Tests use synthetic images and mocked model responses, so they need no provider credentials or paid model calls. The runtime is in wildlife_csi: CLI and run orchestration, provider adapters, S3 and local stores, and taxonomy scoring. Dataset-construction tools are not needed to run the frozen benchmark.