For development, we used uv. Since our deps are in the lockfile, it is possible to pip freeze and pipe to a requirements file from a venv made with uv venv and run with vanilla python/python3. There are no plans to support conda as of right now. We recommend a cuda compute capable GPU to make the ranking and search faster.

Some helpful uv syntax:

• uv venv makes virtual environment from uv.lock file (automatically performed with uv run

• uv run [filename] runs python file. Consider modules i.e. uv run -m module.file

For ease of use, we have also provided a requirements.txt for running with vanilla python, but we recommend still using a venv and the recommended way of interfacing with the codebase is uv. This can be done as follows:

• python3 -m venv <name of venv>
• source <name of venv>/bin/activate
• pip install -r requirements.txt

Important Note: The colbert package is completely deprecated, and has been replaced with colbert-ai.

If using uv:

• uv remove colbert and uv remove colbert-ai, then uv add colbert-ai.

If using pip + venv:

• pip uninstall colbert && pip uninstall colbert-ai
• pip install -r requirements.txt

Analysis is present on the website, which can be run locally using npm run dev from the website folder, or by clicking the link. Note that obviously on a new clone npm install will have to be run if running locally. Please refer to the package-lock.json and README.md in the website folder for more pertintent instructions on accessing and running that content.

Alternatively, findings are present in both overall_analysis and per_query_analysis.

An index can be constructed using either the notebook provided or colbert_local.py. Note we were only able to make this work on POSIX systems due to CPP shenanigans on windows. A sample search can be performed using get_scores.py. Refer to the files for more clear usage instructions, and to the original colbert codebase which we used as a basis for package usage with Indexer and Searcher

Our base dataset was the ms-marco dataset, which has been used as a benchmark for neural ranking models since the original ColBERT paper, if not before that. The link to a huggingface dataset can be found here. There are a LOT of bugs with