Skip to content

Allow overriding the gated Llama-3 tokenizer used for the LLaMa3 prompt style - #151

Open
DrStranded wants to merge 1 commit into
LiveCodeBench:mainfrom
DrStranded:feat/llama3-tokenizer-env
Open

DrStranded wants to merge 1 commit into
LiveCodeBench:mainfrom
DrStranded:feat/llama3-tokenizer-env

Conversation

@DrStranded

Copy link
Copy Markdown

Summary

format_prompt_generation renders the LMStyle.LLaMa3 prompt by loading the tokenizer of meta-llama/Meta-Llama-3-8B-Instruct, purely to call apply_chat_template. That repo is gated, so anyone whose HF account has not been granted access to it gets an HTTP 403 from AutoTokenizer.from_pretrained and every Llama-3-style evaluation fails, even when the model being evaluated is a different (accessible) Llama 3 checkpoint.

This PR keeps the default unchanged and adds an environment-variable override, LCB_LLAMA3_TOKENIZER, so the chat template can be rendered from any Llama 3 family checkpoint the user can access (the chat template is the same across Llama 3 / 3.1 / 3.2 instruct models), e.g.

LCB_LLAMA3_TOKENIZER=meta-llama/Llama-3.2-3B-Instruct python -m lcb_runner.runner.main --model ... --scenario codegeneration

No behaviour change when the variable is not set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant