EvalPort: portable interchange for CodeGenerationProblem tasks #159

Hi LiveCodeBench team — I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open JSON-Schema spec for portable LLM eval test cases, graders, and result sets, with Python/TS SDKs that validate against the schemas.

I was reading lcb_runner/benchmarks/code_generation.py and noticed CodeGenerationProblem (question_content, platform, difficulty, public_test_cases/private_test_cases: list[Test], metadata) plus insert_output_evaluation()'s pass@1 scoring already lines up closely with EvalPort's document model:

• CodeGenerationProblem → EvalPort TestCase (input = question_content + starter_code, expected = test cases)
• Test(input, output, testtype) → EvalPort TestCase.input/expected_output pairs
• insert_output_evaluation()'s pass@1 → EvalPort GraderResult.score

A thin CodeGenerationProblem ↔ TestCase adapter would let LiveCodeBench-derived results round-trip into any of the ~10 frameworks EvalPort targets (Inspect AI, promptfoo, DeepEval, etc.) without each one writing a bespoke LiveCodeBench loader. Not asking you to prioritize this — just flagging it as an option since the schema overlap looked unusually clean.

Schemas: https://github.com/adhabnr-ux/evalport/tree/main/spec/schemas

— Sahi, independent contributor (not affiliated with LiveCodeBench)