You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Would an EvalPort-format export of LiveCodeBench be welcome in EvalPort's benchmarks/? #150
Hi LiveCodeBench team — I maintain EvalPort, an open JSON spec for portable LLM eval datasets and results, whose benchmarks/ directory converts public benchmarks into validated EvalPort suites.
I read the real dataclass in this repo, lcb_runner/benchmarks/code_generation.py:
That maps onto EvalPort cleanly: question_content + starter_code -> TestCase.input, public_test_cases/private_test_cases -> TestCase.expected_output, graded with a code grader running the same stdin/functional check LiveCodeBench already uses, with contest_date preserved as metadata so the contamination-window filtering your release_v* versions already do can still work.
Would your maintainers be open to an EvalPort-format export (respecting your existing release date windows) living in EvalPort's benchmarks/, with attribution and license preserved? Happy to do the conversion and open a PR for review first. No worries if this isn't a fit.
Activity