Skip to content

Would an EvalPort-format export of LiveCodeBench be welcome in EvalPort's benchmarks/? #150

Open

Description

@adhabnr-ux

Hi LiveCodeBench team — I maintain EvalPort, an open JSON spec for portable LLM eval datasets and results, whose benchmarks/ directory converts public benchmarks into validated EvalPort suites.

I read the real dataclass in this repo, lcb_runner/benchmarks/code_generation.py:

class CodeGenerationProblem:
    question_title: str
    question_content: str
    platform: Platform       # leetcode / codeforces / atcoder
    question_id: str
    contest_id: str
    contest_date: datetime
    starter_code: str
    difficulty: Difficulty
    public_test_cases: list[Test]
    private_test_cases: list[Test]
    metadata: dict

That maps onto EvalPort cleanly: question_content + starter_code -> TestCase.input, public_test_cases/private_test_cases -> TestCase.expected_output, graded with a code grader running the same stdin/functional check LiveCodeBench already uses, with contest_date preserved as metadata so the contamination-window filtering your release_v* versions already do can still work.

Would your maintainers be open to an EvalPort-format export (respecting your existing release date windows) living in EvalPort's benchmarks/, with attribution and license preserved? Happy to do the conversion and open a PR for review first. No worries if this isn't a fit.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions