Repository navigation
Some question #2
Description
Some Question
Hello author, I am very interested in your paper. I have read through the entire paper and am preparing to test gpt-3.5-turbo using your open-source code (by calling the API). However, I encountered some issues during implementation and would like to ask for your advice:
- When reading the dataset, I did not seem to find a method to divide the data in code generation into easy/medium/hard categories.
- For the time being, I can only perform tests on code generation and Test-output-prediction.
- The results I found when testing LLM for code generation are as follows:
{"pass@1": 0.0, "pass@5": 0.0, "detail": {"pass@1": {"7": 0.0, "1": 0.0, "3": 0.0, "4": 0.0, "9": 0.0, "2": 0.0, "8": 0.0, "0": 0.0, "6": 0.0, "5": 0.0}, "pass@5": {"7": 0.0, "1": 0.0, "3": 0.0, "4": 0.0, "9": 0.0, "2": 0.0, "8": 0.0, "0": 0.0, "6": 0.0, "5": 0.0}}}
The changes I made to main.py are as follows:
def main():
args = get_args()
if args.scenario == Scenario.codegeneration:
benchmark = load_code_generation_dataset()
format_prompt = format_prompt_generation
elif args.scenario == Scenario.testoutputprediction:
benchmark = load_test_prediction_dataset()
format_prompt = format_prompt_test_output
else:
raise ValueError(f"Scenario {args.scenario} not implemented")
runner = GPT3APIRunner()
results = runner.run_main(benchmark, format_prompt)
copy_benchmark = benchmark[:10]
samples = [problem.get_evaluation_sample() for problem in copy_benchmark]
metrics,result = codegen_metrics(samples,results)
How should I solve this problem? Thank you very much!
Activity
Naman-ntc commentedon Mar 21, 2024
Thanks for your interest in our benchmark. I have added easier support for computing more granular now.
On running
python -m lcb_runner.runner.main --model {model_name} --scenario codegeneration --evaluatewe now generate an_eval_all.jsonLiveCodeBench/lcb_runner/runner/main.py
Lines 78 to 90 in ad9c886
Once this file is constructed, you can get more detailed metrics using
python -m lcb_runner.evaluation.compute_scores --eval_all_file {saved_eval_all_file} # --start_date 2023-09-01Finally, I have added support for the
--continue_existingflag to use the existing model generations already to avoid the recomputation cost. Hope this helps!Please reach out if you have any further questions!