From 2d231d08bfbe8bb62a8527536149f52b16a4956b Mon Sep 17 00:00:00 2001 From: Solal Pirelli Date: Thu, 4 Jun 2026 21:03:59 +0200 Subject: [PATCH 1/2] Minor evaluator guide improvements Was chatting with a colleague today who was frustrated at a (non-systems) AE because his artifact required a lot of RAM and evaluators didn't follow what he thought were basic practices. I figured this can be helpful to add to the guide just in case. --- evaluator-guide.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/evaluator-guide.md b/evaluator-guide.md index c7b4029..36666a0 100755 --- a/evaluator-guide.md +++ b/evaluator-guide.md @@ -88,7 +88,8 @@ Your evaluation plan should answer the following questions: - **Q3**: From a quick glance at the documentation and code comments, do you expect to reach basic understanding of the code? You will not need to understand every line, only what the high-level modules are, how they match paper concepts, and the overall flow of key algorithms if any. - If you need to execute the artifact - - **Q4**: Do you have access to the necessary hardware? + - **Q4**: Do you have access to appropriate hardware? + Check what the artifact states as its dependencies, which can be specialized hardware or simply specific amounts of RAM, CPU cores, and so on. - **Q5**: If the hardware is shared with other evaluators, who will use it when? - **Q6**: Can you install the artifact? - **Q7**: Can you run a simple experiment to completion? @@ -119,6 +120,7 @@ such as by finding the right package containing the dependency for your operatin However, it is the authors' responsibility to make their artifacts work, not yours. You do not need to spend hours trying to debug and fix complex issues. If you encounter a non-trivial error, first ask your fellow evaluators if they encountered it too or if they know how to fix it, then ask the authors to fix it. +Include a summary of your machine specs in any such bug report, such as how much RAM you have, what your CPU is, and so on. **It is acceptable to deny badges if artifacts require unreasonable effort**, especially if such effort could be avoided through automation. For instance, if reproducing a claim requires 50 points of data, and the artifact requires you to manually edit 5 config files then From e1afb9be852a75b6733a773bdfa1cfce9e3b1b46 Mon Sep 17 00:00:00 2001 From: Solal Pirelli Date: Thu, 4 Jun 2026 21:04:32 +0200 Subject: [PATCH 2/2] Markdown... --- evaluator-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evaluator-guide.md b/evaluator-guide.md index 36666a0..67ed7eb 100755 --- a/evaluator-guide.md +++ b/evaluator-guide.md @@ -88,7 +88,7 @@ Your evaluation plan should answer the following questions: - **Q3**: From a quick glance at the documentation and code comments, do you expect to reach basic understanding of the code? You will not need to understand every line, only what the high-level modules are, how they match paper concepts, and the overall flow of key algorithms if any. - If you need to execute the artifact - - **Q4**: Do you have access to appropriate hardware? + - **Q4**: Do you have access to appropriate hardware? Check what the artifact states as its dependencies, which can be specialized hardware or simply specific amounts of RAM, CPU cores, and so on. - **Q5**: If the hardware is shared with other evaluators, who will use it when? - **Q6**: Can you install the artifact?