We currently use labels to track the evaluation status of transcript reviews on the GitHub Pull Requests page, but this process is done manually. To make the Pull Requests page more effective for evaluators, we need to automate the addition of these labels.

By automating the tagging and tracking process, we enhance the monitoring capabilities within the GitHub Pull Requests page and we can then extend this functionality to the new monitoring page as well as the account page of each reviewer in order to provide a better overview of the status of their submission.

Automating these processes will lay the foundation for future enhancements that will streamline the review process and improve productivity for both evaluators and reviewers.

Evaluation Cycle Information

The information needed is related to the evaluation cycle, which includes:

1. Ready for Evaluation: Default status when a review is submitted.
2. Waiting for Changes: Status after an evaluator requests changes from the reviewer.
3. Changes Delayed: Status when changes requested by the evaluator have not been responded to by the reviewer for X weeks.
4. Changes Overdue: Changes have not been responded to by the reviewer for Y weeks. (Y > X)

Implementation Details

Create an evaluation bot to manage the evaluation states by assigning appropriate labels like "Ready for Evaluation", "Waiting for Changes", "Changes Delayed", and "Changes Overdue" based on the predefined scenarios.

Events to Track:

• User opens a new PR => "Ready for Evaluation"
• Evaluator submits a PR review Requesting Changes => "Waiting for Changes"
• Reviewers submits changes => Back to "Ready for Evaluation"
• X days have passed since the last evaluation/PR review => "Changes Delayed"
• Y days have passed => "Changes Overdue"

Notes

• This evaluator bot will reside in its own repository.
• Further research needs to be conducted to determine the best implementation method, with an initial suggestion of GitHub apps or Probot.