Skip to content

feat: quarantine debt CI summary — surface suppressed tests in every run #5

@EmanueleMinotto

Description

@EmanueleMinotto

Opportunity

After every run, Skipper knows exactly which tests it suppressed, why, and until when. Today that information goes nowhere. The opportunity: surface a quarantine debt score in every CI run — making the cost of hiding tests visible and creating accountability pressure that no other tool in this space provides.

Proposed feature

Emit a structured quarantine report at the end of every JUnit run (via a TestExecutionListener / Jupiter extension), targeting two outputs:

  • GitHub Actions job summary (via GITHUB_STEP_SUMMARY)
  • A local skipper-report.json artifact for archiving and downstream tooling

The report includes: number of currently suppressed tests, tests expiring this week, tests re-enabled in this run, and a total quarantine-days of debt score (sum of disabledUntil - today across all active rows).

void emitSummary(QuarantineReport report) throws IOException {
    String summaryFile = System.getenv("GITHUB_STEP_SUMMARY");
    String md = buildMarkdownSummary(report);
    if (summaryFile != null) {
        Files.writeString(Path.of(summaryFile), md, StandardOpenOption.APPEND);
    } else {
        System.out.println(md);
    }
    Files.writeString(Path.of("skipper-report.json"), toJson(report));
}

Wire the emission into the existing Skipper listener/extension so it runs automatically under Gradle and Maven without extra config.

Why this is the unique moat

The quarantine-days of debt number grows when tests are disabled and shrinks when they expire. A team with 200 quarantine-days of debt is hiding a lot of reality. That number in every PR comment is a forcing function that costs zero additional tooling — and no other tool in this space does it.

Acceptance criteria

  • After every test run, a markdown summary is written to GITHUB_STEP_SUMMARY if the env var is set
  • If GITHUB_STEP_SUMMARY is not set, the summary is printed to stdout
  • skipper-report.json is written with the full suppression state of the run
  • Report includes: currently suppressed count, expiring this week, re-enabled this run, quarantine-days of debt
  • Quarantine-days of debt = sum of (disabledUntil - today) in days across all active disabled rows
  • Works under both Gradle and Maven without additional configuration

Effort estimate

~150 lines in the core module, no new dependencies (use java.nio.file + existing JSON lib).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions