Skip to content

JUnit support doesn't count amount of tests correctly #431

@aaronliu0130

Description

@aaronliu0130

let me clarify based on the JUnit XML specification.

The tests and failures attributes on <testsuite> are defined at the <testcase> level, not at the individual lint-finding level:

  • tests is the total number of tests (i.e., <testcase> elements) in the suite, and failures is the total number of failed tests in the suite.

  • The failures attribute specifically means "the total number of tests in the suite that failed," where a failure is a test which the code has explicitly failed by using the mechanisms for that purpose.

In the current junit.def, cpplint's JUnit output collapses all lint findings for a given file into a single <testcase> with a single <failure> element whose text body contains all 123 findings concatenated. So the actual structure is:

  • 2 <testcase> nodes (one per file: src/sillycode.cpp, src/sillycode.w)
  • 2 <failure> nodes (one per testcase)

This means the correct suite-level counts should be tests="2" failures="2", not tests="123" failures="123". The value 123 represents the count of individual lint findings, which is not what these XML attributes are meant to track — JUnit consumers (e.g., Jenkins, GitHub Actions test reporters) will interpret tests="123" as meaning there are 123 separate test cases, leading to incorrect pass/fail metrics.

If you'd prefer to represent 123 individual findings, each one would need its own <testcase> node.

Originally posted by @coderabbitai[bot] in #429 (comment)

I'm currently hoping that just turning each error into a failure so the one test case has 123 failure elements under it would conform. Not sure if it does though.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions