let me clarify based on the JUnit XML specification.
The tests and failures attributes on <testsuite> are defined at the <testcase> level, not at the individual lint-finding level:
-
tests is the total number of tests (i.e., <testcase> elements) in the suite, and failures is the total number of failed tests in the suite.
-
The failures attribute specifically means "the total number of tests in the suite that failed," where a failure is a test which the code has explicitly failed by using the mechanisms for that purpose.
In the current junit.def, cpplint's JUnit output collapses all lint findings for a given file into a single <testcase> with a single <failure> element whose text body contains all 123 findings concatenated. So the actual structure is:
- 2
<testcase> nodes (one per file: src/sillycode.cpp, src/sillycode.w)
- 2
<failure> nodes (one per testcase)
This means the correct suite-level counts should be tests="2" failures="2", not tests="123" failures="123". The value 123 represents the count of individual lint findings, which is not what these XML attributes are meant to track — JUnit consumers (e.g., Jenkins, GitHub Actions test reporters) will interpret tests="123" as meaning there are 123 separate test cases, leading to incorrect pass/fail metrics.
If you'd prefer to represent 123 individual findings, each one would need its own <testcase> node.
Originally posted by @coderabbitai[bot] in #429 (comment)
I'm currently hoping that just turning each error into a failure so the one test case has 123 failure elements under it would conform. Not sure if it does though.
let me clarify based on the JUnit XML specification.
The
testsandfailuresattributes on<testsuite>are defined at the<testcase>level, not at the individual lint-finding level:testsis the total number of tests (i.e.,<testcase>elements) in the suite, andfailuresis the total number of failed tests in the suite.The
failuresattribute specifically means "the total number of tests in the suite that failed," where a failure is a test which the code has explicitly failed by using the mechanisms for that purpose.In the current
junit.def, cpplint's JUnit output collapses all lint findings for a given file into a single<testcase>with a single<failure>element whose text body contains all 123 findings concatenated. So the actual structure is:<testcase>nodes (one per file:src/sillycode.cpp,src/sillycode.w)<failure>nodes (one per testcase)This means the correct suite-level counts should be
tests="2" failures="2", nottests="123" failures="123". The value 123 represents the count of individual lint findings, which is not what these XML attributes are meant to track — JUnit consumers (e.g., Jenkins, GitHub Actions test reporters) will interprettests="123"as meaning there are 123 separate test cases, leading to incorrect pass/fail metrics.If you'd prefer to represent 123 individual findings, each one would need its own
<testcase>node.Originally posted by @coderabbitai[bot] in #429 (comment)
I'm currently hoping that just turning each error into a failure so the one test case has 123 failure elements under it would conform. Not sure if it does though.