Skip to content

Commit 866c01a

Browse files
mpkorstanjemarcphilipp
mpkorstanje
and
marcphilipp
committed
Add note about duplicate test execution with @suite (#5080)
Users are surprised to see that their tests are executed twice[1]. This is the natural consequence of using a test engine to define a suite. We can at least let them know in the documentation. 1. #4327 Co-authored-by: Marc Philipp <mail@marcphilipp.de> (cherry picked from commit c08e421)
1 parent de88e88 commit 866c01a

9 files changed

Lines changed: 127 additions & 10 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -177,8 +177,8 @@ To deprecate an item:
177177

178178
## Building the Project
179179

180-
Please refer to [the readme](README.md#building-from-source) for the most common
181-
build commands.
180+
Please refer to [the readme](README.md#building-from-source) and [the documentation readme](documentation/README.md) for
181+
the most common build commands.
182182

183183
### Build Parameters
184184

‎documentation/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ This following Gradle command generates the HTML version of the User Guide as
1818
`build/docs/asciidoc/user-guide/index.html`.
1919

2020
```
21-
gradlew asciidoctor
21+
./gradlew asciidoctor
2222
```
2323

2424
On Linux operating systems, the `graphviz` package providing `/usr/bin/dot` must be
@@ -30,5 +30,5 @@ This following Gradle command generates the PDF version of the User Guide to
3030
`build/docs/asciidocPdf/user-guide/index.pdf`.
3131

3232
```
33-
gradlew asciidoctorPdf
33+
./gradlew asciidoctorPdf
3434
```

‎documentation/documentation.gradle.kts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -269,7 +269,7 @@ tasks {
269269
outputs.cacheIf { true }
270270
}
271271

272-
val componentDiagram = plantUml.flatMap { it.outputDirectory.file("component-diagram.svg") }
272+
val plantUmlOutputDirectory = plantUml.flatMap { it.outputDirectory }
273273

274274
withType<AbstractAsciidoctorTask>().configureEach {
275275
inputs.files(
@@ -279,15 +279,15 @@ tasks {
279279
generateConsoleLauncherEnginesOptions,
280280
generateApiTables,
281281
generateStandaloneConsoleLauncherShadowedArtifactsFile,
282-
componentDiagram
282+
plantUmlOutputDirectory
283283
)
284284

285285
resources {
286286
from(sourceDir) {
287287
include("**/images/**/*.png")
288288
include("**/images/**/*.svg")
289289
}
290-
from(componentDiagram) {
290+
from(plantUmlOutputDirectory) {
291291
into("user-guide/images")
292292
}
293293
}

‎documentation/src/docs/asciidoc/user-guide/advanced-topics/junit-platform-suite-engine.adoc‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
[[junit-platform-suite-engine]]
22
=== JUnit Platform Suite Engine
33

4-
The JUnit Platform supports the declarative definition and execution of suites of tests
5-
from _any_ test engine using the JUnit Platform.
4+
The Suite Engine supports the declarative selection and execution of tests from
5+
_any_ test engine on the JUnit Platform using the <<launcher-api>>.
6+
7+
image::junit-platform-suite-engine-diagram.svg[role=text-center]
68

79
[[junit-platform-suite-engine-setup]]
810
==== Setup
@@ -58,3 +60,19 @@ all tests of the test suite.
5860
----
5961
include::{testDir}/example/BeforeAndAfterSuiteDemo.java[tags=user_guide]
6062
----
63+
64+
[[junit-platform-suite-engine-duplicate-test-execution]]
65+
==== Duplicate test execution
66+
67+
Depending on the declared test selection, different suites may contain the same tests,
68+
potentially with different configurations.
69+
Moreover, tests in a suite are executed in addition to the tests executed by every other
70+
test engine. This can result in the same tests being executed twice.
71+
72+
image::junit-platform-suite-engine-duplicate-test-execution-diagram.svg[role=text-center]
73+
74+
This can be prevented by configuring your build tool to only include the
75+
`junit-platform-suite` engine. Or by using a naming pattern. For example, name all suites
76+
`*Suite` and all tests `*Test` and configure your build tool to only include the former.
77+
78+
Alternatively, consider <<running-tests-tags, using tags>> to select specific groups of tests.

‎documentation/src/docs/asciidoc/user-guide/advanced-topics/launcher-api.adoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ tests. Moreover, third party test libraries – like Spock or Cucumber – can p
1414
JUnit Platform's launching infrastructure by providing a custom
1515
<<test-engines,TestEngine>>.
1616

17+
image::launcher-api-diagram.svg[role=text-center]
18+
1719
The launcher API is in the `{junit-platform-launcher}` module.
1820

1921
An example consumer of the launcher API is the `{ConsoleLauncher}` in the

‎documentation/src/plantuml/junit-platform-suite-engine-diagram.puml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
@startuml
2+
object "IDEs"
3+
object "Build Tools"
4+
object "Console Launcher"
5+
6+
object "JUnit Platform"
7+
object "Suite Test Engine"
8+
9+
object "@Suite annotated class A"
10+
object "@Suite annotated class B"
11+
object "JUnit Platform (A)"
12+
object "JUnit Platform (B)"
13+
object "Jupiter Test Engine (A)"
14+
object "Jupiter Test Engine (B)"
15+
object "Tests in package A"
16+
object "Tests in package B"
17+
18+
"IDEs" --> "JUnit Platform"
19+
"Build Tools" --> "JUnit Platform"
20+
"Console Launcher" --> "JUnit Platform" : requests discovery and execution
21+
"JUnit Platform" --> "Suite Test Engine": forwards request
22+
23+
"Suite Test Engine" --> "@Suite annotated class A"
24+
"@Suite annotated class A" --> "JUnit Platform (A)"
25+
"JUnit Platform (A)" --> "Jupiter Test Engine (A)"
26+
"Jupiter Test Engine (A)" --> "Tests in package A"
27+
28+
"Suite Test Engine" --> "@Suite annotated class B" : discovers and executes
29+
"@Suite annotated class B" --> "JUnit Platform (B)" : requests discovery and execution
30+
"JUnit Platform (B)" --> "Jupiter Test Engine (B)" : forwards request
31+
"Jupiter Test Engine (B)" --> "Tests in package B" : discovers and executes
32+
@enduml

‎documentation/src/plantuml/junit-platform-suite-engine-duplicate-test-execution-diagram.puml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
@startuml
2+
object "IDEs"
3+
object "Build Tools"
4+
object "Console Launcher"
5+
object "JUnit Platform"
6+
together {
7+
object "Suite Test Engine"
8+
object "Jupiter Test Engine"
9+
}
10+
object "@Suite annotated class"
11+
object "JUnit Platform (@Suite)"
12+
object "Jupiter Test Engine (@Suite)"
13+
together {
14+
object "Example Test A"
15+
object "Example Test A (@Suite)"
16+
}
17+
18+
"IDEs" --> "JUnit Platform"
19+
"Build Tools" --> "JUnit Platform"
20+
"Console Launcher" --> "JUnit Platform" : requests discovery and execution
21+
22+
"JUnit Platform" --> "Suite Test Engine"
23+
"Suite Test Engine" --> "@Suite annotated class" : discovers and executes
24+
"@Suite annotated class" --> "JUnit Platform (@Suite)" : requests discovery and execution
25+
"JUnit Platform (@Suite)" --> "Jupiter Test Engine (@Suite)" : forwards request
26+
"Jupiter Test Engine (@Suite)" --> "Example Test A (@Suite)" : discovers and executes
27+
28+
"JUnit Platform" --> "Jupiter Test Engine": forwards request
29+
"Jupiter Test Engine" --> "Example Test A" #line:red;line.bold;text:red : also discover and executes!
30+
@enduml

‎documentation/src/plantuml/launcher-api-diagram.puml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
@startuml
2+
object "IDEs"
3+
object "Build Tools"
4+
object "Console Launcher"
5+
object "JUnit Platform"
6+
object "Jupiter Test Engine"
7+
object "Cucumber Test Engine"
8+
object "Spock Test Engine"
9+
object "Test Classes"
10+
object "Feature Files"
11+
object "Specifications"
12+
13+
"IDEs" --> "JUnit Platform"
14+
"Build Tools" --> "JUnit Platform"
15+
"Console Launcher" --> "JUnit Platform" : requests discovery and execution
16+
"JUnit Platform" --> "Jupiter Test Engine"
17+
"Jupiter Test Engine" --> "Test Classes"
18+
19+
"JUnit Platform" --> "Cucumber Test Engine"
20+
"Cucumber Test Engine" --> "Feature Files"
21+
22+
"JUnit Platform" --> "Spock Test Engine": forwards request
23+
"Spock Test Engine" --> "Specifications" : discovers and executes
24+
@enduml

‎junit-platform-suite-api/src/main/java/org/junit/platform/suite/api/Suite.java‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626
* {@code @Suite} marks a class as a test suite on the JUnit Platform.
2727
*
2828
* <p>Selector and filter annotations are used to control the contents of the
29-
* suite. Additionally configuration can be passed to the suite via the
29+
* suite. Additionally, configuration can be passed to the suite via the
3030
* configuration annotations.
3131
*
3232
* <p>When the {@link IncludeClassNamePatterns @IncludeClassNamePatterns}
@@ -44,6 +44,17 @@
4444
* annotation disables the latter as a source of parameters so that only explicit
4545
* configuration parameters are taken into account.
4646
*
47+
* <p>Note: Depending on the declared test selection, different suites may contain the
48+
* same tests, potentially with different configurations.
49+
* Moreover, tests in a suite are executed in addition to the tests executed by every
50+
* other test engine. This can result in the same tests being executed twice and can be
51+
* prevented by configuring your build tool to only include the
52+
* {@code junit-platform-suite} engine. Or by using a naming pattern. For example, name
53+
* all suites {@code *Suite} and all tests {@code *Test} and configure your build tool
54+
* to only include the former.
55+
*
56+
* <p>Alternatively, consider using tags to select specific groups of tests.
57+
*
4758
* @since 1.8
4859
* @see Select
4960
* @see SelectClasses

0 commit comments

Comments
 (0)