Skip to content

Commit d14029b

Browse files
pmarchiniaduh95
pmarchini
authored and
aduh95
committed
test_runner: support test order randomization
PR-URL: #61747 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com> Reviewed-By: Ethan Arrowood <ethan@arrowood.dev> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Aviv Keller <me@aviv.sh> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
1 parent ec8c6b9 commit d14029b

28 files changed

Lines changed: 1553 additions & 21 deletions

‎doc/api/cli.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2669,6 +2669,38 @@ changes:
26692669
Configures the test runner to only execute top level tests that have the `only`
26702670
option set. This flag is not necessary when test isolation is disabled.
26712671

2672+
### `--test-random-seed`
2673+
2674+
<!-- YAML
2675+
added: REPLACEME
2676+
-->
2677+
2678+
Set the seed used to randomize test execution order. This applies to both test
2679+
file execution order and queued tests within each file. Providing this flag
2680+
enables randomization implicitly, even without `--test-randomize`.
2681+
2682+
The value must be an integer between `0` and `4294967295`.
2683+
2684+
This flag cannot be used with `--watch` or `--test-rerun-failures`.
2685+
2686+
### `--test-randomize`
2687+
2688+
<!-- YAML
2689+
added: REPLACEME
2690+
-->
2691+
2692+
Randomize test execution order. This applies to both test file execution order
2693+
and queued tests within each file. This can help detect tests that rely on
2694+
shared state or execution order.
2695+
2696+
The seed used for randomization is printed in the test summary and can be
2697+
reused with `--test-random-seed`.
2698+
2699+
For detailed behavior and examples, see
2700+
[randomizing tests execution order][].
2701+
2702+
This flag cannot be used with `--watch` or `--test-rerun-failures`.
2703+
26722704
### `--test-reporter`
26732705

26742706
<!-- YAML
@@ -3582,6 +3614,8 @@ one is included in the list below.
35823614
* `--test-isolation`
35833615
* `--test-name-pattern`
35843616
* `--test-only`
3617+
* `--test-random-seed`
3618+
* `--test-randomize`
35853619
* `--test-reporter-destination`
35863620
* `--test-reporter`
35873621
* `--test-rerun-failures`
@@ -4162,6 +4196,7 @@ node --stack-trace-limit=12 -p -e "Error.stackTraceLimit" # prints 12
41624196
[libuv threadpool documentation]: https://docs.libuv.org/en/latest/threadpool.html
41634197
[module compile cache]: module.md#module-compile-cache
41644198
[preloading asynchronous module customization hooks]: module.md#registration-of-asynchronous-customization-hooks
4199+
[randomizing tests execution order]: test.md#randomizing-tests-execution-order
41654200
[remote code execution]: https://www.owasp.org/index.php/Code_Injection
41664201
[running tests from the command line]: test.md#running-tests-from-the-command-line
41674202
[scavenge garbage collector]: https://v8.dev/blog/orinoco-parallel-scavenger

‎doc/api/test.md‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -626,6 +626,94 @@ prevent shell expansion, which can reduce portability across systems.
626626
node --test "**/*.test.js" "**/*.spec.js"
627627
```
628628

629+
### Randomizing tests execution order
630+
631+
<!-- YAML
632+
added: REPLACEME
633+
-->
634+
635+
> Stability: 1.0 - Early development
636+
637+
The test runner can randomize execution order to help detect
638+
order-dependent tests. When enabled, the runner randomizes both discovered
639+
test files and queued tests within each file. Use `--test-randomize` to
640+
enable this mode.
641+
642+
```bash
643+
node --test --test-randomize
644+
```
645+
646+
When randomization is enabled, the test runner prints the seed used for the run
647+
as a diagnostic message:
648+
649+
```text
650+
Randomized test order seed: 12345
651+
```
652+
653+
Use `--test-random-seed=<number>` to replay the same randomized order
654+
deterministically. Supplying `--test-random-seed` also enables randomization,
655+
so `--test-randomize` is optional when a seed is provided:
656+
657+
```bash
658+
node --test --test-random-seed=12345
659+
```
660+
661+
In most test files, randomization works automatically. One important exception
662+
is when subtests are awaited one by one. In that pattern, each subtest starts
663+
only after the previous one finishes, so the runner keeps declaration order
664+
instead of randomizing it.
665+
666+
Example: this runs sequentially and is **not** randomized.
667+
668+
```mjs
669+
import test from 'node:test';
670+
671+
test('math', async (t) => {
672+
for (const name of ['adds', 'subtracts', 'multiplies']) {
673+
// Sequentially awaiting each subtest preserves declaration order.
674+
await t.test(name, async () => {});
675+
}
676+
});
677+
```
678+
679+
```cjs
680+
const test = require('node:test');
681+
682+
test('math', async (t) => {
683+
for (const name of ['adds', 'subtracts', 'multiplies']) {
684+
// Sequentially awaiting each subtest preserves declaration order.
685+
await t.test(name, async () => {});
686+
}
687+
});
688+
```
689+
690+
Using suite-style APIs such as `describe()`/`it()` or `suite()`/`test()`
691+
still allows randomization, because sibling tests are enqueued together.
692+
693+
Example: this remains eligible for randomization.
694+
695+
```mjs
696+
import { describe, it } from 'node:test';
697+
698+
describe('math', () => {
699+
it('adds', () => {});
700+
it('subtracts', () => {});
701+
it('multiplies', () => {});
702+
});
703+
```
704+
705+
```cjs
706+
const { describe, it } = require('node:test');
707+
708+
describe('math', () => {
709+
it('adds', () => {});
710+
it('subtracts', () => {});
711+
it('multiplies', () => {});
712+
});
713+
```
714+
715+
`--test-randomize` and `--test-random-seed` are not supported with `--watch` mode.
716+
629717
Matching files are executed as test files.
630718
More information on the test file execution can be found
631719
in the [test runner execution model][] section.
@@ -666,6 +754,10 @@ test runner functionality:
666754
* `--test-reporter` - Reporting is managed by the parent process
667755
* `--test-reporter-destination` - Output destinations are controlled by the parent
668756
* `--experimental-config-file` - Config file paths are managed by the parent
757+
* `--test-randomize` - Randomization is managed by the parent process and
758+
propagated to child processes
759+
* `--test-random-seed` - Randomization seed is managed by the parent process and
760+
propagated to child processes
669761

670762
All other Node.js options from command line arguments, environment variables,
671763
and configuration files are inherited by the child processes.
@@ -1572,6 +1664,14 @@ changes:
15721664
that specifies the index of the shard to run. This option is _required_.
15731665
* `total` {number} is a positive integer that specifies the total number
15741666
of shards to split the test files to. This option is _required_.
1667+
* `randomize` {boolean} Randomize execution order for test files and queued tests.
1668+
This option is not supported with `watch: true`.
1669+
**Default:** `false`.
1670+
* `randomSeed` {number} Seed used when randomizing execution order. If this
1671+
option is set, runs can replay the same randomized order deterministically,
1672+
and setting this option also enables randomization. The value must be an
1673+
integer between `0` and `4294967295`.
1674+
**Default:** `undefined`.
15751675
* `rerunFailuresFilePath` {string} A file path where the test runner will
15761676
store the state of the tests to allow rerunning only the failed tests on a next run.
15771677
see \[Rerunning failed tests]\[] for more information.

‎doc/node-config-schema.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -524,6 +524,14 @@
524524
"type": "boolean",
525525
"description": "run tests with 'only' option set"
526526
},
527+
"test-random-seed": {
528+
"type": "number",
529+
"description": "seed used to randomize test execution order"
530+
},
531+
"test-randomize": {
532+
"type": "boolean",
533+
"description": "run tests in a random order"
534+
},
527535
"test-reporter": {
528536
"oneOf": [
529537
{
@@ -902,6 +910,14 @@
902910
"type": "boolean",
903911
"description": "run tests with 'only' option set"
904912
},
913+
"test-random-seed": {
914+
"type": "number",
915+
"description": "seed used to randomize test execution order"
916+
},
917+
"test-randomize": {
918+
"type": "boolean",
919+
"description": "run tests in a random order"
920+
},
905921
"test-reporter": {
906922
"oneOf": [
907923
{

‎doc/node.1‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -498,6 +498,22 @@ Configures the type of test isolation used in the test runner.
498498
A regular expression that configures the test runner to only execute tests
499499
whose name matches the provided pattern.
500500
.
501+
.It Fl -test-random-seed
502+
Set the seed used to randomize test execution order.
503+
This applies to both test file execution order and queued tests within each file.
504+
Providing this flag enables randomization implicitly, even without
505+
\fB--test-randomize\fR.
506+
The value must be an integer between 0 and 4294967295.
507+
This flag cannot be used with \fB--watch\fR or \fB--test-rerun-failures\fR.
508+
.
509+
.It Fl -test-randomize
510+
Randomize test execution order.
511+
This applies to both test file execution order and queued tests within each file.
512+
This can help detect tests that rely on shared state or execution order.
513+
The seed used for randomization is printed in the test summary and can be
514+
reused with \fB--test-random-seed\fR.
515+
This flag cannot be used with \fB--watch\fR or \fB--test-rerun-failures\fR.
516+
.
501517
.It Fl -test-reporter
502518
A test reporter to use when running tests.
503519
.

‎lib/internal/test_runner/runner.js‎

Lines changed: 73 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ const {
5959
validateObject,
6060
validateOneOf,
6161
validateInteger,
62+
validateUint32,
6263
validateString,
6364
validateStringArray,
6465
} = require('internal/validators');
@@ -84,6 +85,7 @@ const {
8485
const { FastBuffer } = require('internal/buffer');
8586

8687
const {
88+
createRandomSeed,
8789
convertStringToRegExp,
8890
countCompletedTest,
8991
kDefaultPattern,
@@ -105,12 +107,14 @@ const kIsolatedProcessName = Symbol('kIsolatedProcessName');
105107
const kFilterArgs = [
106108
'--test',
107109
'--experimental-test-coverage',
110+
'--test-randomize',
108111
'--watch',
109112
'--experimental-default-config-file',
110113
];
111114
const kFilterArgValues = [
112115
'--test-reporter',
113116
'--test-reporter-destination',
117+
'--test-random-seed',
114118
'--experimental-config-file',
115119
];
116120
const kDiagnosticsFilterArgs = ['tests', 'suites', 'pass', 'fail', 'cancelled', 'skipped', 'todo', 'duration_ms'];
@@ -168,6 +172,8 @@ function getRunArgs(path, { forceExit,
168172
argv: suppliedArgs,
169173
execArgv,
170174
rerunFailuresFilePath,
175+
randomize,
176+
randomSeed,
171177
root: { timeout },
172178
cwd }) {
173179
const processNodeOptions = getOptionsAsFlagsFromBinding();
@@ -209,6 +215,12 @@ function getRunArgs(path, { forceExit,
209215
if (rerunFailuresFilePath) {
210216
ArrayPrototypePush(runArgs, `--test-rerun-failures=${rerunFailuresFilePath}`);
211217
}
218+
if (randomize) {
219+
ArrayPrototypePush(runArgs, '--test-randomize');
220+
}
221+
if (randomSeed != null) {
222+
ArrayPrototypePush(runArgs, `--test-random-seed=${randomSeed}`);
223+
}
212224

213225
ArrayPrototypePushApply(runArgs, execArgv);
214226

@@ -646,6 +658,8 @@ function run(options = kEmptyObject) {
646658
lineCoverage = 0,
647659
branchCoverage = 0,
648660
functionCoverage = 0,
661+
randomize: suppliedRandomize,
662+
randomSeed: suppliedRandomSeed,
649663
execArgv = [],
650664
argv = [],
651665
cwd = process.cwd(),
@@ -674,6 +688,56 @@ function run(options = kEmptyObject) {
674688
if (globPatterns != null) {
675689
validateArray(globPatterns, 'options.globPatterns');
676690
}
691+
if (suppliedRandomize != null) {
692+
validateBoolean(suppliedRandomize, 'options.randomize');
693+
}
694+
if (suppliedRandomSeed != null) {
695+
validateUint32(suppliedRandomSeed, 'options.randomSeed');
696+
}
697+
let randomize = suppliedRandomize;
698+
let randomSeed = suppliedRandomSeed;
699+
700+
if (randomSeed != null) {
701+
randomize = true;
702+
}
703+
if (watch) {
704+
if (randomSeed != null) {
705+
throw new ERR_INVALID_ARG_VALUE(
706+
'options.randomSeed',
707+
randomSeed,
708+
'is not supported with watch mode',
709+
);
710+
}
711+
if (randomize) {
712+
throw new ERR_INVALID_ARG_VALUE(
713+
'options.randomize',
714+
randomize,
715+
'is not supported with watch mode',
716+
);
717+
}
718+
}
719+
if (rerunFailuresFilePath) {
720+
validatePath(rerunFailuresFilePath, 'options.rerunFailuresFilePath');
721+
// TODO(pmarchini): Support rerun-failures with randomization by
722+
// persisting the randomization seed in the rerun state file.
723+
if (randomSeed != null) {
724+
throw new ERR_INVALID_ARG_VALUE(
725+
'options.randomSeed',
726+
randomSeed,
727+
'is not supported with rerun failures mode',
728+
);
729+
}
730+
if (randomize) {
731+
throw new ERR_INVALID_ARG_VALUE(
732+
'options.randomize',
733+
randomize,
734+
'is not supported with rerun failures mode',
735+
);
736+
}
737+
}
738+
if (randomize) {
739+
randomSeed ??= createRandomSeed();
740+
}
677741

678742
validateString(cwd, 'options.cwd');
679743

@@ -683,10 +747,6 @@ function run(options = kEmptyObject) {
683747
);
684748
}
685749

686-
if (rerunFailuresFilePath) {
687-
validatePath(rerunFailuresFilePath, 'options.rerunFailuresFilePath');
688-
}
689-
690750
if (shard != null) {
691751
validateObject(shard, 'options.shard');
692752
// Avoid re-evaluating the shard object in case it's a getter
@@ -783,11 +843,18 @@ function run(options = kEmptyObject) {
783843
functionCoverage: functionCoverage,
784844
cwd,
785845
globalSetupPath,
846+
randomize,
847+
randomSeed,
786848
};
849+
787850
const root = createTestTree(rootTestOptions, globalOptions);
788851
let testFiles = files ?? createTestFileList(globPatterns, cwd);
789852
const { isTestRunner } = globalOptions;
790853

854+
if (randomize) {
855+
root.diagnostic(`Randomized test order seed: ${randomSeed}`);
856+
}
857+
791858
if (shard) {
792859
testFiles = ArrayPrototypeFilter(testFiles, (_, index) => index % shard.total === shard.index - 1);
793860
}
@@ -833,6 +900,8 @@ function run(options = kEmptyObject) {
833900
rerunFailuresFilePath,
834901
env,
835902
workerIdPool: isolation === 'process' ? workerIdPool : null,
903+
randomize,
904+
randomSeed,
836905
};
837906

838907
if (isolation === 'process') {

0 commit comments

Comments
 (0)