Skip to content
Prev Previous commit
Next Next commit
doc: add plan wait option to test runner documentation
  • Loading branch information
pmarchini committed Feb 14, 2025
commit 066559778e4dd1cb253a5e6d04addcd7d08d7c94
29 changes: 29 additions & 0 deletions doc/api/test.md
Original file line number Diff line number Diff line change
Expand Up @@ -3388,6 +3388,10 @@ added:
- v22.2.0
- v20.15.0
changes:
- version:
- REPLACEME
pr-url: https://github.com/nodejs/node/pull/56765
description: Add the `options` parameter.
- version:
- v23.4.0
- v22.13.0
Expand All @@ -3396,6 +3400,13 @@ changes:
-->

* `count` {number} The number of assertions and subtests that are expected to run.
* `options` {Object} _(Optional)_ Additional options for the plan.
Comment thread
pmarchini marked this conversation as resolved.
Outdated
* `wait` {boolean|number} The wait time for the plan:
* If `true`, the plan waits indefinitely for all assertions and subtests to run.
Comment thread
pmarchini marked this conversation as resolved.
* If a number, it specifies the maximum wait time in milliseconds
before timing out while waiting for expected assertions and subtests to be matched.
If the timeout is reached, the test will fail.
Comment on lines +3409 to +3411
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the maximal value be pointed out here?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you suggesting that there should be a maximum value possible for the wait?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsumners, yes, at the moment the maximum is const { TIMEOUT_MAX } = require('internal/timers');, here a partial reference #11198

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a need for it. If someone wants to set the timeout to Number.MAX_VALUE, then that's on them. I think most people will be reasonable and set it on the order of seconds.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO, validating the maximum value is a good way to inform the user of misuse.
I honestly would avoid reporting this maximum in the documentation, as has been done in this case https://github.com/nodejs/node/pull/56595/files .

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@atlowChemi @jsumners If you are okay with this, I would avoid documenting this implementation detail!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am good with the PR as it is.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll just address the last comment related to tests and restart the CIs

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pmarchini I left this comment while approving the PR - so this is non-blocking IMHO 🙂

**Default:** `false`.

This function is used to set the number of assertions and subtests that are expected to run
within the test. If the number of assertions and subtests that run does not match the
Expand Down Expand Up @@ -3434,6 +3445,24 @@ test('planning with streams', (t, done) => {
});
```

When using the `wait` option, you can control how long the test will wait for the expected assertions.
Comment thread
pmarchini marked this conversation as resolved.
For example, setting a maximum wait time ensures that the test will wait for asynchronous assertions
to complete within the specified timeframe:

```js
test('plan with wait: 2000 waits for async assertions', (t) => {
t.plan(1, { wait: 2000 }); // Waits for up to 2 seconds for the assertion to complete.

const asyncActivity = () => {
setTimeout(() => {
t.assert.ok(true, 'Async assertion completed within the wait time');
}, 1000); // Completes after 1 second, within the 2-second wait time.
};

asyncActivity(); // The test will pass because the assertion is completed in time.
});
```

### `context.runOnly(shouldRunOnlyTests)`

<!-- YAML
Expand Down
20 changes: 13 additions & 7 deletions lib/internal/test_runner/test.js
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ class TestPlan {
switch (typeof options.wait) {
case 'boolean':
this.wait = options.wait;
this.#waitIndefinitely = true;
this.#waitIndefinitely = options.wait;
break;
case 'number':
validateNumber(options.wait, 'options.wait', 0, TIMEOUT_MAX);
Expand Down Expand Up @@ -231,7 +231,7 @@ class TestPlan {

// If the plan has a maximum wait time, set a timeout.
// Otherwise, the plan will wait indefinitely.
if (!this.#waitIndefinitely) {
if (this.#shouldSetMaxWaitingTime()) {
maxWaitingTimeId = this.#setMaxWaitingTime(done);
}

Expand All @@ -250,7 +250,7 @@ class TestPlan {
#setMaxWaitingTime(done) {
return setTimeout(() => {
const err = new ERR_TEST_FAILURE(
`Plan timed out after ${this.wait}ms with ${this.actual} assertions when expecting ${this.expected}`,
`plan timed out after ${this.wait}ms with ${this.actual} assertions when expecting ${this.expected}`,
kTestTimeoutFailure,
);
// The poller has timed out; the test should fail.
Expand All @@ -264,10 +264,10 @@ class TestPlan {
return;
}
// The plan has not been met.
if (!this.#isWaitDefined()) {
if (!this.#shouldWait()) {
// If the plan doesn't have a wait time defined, the test should fail immediately.
throw new ERR_TEST_FAILURE(
`Plan expected ${this.expected} assertions but received ${this.actual}`,
`plan expected ${this.expected} assertions but received ${this.actual}`,
kTestCodeFailure,
);
}
Expand All @@ -278,8 +278,12 @@ class TestPlan {
this.actual++;
}

#isWaitDefined() {
return this.wait !== undefined;
#shouldWait() {
return this.wait !== undefined && this.wait !== false;
}

#shouldSetMaxWaitingTime() {
return !this.#waitIndefinitely && this.wait !== false;
}
}

Expand Down Expand Up @@ -528,6 +532,8 @@ class Test extends AsyncResource {
super('Test');

let { fn, name, parent } = options;
// TODO(pmarchini): The plan has additional options that a user can set via t.plan.
Comment thread
pmarchini marked this conversation as resolved.
Outdated
// We should allow users to set these options via the options object for consistency.
const { concurrency, entryFile, loc, only, timeout, todo, skip, signal, plan } = options;

if (typeof fn !== 'function') {
Expand Down
12 changes: 12 additions & 0 deletions test/fixtures/test-runner/output/test-runner-plan-timeout.js
Original file line number Diff line number Diff line change
Expand Up @@ -62,4 +62,16 @@ describe('planning with wait', () => {

asyncActivity();
});

it(`planning with wait "options.wait : false" should not wait`, async (t) => {
t.plan(1, { wait: false });

const asyncActivity = () => {
setTimeout(() => {
t.assert.ok(true);
}, platformTimeout(500_000));
};

asyncActivity();
})
});
18 changes: 14 additions & 4 deletions test/fixtures/test-runner/output/test-runner-plan-timeout.snapshot
Original file line number Diff line number Diff line change
Expand Up @@ -66,21 +66,31 @@ TAP version 13
*
*
...
1..5
# Subtest: planning with wait "options.wait : false" should not wait
not ok 6 - planning with wait "options.wait : false" should not wait
---
duration_ms: *
type: 'test'
location: '/test/fixtures/test-runner/output/test-runner-plan-timeout.js:(LINE):3'
failureType: 'testCodeFailure'
error: 'plan expected 1 assertions but received 0'
code: 'ERR_TEST_FAILURE'
...
1..6
not ok 1 - planning with wait
---
duration_ms: *
type: 'suite'
location: '/test/fixtures/test-runner/output/test-runner-plan-timeout.js:(LINE):1'
failureType: 'subtestsFailed'
error: '3 subtests failed'
error: '4 subtests failed'
code: 'ERR_TEST_FAILURE'
...
1..1
# tests 5
# tests 6
# suites 1
# pass 2
# fail 2
# fail 3
# cancelled 1
# skipped 0
# todo 0
Expand Down