Skip to content

ci: shard tests and merge Vitest blob reports - #11233

Draft
Mrtenz wants to merge 12 commits into
mainfrom
mrtenz/test-sharding
Draft

Mrtenz wants to merge 12 commits into
mainfrom
mrtenz/test-sharding

Conversation

@Mrtenz

@Mrtenz Mrtenz commented Nov 2, 2026

Copy link
Copy Markdown
Member

Explanation

The test workflow ran one job per package for each of Node 22, 24, and 26 (test-22, test-24, test-26). The jobs were duplicated to stay under GitHub's 256-job matrix limit, and every package ran regardless of how long it took.

This replaces them with a single test job that runs vitest --shard for every supported Node version. Each shard uploads its Vitest blob report. For each Node version, a merge-test-reports job combines the reports with vitest --merge-reports into one result, including failure details.

The shard count is defined once, in the prepare-environment job (renamed from get-node-versions), and is used for both the matrix and the --shard argument. This also adds .vitest/ to .gitignore, where Vitest writes the blob reports.

References

None.

Checklist

  • I've updated the test suite for new or updated code as appropriate
  • I've updated documentation (JSDoc, Markdown, etc.) for new or updated code as appropriate
  • I've communicated my changes to consumers by updating changelogs for packages I've changed
  • I've introduced breaking changes in this PR and have prepared draft pull requests for clients and consumer packages to resolve them

@Mrtenz
Mrtenz added this pull request to stack #11234 November 2, 2026 10:58
@Mrtenz
Mrtenz force-pushed the mrtenz/test-sharding branch from 676a083 to d789f46 Compare November 2, 2026 13:43
@Mrtenz
Mrtenz removed this pull request from stack #11234 November 2, 2026 13:45
@Mrtenz
Mrtenz added this pull request to stack #11236 November 2, 2026 13:45
@Mrtenz
Mrtenz force-pushed the mrtenz/test-sharding branch from d789f46 to 6e186f0 Compare November 2, 2026 13:47
@Mrtenz
Mrtenz force-pushed the mrtenz/test-sharding branch 2 times, most recently from 47b8e37 to 393627a Compare November 2, 2026 20:20
@Mrtenz
Mrtenz force-pushed the mrtenz/test-sharding branch from 393627a to 33ac410 Compare November 3, 2026 09:55
Base automatically changed from mrtenz/dynamic-node-versions to main November 3, 2026 10:15
@Mrtenz
Mrtenz force-pushed the mrtenz/test-sharding branch 2 times, most recently from 5aa7016 to e9fd789 Compare November 3, 2026 19:55
Run the full suite through `vitest --shard` across 10 shards and every supported Node version, replacing the per-package `test-22`, `test-24`, and `test-26` jobs. The shard count and Node versions come from a renamed `prepare-environment` job, so the count is defined in one place.

Each shard uploads its blob report. A `merge-test-reports` job for each Node version combines them with `vitest --merge-reports` into a single result. This also adds `.vitest/` to `.gitignore`.
Pass `--silent` to `vitest --merge-reports` so the `merge-test-reports` job does not replay console output from the tests. The shard jobs already print it.
The package isn't tested in CI and its tests currently fail. The `test` job now runs `vitest --shard` from the root, so it would pick up the package and fail the run. Leave it out of `projects`.
Set `VITEST_BLOB_LABEL` to the Node version in the `test` job. The label goes into each blob's filename and tags its test files, so reports from different Node versions no longer overwrite each other when merged. A single `merge-test-reports` job can now combine every blob, and the merged output shows which Node version a failure came from.
Add the `github-actions` reporter to the `vitest --merge-reports` step. Vitest enables it automatically only when no `--reporter` is passed, so our explicit `--reporter default` turned it off. The reporter writes a test report to `GITHUB_STEP_SUMMARY` and adds inline annotations for failures.
Each job spends about 25 seconds on setup, and the whole suite takes about 9 minutes of test time, measured on a sharded run. Doubling the shard count cuts the slowest shard from roughly 80 to 55 seconds. That is still about six times fewer jobs than the old one-job-per-package layout.
When a PR doesn't change any root files, the `test` job now writes a temporary
Vitest config that lists only the changed packages and their dependants, and
runs the shards against that. Full runs still use the root config. The `test`
job is skipped when no packages changed, and `merge-test-reports` is skipped
with it. `--passWithNoTests` stops shards that end up with no test files from
failing.
Running 20 shards for a PR that touches one package leaves most of them with
no test files, and each still pays for checkout and setup. Full runs keep 20
shards. Other runs use `min(changed packages, 18) + 2` shards, so between 3
and 20.

The shard step moves from `prepare-environment` to `prepare`, because it needs
the changed paths. The workflow also passes `--silent` to the test step.
Coverage thresholds are configured per package, and a package's files are
split over shards, so they can only be checked once the shard reports are
merged. The shards now run with `--coverage`, and the merge job runs
`--merge-reports --coverage`, which checks each package's thresholds against
the merged data.

`createRootConfig` in `scripts/lib/create-vitest-config.ts` builds the root
config for this. It loads each package's own Vitest config, prefixes its
coverage `include`, `exclude`, and `thresholds` with the package directory,
and only applies thresholds when merging. The root config and the temporary
config for changed-only runs both use it. Workspace paths come from Yarn
through `getAllWorkspaces`, which gains an `includePrivate` option so
`package-template` stays in the root projects.
@Mrtenz
Mrtenz force-pushed the mrtenz/test-sharding branch from e9fd789 to cafccdb Compare November 3, 2026 20:01
Vitest loads `scripts/lib/create-vitest-config.ts` as part of a config, and a
config doesn't get the `tsconfig` path mapping that tests get. The import of
`@metamask/node-utils` therefore resolved to its build output, and failed with
"Failed to resolve entry for package" when the package wasn't built.

The helper now checks whether a config file exists with `access` from
`node:fs/promises`, and the tests mock that instead.
With coverage enabled, the slowest of the 20 shards takes about 170s while the
average takes about 95s, so the slowest shard sets how long the test phase
lasts. More shards shrink the slowest one. Full runs now use 30 shards, and
runs that touch only some packages use between 3 and 30.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant