Skip to content

perf: Reduce filesystem syscalls in globwalk, SCM hashing, and task scheduling#11907

Merged
anthonyshew merged 3 commits intomainfrom
anthonyshew/perf-optimizations
Feb 19, 2026
Merged

perf: Reduce filesystem syscalls in globwalk, SCM hashing, and task scheduling#11907
anthonyshew merged 3 commits intomainfrom
anthonyshew/perf-optimizations

Conversation

@anthonyshew
Copy link
Copy Markdown
Contributor

@anthonyshew anthonyshew commented Feb 19, 2026

Summary

Reduces turbo run --dry wall-clock time by 5-12% on large monorepos (~1700 tasks) by cutting unnecessary filesystem operations and CPU work.

User time dropped 39% (3.7s → 2.25s) and system time dropped 30% (1.87s → 1.32s) on the largest benchmark repo. Wall-clock improvement is bounded by irreducible I/O (git subprocesses, remaining directory walks).

Changes

Commit 1: Fast-path invariant globs

Literal file globs like package.json and turbo.json — appended to every task's include list — now resolve with a single symlink_metadata() instead of a full directory traversal via wax::Glob::walk(). Also separates glob compilation from the walk so EMFILE retries don't recompile patterns, and removes per-call tracing from visit_file/glob_with_contextual_error (~6000 calls in large repos).

Commit 2: In-memory exclude matching + parallel hashing

Exclude globs are now matched against already-known paths in memory instead of spawning a separate filesystem globwalk. hash_objects uses rayon for parallel file hashing. git_ls_tree_repo_root_sorted builds a BTreeMap directly, skipping the HashMap→BTreeMap conversion.

Commit 3: Arc-share FileHashes + batch mutex

expanded_hashes stores Arc<FileHashes> so distributing results to ~1700 tasks is a refcount bump, not a HashMap clone. calculate_dependency_hashes acquires the tracker mutex once instead of per-dependency.

Benchmarks

hyperfine with --warmup 5, 10 runs each. faster = this PR, baseline = current main.

Large repo (~1700 tasks):

faster:   2.004 s ± 0.057 s
baseline: 2.165 s ± 0.033 s
→ 1.08 ± 0.03x faster (User: 2.26s vs 3.70s, System: 1.32s vs 1.87s)
faster:   1.996 s ± 0.054 s
baseline: 2.241 s ± 0.192 s
→ 1.12 ± 0.10x faster

Medium repo (~200 tasks):

faster:   1.135 s ± 0.074 s
baseline: 1.158 s ± 0.067 s
→ 1.02 ± 0.09x faster

Small repo (~5 tasks):

faster:   640.6 ms ± 27.0 ms
baseline: 655.9 ms ± 40.3 ms
→ 1.02 ± 0.08x faster

Literal file globs (e.g. package.json, turbo.json) are resolved with a
single symlink_metadata() syscall instead of a full directory traversal.
This eliminates thousands of redundant readdir calls in large monorepos
where config files are appended to every task's include list.

Also separates glob compilation from filesystem walking so EMFILE retries
don't recompile patterns, and removes per-call tracing instrumentation
from visit_file and glob_with_contextual_error (called ~6000 times in
large repos).
Three optimizations to the SCM layer:

1. Exclude globs in get_package_file_hashes_from_inputs_and_index are now
   matched in-memory against already-known paths instead of spawning a
   separate filesystem globwalk.

2. hash_objects uses rayon to parallelize file hashing across threads.
   Each file already has its own EMFILE retry logic.

3. git_ls_tree_repo_root_sorted returns a BTreeMap directly, eliminating
   the intermediate HashMap->BTreeMap conversion in RepoGitIndex::new.
- expanded_hashes now stores Arc<FileHashes> so task distribution is a
  refcount bump instead of cloning the entire HashMap per task.

- calculate_dependency_hashes acquires the tracker mutex once for all
  dependencies instead of once per dependency.

- Vendor::infer() hoisted before the scheduling loop.
@vercel
Copy link
Copy Markdown
Contributor

vercel Bot commented Feb 19, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
examples-basic-web Building Building Preview, Comment, Open in v0 Feb 19, 2026 0:39am
examples-designsystem-docs Ready Ready Preview, Comment, Open in v0 Feb 19, 2026 0:39am
examples-gatsby-web Ready Ready Preview, Comment, Open in v0 Feb 19, 2026 0:39am
examples-kitchensink-blog Ready Ready Preview, Comment, Open in v0 Feb 19, 2026 0:39am
examples-nonmonorepo Ready Ready Preview, Comment, Open in v0 Feb 19, 2026 0:39am
examples-svelte-web Building Building Preview, Comment, Open in v0 Feb 19, 2026 0:39am
examples-tailwind-web Ready Ready Preview, Comment, Open in v0 Feb 19, 2026 0:39am
examples-vite-web Ready Ready Preview, Comment, Open in v0 Feb 19, 2026 0:39am
turbo-site Ready Ready Preview, Comment, Open in v0 Feb 19, 2026 0:39am
turborepo-agents Ready Ready Preview, Comment, Open in v0 Feb 19, 2026 0:39am
turborepo-test-coverage Ready Ready Preview, Comment, Open in v0 Feb 19, 2026 0:39am

@github-actions
Copy link
Copy Markdown
Contributor

Coverage Report

Metric Coverage
Lines 74.35%
Functions 46.20%
Branches 0.00%

View full report

@anthonyshew anthonyshew merged commit 9f36363 into main Feb 19, 2026
103 of 104 checks passed
@anthonyshew anthonyshew deleted the anthonyshew/perf-optimizations branch February 19, 2026 12:55
github-actions Bot added a commit that referenced this pull request Feb 19, 2026
## Release v2.8.11-canary.5

Versioned docs: https://v2-8-11-canary-5.turborepo.dev

### Changes

- release(turborepo): 2.8.11-canary.4 (#11904) (`91cf34b`)
- perf: Reduce filesystem syscalls in globwalk, SCM hashing, and task
scheduling (#11907) (`9f36363`)

---------

Co-authored-by: Turbobot <turbobot@vercel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant