ci: add import profiler check across monorepo#17657
Conversation
There was a problem hiding this comment.
Code Review
This pull request adds a new import_profile session to the noxfile.py.j2 template to measure and ensure import times remain below defined thresholds. Feedback points out that because this template is used in standalone repositories outside of the monorepo, referencing a relative path to the monorepo's root scripts directory will cause default nox runs to fail in those environments. It is recommended to check for the script's existence and gracefully skip the session if it is missing, while also using pathlib.Path for cleaner path handling.
There was a problem hiding this comment.
I left a couple comments. It would be good to clarify the goal of this check though.
What counts as a failure? Can we calculate an import time diff before and after the target changes, instead of just reporting an absolute value? How are developers intended to interact with this?
| ;; | ||
| *) | ||
| nox -s ${TEST_TYPE} | ||
| retval=$? |
There was a problem hiding this comment.
This PR doesn't touch any packages, so it's hard to tell what the action will look like. Can you temporarily touch some files to test it out?
There was a problem hiding this comment.
We should also see how long the new check would take if all packages were updated. In #17438, I added a new unit_test:all_packages tag, that will run against all packages when added. Maybe we should support that here too
It's possible we would only want to profile a few key packages instead of all of them. Most generated libraries should be very similar
113b3d5 to
9472ab6
Compare
Overview
This PR integrates the import profiler script (introduced in #17467) into our automated CI pipeline.
Why this matters: Import times significantly impact CLI responsiveness and cold starts for Serverless products like Cloud Run and Cloud Functions. Ideally, library imports should stay under 500ms, and anything taking over 1 second is a target for optimization. This CI check helps us proactively track metrics like import latency, memory footprint, and code volume to prevent performance regressions on critical libraries.
The primary goal of this check is to track and enforce performance standards for package import times across the repository, especially following our recent work on lazy loading and cold-start optimizations.
By running this benchmark as a CI check with a defined failure threshold, we can programmatically prevent major regressions in module initialization times before they are merged, ensuring downstream consumers aren't impacted by unexpectedly slow startup times.
Changes Included
.github/workflows/import-profiler.yml) that triggers on PRs and merge groups. The workflow is pinned specifically to Python 3.15.noxfile.py.j2) to natively generate a standardizedimport_profileNox session for all packages.ci/run_single_test.shto execute the new Nox session. To avoid breaking the pipeline on older packages that haven't been regenerated yet, it verifies if theimport_profilesession exists before running it, and gracefully skips it if it doesn't.profiler.pyto accept a--fail-thresholdargument (to fail CI if the P99 time exceeds an absolute limit like 5 seconds) and a--packageargument (to internally auto-detect the underlying module namespace natively).import_profile:all_packageslabel to a Pull Request will now force the CI pipeline to run the profiler check against all packages in the mono-repo, instead of just the modified ones.Related PRs