Deploy benchmarks

This repository measures two things for several deployment tools:

• Packaging: how long a tool takes to turn Lambda functions into what it uploads, and how big that upload is. For containers, only the image size.
• Deploying a change: how long it takes from starting a deploy until the changed code answers requests.

The tools are Stacktape, Serverless Framework, SST, AWS CDK, Render, Heroku and Ravion. Stacktape publishes these results on its website, so we wrote down every choice we made. If you think a tool is set up worse than its users would set it up, please open an issue.

Every test has its own branch, named <app>/<tool>, for example lambda-25/sst. A branch holds only that application and that tool's configuration, so it deploys as it is. Its README gives the tool version, the setup, the commands and the shape of the result. The list of branches is below.

You need Node.js 22 or newer, npm, Git, Docker with BuildKit, zip, the AWS CLI and a disposable AWS account. The AWS deployments run in eu-west-1. To measure how long SST takes to deploy a change to the 25-function Lambda backend:

git clone https://github.com/stacktape/deploy-benchmarks && cd deploy-benchmarks
npm ci
export AWS_PROFILE=<profile of a disposable account>
node bench.mjs setup  sst lambda-25            # write the lambda-25/sst branch to work/ and deploy it once (not measured)
node bench.mjs deploy sst lambda-25 --runs 10  # deploy a code change ten times
node report.mjs                                # median, fastest and slowest run
node bench.mjs remove sst lambda-25            # delete everything setup created

Replace sst lambda-25 with any tool and application from the list of branches. The other measurements are node bench.mjs package (packaging and image size) and node bench.mjs push (Git push), described in Run the benchmark. Each tool needs these credentials:

These are the results of a qualification run on 17 October 2026, between 13:47 and 20:37 UTC. It checked the harness on a dedicated machine and measured the Stacktape fixes merged up to d51f56ad. The figures on stacktape.com will come from a rerun with the released Stacktape CLI (Stacktape's launch item L20), so these numbers can change before then.

• Machine: an EC2 c7i.2xlarge (8 vCPUs, 16 GB) in eu-west-1, Amazon Linux 2023, Node.js 24.15.0, Docker 29.7.2 with the containerd image store and buildx 0.38. Each result records the machine.
• Stacktape: CLI 4.0.0-bench.d51f56ad, built from source as described in Build Stacktape from source.
• Other tools: Serverless Framework 4.44.0, SST 4.17.2, CDK CLI 2.1145.0 and Heroku CLI 11.12.0, as pinned in configs/. Render and Heroku built with what they offered that day.
• Harness: this repository from commit b10797f on. The fixes made during the run are listed under Changes during the run.
• Order: packaging ran alone on the machine. The deploy tests then ran in rounds, one run of each tool per round, with up to four series at a time (Lambda, and one per container application); the Git push pairs ran next to them. The machine's 5-minute load average stayed below 3 during the deploy tests.
• Raw data: every run is in results/. node report.mjs prints every series; its full output is at the end of this section.

Unless a cell gives n, it is the median of at least 10 runs, with the fastest and slowest run in brackets. Stacktape Git-push samples have smaller counts, shown explicitly below. Times are in seconds. Bold is the fastest or smallest in its row. A blank cell means the tool has no such test (see What we measured). [n failed] counts runs that failed and are left out of the median.

Not measured yet:

• Stacktape Git push remains unmeasured for lambda-1, container-python and five-services. The three measured apps use a separate source-built CLI and test runner image, as described below.
• Ravion is pending: our Ravion account needs a setup step from its owner.
• Ravion five-services is blocked by a Ravion bug: its EC2 builder applies source_base_path twice (see Ravion).

Packaging time in seconds, empty build cache: median (fastest–slowest)

Packaging time in seconds, after a source change: median (fastest–slowest)

Upload size in MB (functions, layers and source maps)

Container image size in MB (compressed, one measurement)

Deploy from the CLI: seconds until the new code serves

Deploy from the CLI, shared-package change (every function): seconds until the new code serves

Git push with an empty build cache: seconds from git push until the new code serves

Git push 15 minutes later: seconds from git push until the new code serves

Reading the results:

• Deploy time is time until the new code serves, not until the command exits. Each run also records the command's own duration as commandMs. Stacktape's default web service (an HTTP API in front of the container, with no health check) serves the new code after about 40 seconds, but stacktape deploy exits only after about 185 seconds, when ECS has finished the rolling deployment (median commandMs: 184.5 s for container-node, 194.2 s for container-python, 203.1 s for five-services). SST does the opposite: sst deploy starts the ECS update and exits after about 8 seconds, before the new task serves.
• Stacktape's shared change on lambda-5 and lambda-25 goes through a CloudFormation update. A change to the shared package changes Stacktape's shared chunk layer, which its direct code update does not handle yet, so these runs take 30–38 seconds instead of about 10.
• SST's failed runs happened between 14:56 and 14:58 UTC, when the machine's /tmp was full. SST's Pulumi engine leaves a 22 MB pulumi-asset-* file in /tmp on every deploy, and we deleted the old files during the run. The runs are kept, and make-up runs brought each series to 10.
• Render's failed run on five-services: the webhooks service returned HTTP 404 during the first run, about ten minutes after the service had been recreated.
• Render deployed from a second copy of the fork. Render's GitHub app had no access to our fork, so the Render services deployed the same branches and commits from another repository the app could see.

These separate qualification samples ran on 17–18 October 2026 through the dev Console, with workloads in eu-west-1. They use CLI commit 16d16c8ed97e95c8129f684cbf143341da29bbf5, version 4.0.0-bench.16d16c8e, baked into test AMI ami-01572ab71483bcc1e (build bmk-push-36c3460d). No npm publication was needed. Other tools' numbers and the CLI-only Stacktape results above retain their original revisions and machines. The observer was WSL on an i7-13700K; builds ran on a c7a.2xlarge runner with a persistent cache disk in the connected AWS account, not on the observer or the other qualification run's c7i.2xlarge.

A push to the project's connected GitHub branch triggers the Console runner. Cold means its instance, cache disk and database runner record were removed before the push. Warm is the paired push 15 minutes after readiness; the logs confirm that every included warm sample reused a running instance. The idle timer starts after the job finishes, which can be much later than readiness for containers. The separate resume samples waited for the runner to become stopped through idle hibernation before pushing; they reuse its disk and caches.

Only rows with at >= 2026-10-17T22:00Z count. Rounds 1–10 used a WSL monotonic clock that ran approximately 9% fast, inflating elapsed readiness by roughly 12–22 seconds on cold pushes. All 60 rows remain in results/deploy-stacktape-push.jsonl, marked with an excluded reason, and are omitted from medians and phase summaries. The surviving rerun stopped before reaching ten samples; it contains 4 cold/3 warm Lambda samples per app and 5 cold/4 warm container samples. No additional rounds were collected.

Stacktape Git push: seconds until new code serves, median (min–max), valid n

The nine resume samples are preserved verbatim in results-extra/resume.jsonl. Their collector saved readiness and runner state, but did not persist the main harness's businessOk/unchangedOk flags; they are reported separately rather than silently treated as fully checked deploy rows. Cold and warm rows all passed those two checks.

Runner phase breakdown, cold samples: median seconds (min–max)

CloudWatch timestamps provide the phase boundaries, with about one-second log resolution. The last column combines the readiness observer with the runner timestamp. Medians of individual phases need not sum to the median total.

Warm queue-to-ready medians are 0.5 s; hibernation-resume medians are about 19–21 s. Warm job-to-install is 3 s, versus 21 s cold. Lambda dependency installation takes 8 s cold and 5 s warm before the CLI starts. Cold CLI Build & Package medians are 3.2 s (lambda-5), 4.8 s (lambda-25) and 11.7 s (container-node); warm medians are 1.9, 4.9 and 4.6 s. Container CLI completion takes about 207 s, although new code serves about 45–52 s after CLI start. That ECS settling time is not added to readiness.

The largest optimization candidates, based on these samples and the earlier agents' log/code diagnosis, are:

1. Fix runner IAM propagation and SSM retry startup. Eleven of the 13 valid cold samples took 161–190 s from push to runner-ready, versus 71–72 s for the two fast samples. Earlier log diagnosis attributes about 95 s of the slow path to a fresh instance profile reaching SSM late, backing off, and an instance restart. Avoiding that retry is the largest cold-start opportunity; this is an estimate, not a measured optimized implementation.
2. Keep runner tools ready without loading unused images. Cold job-to-install is 21 s versus 3 s warm. Earlier diagnosis found a silent roughly 20 s docker load of the preseeded ~480 MB Railpack archive before checkout, even for Lambda jobs. Lazy or image-ready tool preparation could recover roughly 18–20 s; checkout itself was ~1 s.
3. Remove the duplicate Lambda dependency install. The runner installs dependencies, then the CLI's workspace detection installs them again during Initialize (~6.8–7.3 s). Reusing the completed install could save roughly 6 s per Lambda push. This needs implementation and fresh qualification before claiming a speedup.

These are prerelease qualification figures. Before using them as launch claims, rerun with the published CLI and release runner image, verify the observer clock rate against AWS timestamps, and disclose actual running/stopped state at the 15-minute push. The temporary test image and projects are removed after this qualification.

The harness was fixed three times while the series ran. Runs that a harness bug made wrong are not deleted: they are in results/superseded.jsonl, each with the reason in superseded, and a make-up run or rerun replaced each of them.

• CDK's warm upload size counted the asset directories of earlier synths left in cdk.out (61 MB instead of 9.4 MB for lambda-1). Fixed in 3bde156; the CDK warm packaging series was rerun (30 runs).
• The business check called each service once and could reach an old task during a rolling update. Fixed in 7e9ce35, which gives a changed service up to a minute to answer from the new revision; the clock was not changed. 21 SST and Render runs before the fix failed only this check.
• Git push order: the harness could push while the previous GitHub Actions deploy was still waiting for ECS, so CDK's next workflow found its stack in UPDATE_IN_PROGRESS and timed out. Fixed in 954bfb4, which waits for the branch's previous workflow; two CDK container runs are superseded.

The Lambda applications are the backend of a small online shop, built like many real Lambda backends: an npm workspace whose functions all import a shared service package (Prisma with PostgreSQL, the AWS SDK, Stripe, Slack, Google Sheets, OpenAI, email) and a shared util package. Most functions are queue workers and scheduled jobs; four are HTTP APIs; one builds a PDF, an Excel workbook and a Word document. The 1- and 5-function applications are subsets of the 25-function one. The container applications are small commerce APIs whose code does real work. All of them are described in fixtures/README.md.

In the five-service application, the change edits three services. The other two must keep running their old code. The Lambda applications have a second deploy test, the shared change: it edits a module of the shared service package, which every function imports, so every function must be rebuilt.

The change. Every function and service has a small revision file, for example create-order.revision.ts. A deploy run writes a new revision string into the revision file of every changed function or service. Every response from the applications contains the revision of the code that produced it. All tools get the same change. The shared change (--change shared) instead edits serviceRevision in packages/service/src/orders.ts, which every Lambda response reports.

When a deploy counts as done. The clock starts when the deploy command starts. For a Git push, it starts when git push returns. The clock stops when every changed function or service has returned the new revision. Functions are invoked every 250 ms and services are called every second. The clock does not wait for the command to exit; the command's own duration is recorded separately. After the clock stops, every function and service must also return the right business result (for example a cart total of 33.00) from the expected revision. During a rolling update a load balancer still sends some requests to the old code, so a changed service gets up to a minute to answer from the new revision; the clock has already stopped. Runs that fail this check are kept in the raw data but left out of the results.

Packaging. The clock covers the tool's own packaging command. Before each run we delete the tool's previous build output. Every series starts with one run that is not measured, so nothing a tool downloads on first use is timed. The "warm" series keeps the previous build and changes only the source, like a developer's next build.

Container build time is part of the deploy, not packaging. No tool builds a container image except as part of a deploy, so the deploy test times each tool's whole deploy command, image build included; for containers, the packaging test reports only the image size.

Upload size. For Lambda, this is everything the tool uploads as code: each function's ZIP, each layer it creates, and the source maps it uploads next to the ZIPs (SST and Stacktape do this). For containers, it is the compressed image size reported by the local Docker image store, which is what a push to an empty registry uploads. Heroku reports a compressed slug instead of an image. Render does not report a size.

Not measured. Creating the infrastructure the first time, installing tools and dependencies, and the checks after each run.

Recorded for every run: the tool and its version, the Stacktape commit when Stacktape is built from source, the machine, the timings, the sizes and a log of every command.

Every tool uses its documented defaults, except where noted in Where a tool needed help. On AWS, every function uses Node.js 22 with 512 MB of memory and a 30-second timeout, and every container uses 0.25 vCPU and 512 MB on Fargate.

Regions and instance sizes. The AWS tools deploy to eu-west-1 (Ireland), and the measuring machine runs there too. Heroku apps run in Heroku's EU region, which is also in Ireland, and Render services in Frankfurt, Render's closest region. Each Heroku app runs on one Basic dyno (512 MB, always on) and each Render service on one Starter instance (0.5 CPU, 512 MB). These are the smallest always-on sizes of each platform, comparable to the 0.25 vCPU, 512 MB Fargate tasks. The npm tools are pinned in each project's package.json and package-lock.json. Render, Heroku and Ravion choose their own build images, so they run whatever version the platform offers at the time of the measurement. The exact files are in the test branches and in configs/; node configs.mjs regenerates them from the tool modules in tools/.

• Configuration: configs/stacktape/, written by tools/stacktape.mjs.
• Version: the CLI built from source at a commit that each result records. See Build Stacktape from source.
• Packaging: stacktape package. Deploy: stacktape deploy.
• Functions and the Node.js services use Stacktape's js-bundle packaging. The Python service uses buildpack.
• stacktape deploy updates code-only changes directly, without a CloudFormation update. This is its default.
• Git push: Stacktape Console's Git integration, which deploys on the project's runner in your AWS account (a c7a.2xlarge EC2 instance with a cache disk). Stacktape Git push runs describes what the first push and the push 15 minutes later find on the runner.

• Configuration: configs/serverless/, written by tools/serverless.mjs.
• Version: 4.44.0.
• Packaging: serverless package. Deploy: serverless deploy.
• The Serverless service is the monorepo's packages/functions package, where its serverless.yml lives. That is the usual layout, and Serverless installs external packages at the versions in that package's package.json.
• Version 4 with its built-in esbuild bundling (build.esbuild), with esbuild's default options. Without that setting, version 4 bundles only TypeScript handlers. We turned it on because most Serverless projects bundle their code. In the last month, the serverless package had 5.9 million downloads, serverless-esbuild 2.0 million, serverless-webpack 1.0 million and serverless-plugin-typescript 0.5 million. Weekly, version 3 had 681,000 downloads and version 4 had 511,000 (October 2026).
• serverless deploy function is faster for a single function, but it is a separate command and skips CloudFormation, so we use serverless deploy.
• Git push: GitHub Actions, using the shared workflow. Serverless Framework v4 has no Git deployment service of its own.

• Configuration: configs/sst/, written by tools/sst.mjs.
• Version: 4.17.2.
• Deploy: sst deploy. Functions use sst.aws.Function; services use sst.aws.Service on an SST cluster.
• Packaging: SST has no command that only packages. sst diff on a deployed stage builds and zips every function, then compares the plan with what is deployed. Its time therefore includes that comparison, which reads from AWS.
• Containers are built from a plain Dockerfile: the official slim image, a locked production install, then the source. SST runs docker build during sst deploy; for the image size, the packaging test runs the same docker build directly.
• Git push: the shared GitHub Actions workflow. SST's own Autodeploy needs an SST Console account and is not measured.

• Configuration: configs/cdk/, written by tools/cdk.mjs.
• Version: CDK CLI 2.1145.0 with aws-cdk-lib 2.273.0, constructs 10.8.1 and esbuild 0.28.2.
• Deploy: cdk deploy. Functions use NodejsFunction; services use ApplicationLoadBalancedFargateService.
• Packaging: cdk synth, which bundles each function with esbuild (installed in the project, as CDK recommends). CDK zips the bundles later, during cdk deploy, so zipping is not part of CDK's packaging time. For the upload size we zip each bundle at the same compression level CDK uses.
• Containers use the same Dockerfile as SST, and the image size comes from the same direct docker build.
• Git push: the shared GitHub Actions workflow.
• CDK with hotswap is reported as a separate tool, with its own branches (<app>/cdk-hotswap). cdk deploy --hotswap updates Lambda code and ECS services directly, which is CDK's fast path for code changes. CDK documents it for development stacks only, because it leaves the CloudFormation stack out of date. Its first deploy is a normal cdk deploy. It has no Git push test.

• Configuration: configs/render/, written by tools/render.mjs.
• Render's native Node.js and Python runtimes on Starter instances in Frankfurt, with the build and start commands from Render's Express and FastAPI guides.
• Render deploys only from a Git repository, so it has no laptop deploy. Build time comes from Render's build events.
• node bench.mjs setup creates the services of render.yaml through the Render API. The first push of each pair starts with "Clear build cache & deploy" of the previous commit, through the API, and waits until it is live.

• tools/heroku.mjs. No extra configuration: the applications already have a Procfile and runtime version files.
• The default stack and the official Node.js and Python buildpacks. Each service is its own Heroku app, in the EU region, on one Basic dyno.
• Laptop deploy: git push to Heroku's Git remote. Git push: Heroku's GitHub integration with automatic deploys.
• Heroku builds the repository root. For the five-service application we push each service with git subtree push, and its GitHub integration cannot deploy it.

• tools/ravion.mjs. Ravion's configuration lives in the Ravion project, not in the test branch. Railpack detects the runtime, dependencies and start command from the source.
• Ravion deploys only from a Git repository, so it has no laptop deploy. Build time and image size come from the build step of Ravion's pipeline runs and from ECR.
• One Ravion project (bmk) with one environment (bench) in eu-west-1, set up as in Ravion's Node.js guide with the current module versions:
  • rvn-aws-network 1.2.1 with its defaults (three subnet pairs, one NAT gateway).
  • rvn-ecs-cluster 2.2.0 with use_ravion_managed_domains: true and load_balancer_deletion_protection_enabled: false.
  • One rvn-ecs-web 2.3.0 per service: build_source: railpack, source_repo: github.com/<you>/deploy-benchmarks, container_port: 3000, health_check_path: /health, fargate_size: {vcpu: 0.25, memory_gb: 0.5} and build_capacity_type: ec2-spot. The module and service names are container-node/bmk-container-node, container-python/bmk-container-python, and for five services five-services-<service>/bmk-fs-<service> with source_base_path: <service>. Every other input keeps its default, including Railpack's commands, autoscaling from one task and rolling deploys.
  • One pipeline per service, with the module's given ID: a build step with branch and ref from the push, then a deploy step with the build's image_digest, and a webhook:github push trigger filtered to the test's branch (and, for five services, paths: ['<service>/**']).
  • The project's Terraform change and destroy pipelines are copies of Ravion's system pipelines with infrastructure: {type: ec2, instance_size: c7a.large} instead of sandboxes. They only create the infrastructure, which is not measured.
• The five-service application does not build on Ravion yet. Ravion's EC2 builder applies source_base_path twice (directory .../api/api does not exist), so a service in a subdirectory cannot be built with Railpack.

Left at each tool's default: the AWS SDK and Prisma. Stacktape, Serverless Framework and AWS CDK all leave @aws-sdk/* out of the bundle, because the Node.js 22 Lambda runtime provides it, and bundle Prisma's client, whose query compiler is plain JavaScript and WebAssembly in Prisma 7. prisma generate runs on npm ci for every tool.

• Stacktape does not have to win. The published results will show every test where another tool is faster or smaller. Source maps count toward the upload size of the tools that upload them (SST and Stacktape), and AWS CDK uploads none by default.
• The tools roll out changes differently. Serverless Framework and AWS CDK go through CloudFormation. Stacktape and CDK with hotswap update code directly. SST uses Pulumi. Render, Heroku and Ravion run their own pipelines. We measure each tool's normal path; the guarantees are not the same.
• First response. A service counts as updated at its first response from the new code. During a rolling update, some requests can still reach the old code for a short time.
• SST's packaging time includes the plan comparison of sst diff, because SST has no packaging-only command.
• CDK's and Stacktape's packaging times do not include zipping that happens only during deploy (CDK's function bundles, Stacktape's shared layers). Their upload sizes do include those ZIPs.
• Hosted builders. Render and Heroku build on the provider's machines, so their build times depend on the provider's capacity. Ravion builds on EC2 Spot instances in the connected account, which depend on Spot capacity. Render has no way to empty only the build cache, so its first-push runs need a manual cache reset.
• One machine, one region, one period. The results describe this setup, not every project.

Run on a dedicated machine in eu-west-1, the region of the deployments; we use EC2. Image sizes need Docker's containerd image store, the default of Docker Desktop and Docker Engine 29; the older overlay2 store reports unpacked sizes, so the harness refuses it. Everything the benchmark creates is named bmk-*. Install the harness's own dependency (the AWS SDK, to invoke functions) and set the credentials from the table above:

npm ci
export AWS_PROFILE=...

Tools are stacktape, serverless, sst, cdk, cdk-hotswap, render, heroku and ravion. Applications are lambda-1, lambda-5, lambda-25, container-node, container-python and five-services.

node bench.mjs setup   sst lambda-5                         # write the project and deploy it (not measured)
node bench.mjs package sst lambda-5 --runs 10               # package from an empty build cache
node bench.mjs package sst lambda-5 --runs 10 --warm        # package after a source change
node bench.mjs package sst container-node                   # container image size
node bench.mjs deploy  sst lambda-5 --runs 10               # deploy a code change from this machine
node bench.mjs deploy  sst lambda-5 --runs 10 --change shared   # change the shared service package instead
node bench.mjs remove  sst lambda-5                         # delete everything setup created
node report.mjs                                             # median, fastest and slowest run of every series

setup writes the test's branch to work/bmk-<tool>-<app>/ as its own Git repository, with the same commit as the published branch, and installs its locked dependencies. Command logs are in logs/. SST's function packaging needs a deployed stage, so run setup first. To spread network and AWS variation evenly, run one round of every tool before the next round:

for round in $(seq 10); do
  for tool in stacktape serverless sst cdk cdk-hotswap; do node bench.mjs deploy $tool lambda-25 --runs 1; done
done

The published results use a Stacktape CLI built from source. Each result records the commit in stacktapeCommit.

git clone https://github.com/stacktape/stacktape && cd stacktape
git checkout d51f56ad17b9a24056046d7af99bd3716afb30da   # the commit in tools/stacktape.mjs
pnpm install --frozen-lockfile
pnpm turbo run build --filter=@stacktape/init-ui
cd apps/cli
bun scripts/build-dist-package.ts --platform linux --version 4.0.0-bench.$(git rev-parse --short=8 HEAD) --keepUnarchived
export STACKTAPE_BIN=$PWD/__dist/linux/stacktape

This is the build the release workflow makes, with Node.js 24.15, pnpm 11.17 and Bun 1.4.1. The harness checks that the version of STACKTAPE_BIN ends in the pinned commit.

node bench.mjs push <tool> <app> --runs 5 commits a change to the project in work/, pushes it to the test's branch on the project's origin remote and measures until the new code answers. Each run is a pair: a first push with an empty build cache, then a second push 15 minutes after the first one finished.

You need a fork of this repository with all branches (untick "Copy the main branch only"), because the platforms deploy from your fork's branch. The project that setup writes has the same commit as the published branch, so its pushes continue your fork's branch. Set BENCH_FORK before setup, which adds the fork as the project's origin:

export BENCH_FORK=https://github.com/<you>/deploy-benchmarks

GitHub doesn't run workflows in a new fork until you turn them on in its Actions tab. The AWS role for the workflows must trust the fork's OIDC subject. New repositories get immutable subjects such as repo:<you>@<owner id>/deploy-benchmarks@<repository id>:ref:refs/heads/<branch>; gh api repos/<you>/deploy-benchmarks/actions/oidc/customization/sub shows the prefix.

Then connect the deploy service to your fork's branch. Each test branch's README has these steps for its application:

Stacktape

• Deploy the test's branch once with stacktape deploy from a terminal and answer yes to "Set up automatic deployments on push". This connects the repository and the test's branch to the project with the stage bmk in eu-west-1. Stacktape's GitHub app must have access to the repository.
• Every push then deploys on the project's runner in your AWS account: an EC2 instance with a cache disk, launched for the first push and hibernated after 15 idle minutes. The runner installs the Stacktape version set for the project in the Console from npm unless its image already contains that version. A build from source is not on npm; the main README describes how we measured one.
• To empty the runner's caches before each first push, remove the project's runner (its instance, cache disk and record) and pass --cold-reset manual. The Console can't do this yet; see the main README.

Serverless Framework

• Turn on GitHub Actions in your fork. This branch's .github/workflows/deploy.yml deploys every push to the branch.
• Add the repository secrets AWS_ROLE_ARN and SERVERLESS_ACCESS_KEY. AWS_ROLE_ARN is a role in the disposable account that the fork can assume through GitHub OIDC.
• Sign gh in to an account that can manage the fork's Actions caches (gh auth login or GH_TOKEN). The first push of each pair deletes them with gh cache delete --all.

SST

• Turn on GitHub Actions in your fork. This branch's .github/workflows/deploy.yml deploys every push to the branch.
• Add the repository secret AWS_ROLE_ARN. AWS_ROLE_ARN is a role in the disposable account that the fork can assume through GitHub OIDC.
• Sign gh in to an account that can manage the fork's Actions caches (gh auth login or GH_TOKEN). The first push of each pair deletes them with gh cache delete --all.

AWS CDK

• Turn on GitHub Actions in your fork. This branch's .github/workflows/deploy.yml deploys every push to the branch.
• Add the repository secret AWS_ROLE_ARN. AWS_ROLE_ARN is a role in the disposable account that the fork can assume through GitHub OIDC.
• Sign gh in to an account that can manage the fork's Actions caches (gh auth login or GH_TOKEN). The first push of each pair deletes them with gh cache delete --all.

AWS CDK with hotswap: no Git push test.

Render

• Install Render's GitHub app on your fork. With BENCH_FORK set, node bench.mjs setup creates the services of render.yaml through the Render API, deploying the test's branch of your fork on every push. A Blueprint created from render.yaml in the Render dashboard is the same.
• The first push of each pair starts with a "Clear build cache & deploy" of the previous commit through the Render API and waits until it is live.

Heroku

• node bench.mjs setup creates the Heroku app. In the Heroku dashboard, connect it to your fork and the test's branch, and turn on automatic deploys.
• The first push of each pair empties the build cache with heroku builds:cache:purge.

Ravion

• In Ravion, connect your AWS account, and install Ravion's GitHub app on your fork in Settings / Git connections.
• Create the Ravion project described in the main README: a network, an ECS cluster, one web module with the repository root as its source directory, and one pipeline that builds and deploys every push to the test's branch.
• Set the RAVION_URL_* variables to the service URLs.
• The first push of each pair empties the build cache: it deletes the buildcache and railpack tags from each changed service's ECR repository.

For Render, Ravion and Heroku, connect the platform before setup: they deploy only from Git, and setup checks the deployed services.

The test branches and configs/ are generated from tools/, fixtures/, lib.mjs and project.mjs. Don't edit them by hand. After a change to those files:

node configs.mjs           # rewrite configs/, the pinned lockfiles and the generated parts of this README
git commit -am "..."       # the branches are generated from committed files
node branches.mjs --push   # rebuild every <app>/<tool> branch and force-push it
node branches.mjs --check  # confirm that every branch on GitHub matches main

node configs.mjs keeps the versions in each lockfile unless a pinned tool version changes. Each branch is a single commit with a fixed author and date, so the same files always give the same commit, on any machine. node configs.mjs --check and node branches.mjs --check change nothing and fail when something is out of date.

We want each tool to look the way its users would set it up. If a configuration here is worse than what you would use, or a measurement favors one tool, please open an issue in this repository. Tell us what you would change and, if you can, link the documentation that supports it. We will rerun the affected tests.

MIT