Skip to content

ci: Run optional E2E tests in a separate workflow - #25548

Draft
andreiborza wants to merge 1 commit into
developfrom
ab/optional-e2e-workflow
Draft

andreiborza wants to merge 1 commit into
developfrom
ab/optional-e2e-workflow

Conversation

@andreiborza

Copy link
Copy Markdown
Member

What

Optional E2E tests now run in their own workflow. The build workflow starts it, waits for it to finish, and reports failures in one PR comment.

Why

Failed optional tests made the PR show a red cross, even though the PR could be merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment on lines +153 to +160
run: ${{ matrix.build-command || 'pnpm test:build' }}

- wait: [install-playwright]

- name: Run E2E test
working-directory: ${{ runner.temp }}/test-application
timeout-minutes: 10
run: ${{ matrix.assert-command || 'pnpm test:assert' }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

workflow_dispatch matrix inputs run arbitrary shell with E2E secrets

Free-form inputs.matrix values are expanded directly into run: steps while Cloudflare, Sentry, OpenRouter, and OpenAI secrets are present, so a workflow_dispatch caller can execute arbitrary commands and exfiltrate those credentials. Keep build/assert as fixed commands (or an allowlist) and pass only safe matrix fields via env.

Evidence
  • on.workflow_dispatch.inputs.matrix is a required free-form string and becomes the job matrix via fromJson(inputs.matrix || ...) at line 77.
  • Build E2E app and Run E2E test use run: ${{ matrix.build-command || 'pnpm test:build' }} and run: ${{ matrix.assert-command || 'pnpm test:assert' }}, so matrix fields are interpolated into the shell script before execution.
  • The same job exports secrets.CLOUDFLARE_API_TOKEN, secrets.E2E_TEST_AUTH_TOKEN, secrets.E2E_OPENROUTER_API_KEY, and conditionally secrets.E2E_OPENAI_DECISIONS_API_KEY into the process environment.
  • matrix.test-application is also interpolated into the ci:copy-to-temp run: step, giving a second expression-injection sink from the same caller-controlled matrix.

Identified by Warden · security-review · MFJ-W8L

@github-actions

Copy link
Copy Markdown
Contributor

Optional E2E tests

Some optional checks were not successful. They do not block the merge. 2 failing, 62 successful. See the run.

Check Status
❌ CI: Optional E2E Tests / E2E solidstart-2 Test (optional) Failing after 1m Details
❌ CI: Optional E2E Tests / E2E supabase-nextjs Test (optional) Failing after 43s Details
Successful checks
Check Status
✅ CI: Optional E2E Tests / E2E cloudflare-agent Test (optional) Successful in 44s Details
✅ CI: Optional E2E Tests / E2E cloudflare-think (ai v6) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E angular (canary) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nextjs-16-tunnel (cloudflare) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-flue Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E gen-ai-libraries Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nextjs-14 (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nextjs-16-standalone (canary, turbopack) Test (optional) Successful in 52s Details
✅ CI: Optional E2E Tests / E2E node-anthropic-send-to-sentry Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nextjs-orpc Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (vinext) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E node-express-mcp-v2 Test (optional) Successful in 50s Details
✅ CI: Optional E2E Tests / E2E cloudflare-workers-send-to-sentry Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nextjs-15 (turbo) Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E node-fastify-6 (tracer provider) Test (optional) Successful in 53s Details
✅ CI: Optional E2E Tests / E2E cloudflare-pi-durable (latest) Test (optional) Successful in 50s Details
✅ CI: Optional E2E Tests / E2E node-flue (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E astro-7 Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (deno) Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (canary, webpack) Test (optional) Successful in 6m Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (canary, turbopack) Test (optional) Successful in 5m Details
✅ CI: Optional E2E Tests / E2E debug-id-sourcemaps Test (optional) Successful in 31s Details
✅ CI: Optional E2E Tests / E2E node-express-mcp-v1 Test (optional) Successful in 47s Details
✅ CI: Optional E2E Tests / E2E node-eve Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E astro-7-static Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nextjs-15 (latest) Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E node-express-send-to-sentry Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-eve (orchestrion) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-fastify-6 Test (optional) Successful in 41s Details
✅ CI: Optional E2E Tests / E2E node-eve (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-express-send-to-sentry-static Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (vinext, bun) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react-cloudflare Test (optional) Successful in 42s Details
✅ CI: Optional E2E Tests / E2E node-mastra Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-openai-decisions-send-to-sentry (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E cloudflare-mastra Test (optional) Successful in 52s Details
✅ CI: Optional E2E Tests / E2E nextjs-16 (cloudflare, latest) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E node-mastra (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nuxt-4 (canary) Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E node-openai-send-to-sentry Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E deno (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E gen-ai-libraries (cloudflare) Test (optional) Successful in 53s Details
✅ CI: Optional E2E Tests / E2E node-anthropic-send-to-sentry (latest) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps (client source maps kept) Test (optional) Successful in 43s Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps Test (optional) Successful in 39s Details
✅ CI: Optional E2E Tests / E2E cloudflare-think Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nuxt-4-static (canary) Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E cloudflare-flue-send-to-sentry Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nextjs-14 (canary) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E node-openai-decisions-send-to-sentry Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E react-router-7-framework-instrumentation Test (optional) Successful in 3m Details
✅ CI: Optional E2E Tests / E2E angular (canary) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E react-send-to-sentry Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E nuxt-5-sourcemaps (static / nuxt generate) Test (optional) Successful in 44s Details
✅ CI: Optional E2E Tests / E2E nuxt-5 Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E node-pi-durable (latest) Test (optional) Successful in 49s Details
✅ CI: Optional E2E Tests / E2E cloudflare-pi-durable Test (optional) Successful in 46s Details
✅ CI: Optional E2E Tests / E2E node-openai-send-to-sentry (latest) Test (optional) Successful in 2m Details
✅ CI: Optional E2E Tests / E2E vue-3 (canary) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E vue-3 (no Options API) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E nuxt-3 (canary) Test (optional) Successful in 1m Details
✅ CI: Optional E2E Tests / E2E node-pi-durable Test (optional) Successful in 50s Details

run: ${{ matrix.build-command || 'pnpm test:build' }}

- wait: [install-playwright]
- name: Trigger optional E2E tests

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no strong feeling, but maybe this could be a JS-based in-repo action (like size limit etc)? would maybe be a bit easier to read/adjust in the future?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants