Skip to content

Add config-in-database live-tests workflow and config editing e2e tests#7231

Merged
shuyangli merged 4 commits intomainfrom
sl/config-editing-e2e-tests
Apr 13, 2026
Merged

Add config-in-database live-tests workflow and config editing e2e tests#7231
shuyangli merged 4 commits intomainfrom
sl/config-editing-e2e-tests

Conversation

@shuyangli
Copy link
Copy Markdown
Contributor

@shuyangli shuyangli commented Apr 8, 2026

Introduce a new CI workflow that exercises the config-in-database read/write path end-to-end. Unlike live-tests.yml, this workflow has no batch_writes matrix: it boots Postgres, stores the Postgres live-tests config glob via gateway --store-config, then runs the full live suite against a gateway booted from the database with TENSORZERO_INTERNAL_FLAG_ENABLE_CONFIG_IN_DATABASE=true.

Unfortunately these tests need to run serially because concurrent edits conflict; I don't think we can get around this.


Note

Medium Risk
Adds a new required CI job and a serial config-editing e2e suite that can affect pipeline duration/flakiness and gate merges if unstable.

Overview
Adds a new live-tests-config-in-database CI workflow that boots the gateway in config-in-database mode (Postgres-only) via a compose override, runs the standard live test suite, then runs a dedicated serial config-editing nextest profile.

Introduces new e2e tests under config_editing:: that exercise GET /internal/config_toml, POST /internal/config_toml/validate, and POST /internal/config_toml/apply (including CAS conflict/no-op, validation failure, and prompt/template update effects), plus adds the config-editing nextest profile and excludes these tests from the default e2e profile.

Adds config-in-db test fixtures/compose wiring (including a migration service that loads TOML into stored config tables) and wires the new job into general.yml and ci/check-all-general-jobs-passed.sh so it participates in required checks.

Reviewed by Cursor Bugbot for commit 091eabf. Bugbot is set up for automated code reviews on this repo. Configure here.

@shuyangli shuyangli force-pushed the sl/config-editing-via-toml branch from 66901b7 to b0f17d7 Compare April 8, 2026 18:39
@shuyangli shuyangli force-pushed the sl/config-editing-e2e-tests branch from 6689ce6 to abd7197 Compare April 8, 2026 18:39
@shuyangli shuyangli force-pushed the sl/config-editing-via-toml branch from b0f17d7 to a85514b Compare April 8, 2026 18:55
@shuyangli shuyangli force-pushed the sl/config-editing-via-toml branch 3 times, most recently from 85beeb8 to fdc9aa1 Compare April 8, 2026 19:33
@shuyangli shuyangli force-pushed the sl/config-editing-e2e-tests branch from abd7197 to 5a11ce6 Compare April 8, 2026 19:33
@shuyangli shuyangli force-pushed the sl/config-editing-via-toml branch from fdc9aa1 to 9888492 Compare April 8, 2026 20:33
@shuyangli shuyangli force-pushed the sl/config-editing-e2e-tests branch from 5a11ce6 to b3ff029 Compare April 8, 2026 20:33
@shuyangli shuyangli force-pushed the sl/config-editing-via-toml branch from 9888492 to 95ea677 Compare April 8, 2026 20:37
@shuyangli shuyangli force-pushed the sl/config-editing-e2e-tests branch from b3ff029 to 66c4f7e Compare April 8, 2026 20:37
@shuyangli shuyangli force-pushed the sl/config-editing-via-toml branch from 95ea677 to 60adc3e Compare April 8, 2026 20:42
@shuyangli shuyangli force-pushed the sl/config-editing-via-toml branch 3 times, most recently from 533b3be to 70ff73b Compare April 8, 2026 21:18
@shuyangli shuyangli force-pushed the sl/config-editing-e2e-tests branch from 66c4f7e to ac49210 Compare April 8, 2026 21:19
@shuyangli shuyangli force-pushed the sl/config-editing-via-toml branch from 70ff73b to 864e421 Compare April 8, 2026 21:25
@shuyangli shuyangli force-pushed the sl/config-editing-e2e-tests branch 2 times, most recently from 53c381d to 51d4005 Compare April 8, 2026 22:15
@shuyangli shuyangli force-pushed the sl/config-editing-via-toml branch from 864e421 to 50bd715 Compare April 8, 2026 22:15
@shuyangli shuyangli force-pushed the sl/config-editing-e2e-tests branch 2 times, most recently from df0aba8 to 9ad935b Compare April 9, 2026 13:48
@shuyangli shuyangli changed the base branch from sl/config-editing-via-toml to sl/add-cli-store-config-option April 9, 2026 13:49
@shuyangli shuyangli force-pushed the sl/config-editing-e2e-tests branch from 516678e to 9d03833 Compare April 13, 2026 16:57
@shuyangli shuyangli force-pushed the sl/add-cli-store-config-option branch from 15f76cb to 5dc6d5b Compare April 13, 2026 16:57
@shuyangli shuyangli force-pushed the sl/config-editing-e2e-tests branch 2 times, most recently from b2057b4 to 9ef1dec Compare April 13, 2026 18:02
@shuyangli shuyangli force-pushed the sl/add-cli-store-config-option branch from 5882069 to 04e60b5 Compare April 13, 2026 18:16
@shuyangli shuyangli force-pushed the sl/config-editing-e2e-tests branch 2 times, most recently from 22252e8 to c236373 Compare April 13, 2026 18:34
@shuyangli shuyangli force-pushed the sl/add-cli-store-config-option branch from 04e60b5 to fc53808 Compare April 13, 2026 18:34
@shuyangli shuyangli marked this pull request as ready for review April 13, 2026 18:39
Copy link
Copy Markdown
Contributor Author

Majority of the diff is 2 duplicated function config files because they included a prompt that needed template_filesystem_access. This one patches them out.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c2363739d3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/general.yml
Comment thread crates/tensorzero-core/tests/e2e/config_editing/mod.rs Outdated
@shuyangli shuyangli force-pushed the sl/config-editing-e2e-tests branch from c236373 to a262b5e Compare April 13, 2026 19:05
Base automatically changed from sl/add-cli-store-config-option to main April 13, 2026 19:49
Introduce a new CI workflow that exercises the config-in-database read/write
path end-to-end. Unlike `live-tests.yml`, this workflow has no batch_writes
matrix: it boots Postgres, stores the Postgres live-tests config glob via
`gateway --store-config`, then runs the full live suite against a gateway
booted from the database with `TENSORZERO_INTERNAL_FLAG_ENABLE_CONFIG_IN_DATABASE=true`.

Unfortunately these tests need to run serially because concurrent edits conflict;
I don't think we can get around this.

Add e2e tests for config-editing PR #7185 review fixes

Covers the three behavioral fixes from the PR review:

- `test_config_editing_apply_persists_transitive_include_templates`:
  regression test for the Codex P1 — after applying a config whose
  template transitively includes a filesystem-backed template via
  MiniJinja `{% include %}`, the discovered template must be persisted
  in `stored_files` with the `ui-config-editor` creation source.
  Without the `write_stored_config_in_tx` merge fix, the row is never
  written and a reload from stored-config tables on a fresh gateway
  would break. Also sanity-checks the rendered system prompt on a
  downstream inference.

- `test_config_editing_validate_accepts_transitive_include`: exercises
  the validate endpoint's happy path for filesystem discovery (the
  client-side precheck before `Apply` is clicked).

- `test_config_editing_cas_rejects_stale_base_signature_for_new_edit`:
  after the CAS check simplification (removal of the
  `canonical_signature` short-circuit), a new edit built on top of a
  stale `base_signature` must still 409.

Also updates the no-op apply test's doc comment now that the
special-case short-circuit has been removed, and fixes stale
`config::` imports to `config_toml::`.
- Add live-tests-config-in-database to ALLOWED_SKIP in check-all-general-jobs-passed.sh
  so merge-queue runs don't fail when the job is legitimately skipped
- Delete both transitive-include e2e tests: the config-in-database gateway
  explicitly bans template_filesystem_access, so these tests can never pass
  in the config-editing workflow
@shuyangli shuyangli force-pushed the sl/config-editing-e2e-tests branch from 2f0a711 to 091eabf Compare April 13, 2026 19:50
@shuyangli shuyangli enabled auto-merge April 13, 2026 19:50
@shuyangli
Copy link
Copy Markdown
Contributor Author

@BugBot review

Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 091eabf. Configure here.

@shuyangli shuyangli added this pull request to the merge queue Apr 13, 2026
Merged via the queue into main with commit e24a55c Apr 13, 2026
83 checks passed
@shuyangli shuyangli deleted the sl/config-editing-e2e-tests branch April 13, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants