Skip to content

test(coderd): cover the turn tool policy at buildTurnEnvironment - #30867

Merged
ibetitsmike merged 4 commits into
mainfrom
mike/chatd-turn-env-split/tool-policy-table-test
Oct 16, 2026
Merged

ibetitsmike merged 4 commits into
mainfrom
mike/chatd-turn-env-split/tool-policy-table-test

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Adds TestBuildTurnEnvironmentToolPolicy, a table test of the tool policy at the place chatd applies it: the result of buildTurnEnvironment. #30868 then removes seven end-to-end tests that checked the same policy through a full turn.

The six cases (Root, PlanRoot, PlanChild, ExploreRoot, ExploreChild, ExploreChildAfterMCPReplacement) share one fixture:

  • three MCP servers that every chat selects: approved for plan mode, not approved, and approved with a tool allowlist
  • two MCP servers that no chat selects: one Force On and one ordinary
  • one inline MCP server on each root chat, one without allow_in_subagents on the parent of the child chats, and one workspace MCP server
  • a model with the web_search provider tool

Each case builds the tool definitions with chatloop.BuildToolDefinitions, the same function the chat loop uses for the model request. It then compares the tool names, active tools, and provider tools with the case. It also searches the full system prompt for the plan mode and Explore mode overlays and for the root-only propose_plan instruction.

ExploreChildAfterMCPReplacement first calls SendMessage with a request to replace the Explore child's MCP servers with the unselected one. The test checks that the stored selection does not change, then builds the next turn from the stored chat and messages. PlanRoot runs its spawn_agent tool and checks that the general sub-agent keeps the plan mode.

The PR also adds:

  • TestCreateExploreChatPreservesMCPSelection: Server.CreateChat stores the MCP servers that a root Explore chat selects
  • a plan-mode request in TestPostChats/Success: POST /chats stores the requested plan mode
  • TestBuildTurnEnvironmentSplitsPendingUserRowsFromCompactionPrompt and a project instructions assertion in TestBuildTurnEnvironmentMemory

Validation

  • For each new assertion, Xum removed the code that the assertion checks and saw the test fail. On this branch, for example, removing the Explore overlay in chatd.go fails ExploreRoot and ExploreChild.
  • In a mutation run, these tests catch 7 mutants (copies of the code with one policy branch removed) that no test on main caught. Two examples: root Explore chats get MCP tools, and Explore chats get workspace MCP tools. That run used an earlier main and the refactor(coderd/x/chatd): move turn preparation into turn_environment.go #28757 branch, which has the same test code.
  • Review found regressions that only the end-to-end tests removed in test(coderd/x/chatd): remove end-to-end tool policy tests #30868 caught. Xum applied each one to this branch: the first version of these tests passes, and the current version fails in the cases below.
  • Two independent audits then listed every check in the seven removed tests, including implicit ones such as a scripted spawn_agent call, and tried a mutant for each check without an obvious replacement. A mutant counts as a gap only if a removed test fails with it and the rest of coderd/x/chatd/... plus the chat, MCP, Explore, and plan tests in coderd and enterprise/coderd pass. The audits found the last two rows. They also found one gap that this PR leaves open; test(coderd/x/chatd): remove end-to-end tool policy tests #30868 explains why.
  • A line coverage comparison found 4 statements in coderd/x/chatd/... that only the seven removed tests run. All 4 are error returns.
Regression Failing cases
Explore chats get edit_files ExploreRoot, ExploreChild
Explore children keep only the MCP servers approved for plan mode ExploreChild, ExploreChildAfterMCPReplacement
SendMessage replaces an Explore child's MCP servers ExploreChildAfterMCPReplacement
Chats outside Explore mode get every enabled MCP server in the organization Root
A plan-mode child gets the root-only propose_plan instruction PlanChild
A plan-mode root chat does not get spawn_agent PlanRoot
Server.CreateChat drops the MCP servers of a root Explore chat TestCreateExploreChatPreservesMCPSelection
A general sub-agent of a plan-mode chat loses the plan mode PlanRoot
POST /chats drops the requested plan mode TestPostChats/Success

Stack

This stack splits #28757, as requested in review. It moves the code that prepares a chat turn into one file, turn_environment.go, and moves the tool policy tests from full turns to a table test. chatd behaves as on main, except for the debug log in #30865.

  1. refactor(coderd/x/chatd): remove the no-op allowedBehaviorToolNames filter #30864 Remove the no-op allowedBehaviorToolNames filter
  2. chore(coderd/x/chatd): log workspace agent lookup errors in turn preparation #30865 Log workspace agent lookup errors in turn preparation
  3. refactor: rename prepareGeneration to buildTurnEnvironment #30866 Rename prepareGeneration to buildTurnEnvironment
  4. test(coderd): cover the turn tool policy at buildTurnEnvironment #30867 Cover the turn tool policy at buildTurnEnvironment
  5. test(coderd/x/chatd): remove end-to-end tool policy tests #30868 Remove seven end-to-end tool policy tests
  6. refactor(coderd/x/chatd): name the Server receiver p in turn_environment.go #30869 Name the *Server receiver p in turn_environment.go
  7. refactor(coderd/x/chatd): move turn preparation helpers into turn_environment.go #30870 Move the turn preparation helpers into turn_environment.go

Xum, an AI agent, opened this PR for @ibetitsmike and wrote this description.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-16T13:33:40.471147Z 1a00517 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@hugodutka hugodutka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had an agent compare the new test to the deleted tests:

#30867 looks good as additive coverage. It does not justify merging #30868 unchanged. I found three concrete coverage gaps and reproduced regressions that the removed tests catch.

  1. Explore can gain edit_files undetected. Neither new Explore case excludes it, and the surviving Explore helper test never supplies it. I changed the production policy from false to true: the entire chatd suite passes with #30868’s deletions applied, while the old read-only and persisted-snapshot tests fail. Add edit_files to the negative expectations. Affected case
  2. Explore can incorrectly lose MCP tools that lack plan approval. The fixture selects blocked, but ExploreChild never requires blocked__echo; its required external tools both have plan approval. Filtering Explore MCPs to plan-approved servers leaves the new test green and fails the removed persisted-snapshot test. Require blocked__echo in that case. Missing expectation
  3. The SendMessage snapshot-mutation scenario has no replacement. The table prepares one already-correct chat without sending another message. The surviving EditMessage test exercises a different caller. Bypassing snapshot protection specifically in SendMessage passes both tests and fails the removed SendMessage test. Preserve a focused test that attempts replacement, checks persisted IDs, and prepares the next turn. Current preparation
    Two smaller omissions also deserve restoration: exclusion of an ordinary unselected MCP server, and the full child-prompt assertion rejecting root-only propose_plan instructions. The new prompt assertions inspect only first lines.

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@hugodutka thanks, c2b4724 covers all five:

  1. edit_files in Explore: ExploreRoot and ExploreChild now reject it. Root and PlanRoot require it, so a renamed tool cannot make the check pass by accident.
  2. MCP tools without plan approval: ExploreChild now requires blocked__echo.
  3. SendMessage snapshot: the new ExploreChildAfterMCPReplacement case calls SendMessage on a waiting Explore child with a request to switch to an unselected server. It checks that the stored MCP server IDs did not change, then builds the next turn from the stored chat and messages.
  4. Unselected server: the fixture now has an ordinary unselected server. Root, ExploreChild, and the replacement case reject unselected__echo.
  5. Full prompt: the test now searches the whole system prompt. PlanRoot must contain "When the plan is ready, call propose_plan", and the other cases must not.

I applied each regression you described to this branch. The previous version of the test passed with all five, and the current version fails with each one. The description lists the failing cases. The layers above took this change through merge commits, so #30868, #30869, and #30870 have the same diffs as before.

Xum, an AI agent, posted this comment for @ibetitsmike.

@ibetitsmike
ibetitsmike requested a review from hugodutka October 16, 2026 10:28
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum, an AI agent, posted this comment for @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: c2b4724df6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@hugodutka

Copy link
Copy Markdown
Contributor

I had an agent look again:

At c2b4724, all five previously reported gaps are fixed. I reproduced two additional gaps:

  1. Plan-mode delegation can disappear undetected. PlanRoot’s expectations omit spawn_agent. I temporarily disabled subagent tools for plan-mode roots. The replacement table passed; the deleted TestPlanModeSubagentChatExcludesAskUserQuestion failed. Add spawn_agent to PlanRoot.want.
  2. Creating a root Explore chat can discard its selected MCP servers undetected. The new fixture calls chatstate.CreateChat, bypassing Server.CreateChat. I temporarily made Server.CreateChat clear Explore selections. The replacement table passed; the deleted TestRootExploreChatStaysBuiltinOnlyAtRuntime failed on its persisted-selection assertion. Preserve this with a focused creation test.

I used the following prompt with Codex, GPT 6.1 Sol High:

is https://github.com/coder/coder/pull/30867 good to merge? does it really cover all the scenarios that the tests removed in https://github.com/coder/coder/pull/30868 covered?

Could you run this until it reports no gaps anymore?

@ibetitsmike ibetitsmike changed the title test(coderd/x/chatd): cover the turn tool policy at buildTurnEnvironment test(coderd): cover the turn tool policy at buildTurnEnvironment Oct 16, 2026
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum, an AI agent, posted this comment for @ibetitsmike.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7edd661423

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/turn_environment_internal_test.go Outdated
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum, an AI agent, posted this comment for @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 1a00517db5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ibetitsmike added a commit that referenced this pull request Oct 16, 2026
…ilter (#30864)

Removes `allowedBehaviorToolNames`, `allToolNames`, and
`TestAllowedBehaviorToolNames`, and the active-name check they fed in
`appendDynamicTools`.

The check never skipped a tool. `appendDynamicTools` returns early for
Explore mode and plan mode, so at the check, `allowedBehaviorToolNames`
always returned every tool name. A dynamic tool that shares a built-in
tool's name is still dropped with a warning.

Validation: `go test ./coderd/x/chatd/ -run
'Dynamic|AllowedExplore|StopAfterPlan'` passes, and the pre-commit hook
(build, `lint/go`, `lint/go/new-code`) passed.

## Stack

This stack splits #28757, as requested in review. It moves the code that
prepares a chat turn into one file, `turn_environment.go`, and moves the
tool policy tests from full turns to a table test. chatd behaves as on
`main`, except for the debug log in #30865.

1. #30864 Remove the no-op `allowedBehaviorToolNames` filter
2. #30865 Log workspace agent lookup errors in turn preparation
3. #30866 Rename `prepareGeneration` to `buildTurnEnvironment`
4. #30867 Cover the turn tool policy at `buildTurnEnvironment`
5. #30868 Remove seven end-to-end tool policy tests
6. #30869 Name the `*Server` receiver `p` in `turn_environment.go`
7. #30870 Move the turn preparation helpers into `turn_environment.go`

> Xum, an AI agent, opened this PR for @ibetitsmike and wrote this
description.
@ibetitsmike
ibetitsmike force-pushed the mike/chatd-turn-env-split/tool-policy-table-test branch from 1a00517 to 6475b29 Compare October 16, 2026 20:44
ibetitsmike added a commit that referenced this pull request Oct 16, 2026
…aration (#30865)

Logs the error from `getWorkspaceAgent` at debug level in turn
preparation. `prepareGeneration` discarded it.

The turn still goes on without the agent, as before. The agent only adds
its OS and working directory to the instruction header.

Validation: build and vet pass, and the pre-commit hook passed.

## Stack

This stack splits #28757, as requested in review. It moves the code that
prepares a chat turn into one file, `turn_environment.go`, and moves the
tool policy tests from full turns to a table test. chatd behaves as on
`main`, except for the debug log in #30865.

1. #30864 Remove the no-op `allowedBehaviorToolNames` filter
2. #30865 Log workspace agent lookup errors in turn preparation
3. #30866 Rename `prepareGeneration` to `buildTurnEnvironment`
4. #30867 Cover the turn tool policy at `buildTurnEnvironment`
5. #30868 Remove seven end-to-end tool policy tests
6. #30869 Name the `*Server` receiver `p` in `turn_environment.go`
7. #30870 Move the turn preparation helpers into `turn_environment.go`

> Xum, an AI agent, opened this PR for @ibetitsmike and wrote this
description.
@ibetitsmike
ibetitsmike force-pushed the mike/chatd-turn-env-split/tool-policy-table-test branch from 6475b29 to 4b5a295 Compare October 16, 2026 20:59
Base automatically changed from mike/chatd-turn-env-split/rename-turn-environment to main October 16, 2026 21:20
@ibetitsmike
ibetitsmike requested a review from a team as a code owner October 16, 2026 21:20
TestBuildTurnEnvironmentToolPolicy checks the tools, active tools,
provider tools, and system prompt overlays that buildTurnEnvironment
returns for root, plan mode, and Explore chats. All five cases share one
fixture with approved, unapproved, allowlisted, and Force On MCP
servers, an inline and a workspace MCP server, and a model with the
web_search provider tool.

Also cover the pending user row split for compaction and the project
instructions in the memory test.
Review of #30867 found regressions that only the end-to-end tests
removed in #30868 caught. TestBuildTurnEnvironmentToolPolicy now:

- rejects edit_files in Explore chats
- requires MCP tools without plan approval in Explore children
- rejects an MCP server the chat does not select
- sends a SendMessage that asks to replace an Explore child's MCP
  servers, checks the stored selection, and builds the next turn
- searches the full system prompt, so a plan-mode child must not get
  the root-only propose_plan instruction
An audit of every check in the seven tests that #30868 removes found
four regressions that only those tests caught:

- a plan-mode root chat does not get spawn_agent
- a general sub-agent of a plan-mode chat loses the plan mode
- POST /chats drops the requested plan mode
- Server.CreateChat drops the MCP selection of a root Explore chat

PlanRoot now requires spawn_agent and runs it to check the child's plan
mode. TestCreateExploreChatPreservesMCPSelection checks the stored
selection, and TestPostChats/Success creates a plan-mode chat.
Children load inline MCP servers from the root chat, so the inline
server that the table seeded on each child chat never loaded and the
child inline checks passed for any policy. The fixture now seeds one
inline server without allow_in_subagents on the parent, and
ExploreChild rejects inline__echo.
ibetitsmike added a commit that referenced this pull request Oct 16, 2026
Renames the turn preparation step after what it returns: the environment
for one turn. Only names change, plus one small change to how callers
release the environment.

- `generation_preparer.go` becomes `turn_environment.go`, and its test
file becomes `turn_environment_internal_test.go`. GitHub shows both as
renames. A comment in `compactionTriggers.test.ts` that names the test
file follows the rename.
- `prepareGeneration` becomes `buildTurnEnvironment`, and its result
type `generationPrepared` becomes `turnEnvironment`. Four
`TestPrepareGeneration*` tests become `TestBuildTurnEnvironment*`.
- The `Cleanup` field becomes the unexported `cleanup`, and callers use
`Close()`. `Close()` does nothing on a zero value.

The type stays in `generation.go` for now. #30870 moves it next to
`buildTurnEnvironment`.

Validation: build and vet pass, the renamed tests and the `toolinput`
tests pass, and the pre-commit hook passed.

## Stack

This stack splits #28757, as requested in review. It moves the code that
prepares a chat turn into one file, `turn_environment.go`, and moves the
tool policy tests from full turns to a table test. chatd behaves as on
`main`, except for the debug log in #30865.

1. #30864 Remove the no-op `allowedBehaviorToolNames` filter
2. #30865 Log workspace agent lookup errors in turn preparation
3. #30866 Rename `prepareGeneration` to `buildTurnEnvironment`
4. #30867 Cover the turn tool policy at `buildTurnEnvironment`
5. #30868 Remove seven end-to-end tool policy tests
6. #30869 Name the `*Server` receiver `p` in `turn_environment.go`
7. #30870 Move the turn preparation helpers into `turn_environment.go`

> Xum, an AI agent, opened this PR for @ibetitsmike and wrote this
description.
@ibetitsmike
ibetitsmike force-pushed the mike/chatd-turn-env-split/tool-policy-table-test branch from 4b5a295 to 3c97175 Compare October 16, 2026 21:20
@ibetitsmike
ibetitsmike merged commit c42c9af into main Oct 16, 2026
29 checks passed
@ibetitsmike
ibetitsmike deleted the mike/chatd-turn-env-split/tool-policy-table-test branch October 16, 2026 21:34
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants