Repository navigation
test(coderd): cover the turn tool policy at buildTurnEnvironment - #30867
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
hugodutka
left a comment
There was a problem hiding this comment.
I had an agent compare the new test to the deleted tests:
#30867 looks good as additive coverage. It does not justify merging #30868 unchanged. I found three concrete coverage gaps and reproduced regressions that the removed tests catch.
- Explore can gain edit_files undetected. Neither new Explore case excludes it, and the surviving Explore helper test never supplies it. I changed the production policy from false to true: the entire chatd suite passes with #30868’s deletions applied, while the old read-only and persisted-snapshot tests fail. Add edit_files to the negative expectations. Affected case
- Explore can incorrectly lose MCP tools that lack plan approval. The fixture selects blocked, but ExploreChild never requires blocked__echo; its required external tools both have plan approval. Filtering Explore MCPs to plan-approved servers leaves the new test green and fails the removed persisted-snapshot test. Require blocked__echo in that case. Missing expectation
- The SendMessage snapshot-mutation scenario has no replacement. The table prepares one already-correct chat without sending another message. The surviving EditMessage test exercises a different caller. Bypassing snapshot protection specifically in SendMessage passes both tests and fails the removed SendMessage test. Preserve a focused test that attempts replacement, checks persisted IDs, and prepares the next turn. Current preparation
Two smaller omissions also deserve restoration: exclusion of an ordinary unselected MCP server, and the full child-prompt assertion rejecting root-only propose_plan instructions. The new prompt assertions inspect only first lines.
|
@hugodutka thanks, c2b4724 covers all five:
I applied each regression you described to this branch. The previous version of the test passed with all five, and the current version fails with each one. The description lists the failing cases. The layers above took this change through merge commits, so #30868, #30869, and #30870 have the same diffs as before.
|
|
@codex review
|
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
I had an agent look again:
I used the following prompt with Codex, GPT 6.1 Sol High:
Could you run this until it reports no gaps anymore? |
|
@codex review
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7edd661423
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review
|
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…ilter (#30864) Removes `allowedBehaviorToolNames`, `allToolNames`, and `TestAllowedBehaviorToolNames`, and the active-name check they fed in `appendDynamicTools`. The check never skipped a tool. `appendDynamicTools` returns early for Explore mode and plan mode, so at the check, `allowedBehaviorToolNames` always returned every tool name. A dynamic tool that shares a built-in tool's name is still dropped with a warning. Validation: `go test ./coderd/x/chatd/ -run 'Dynamic|AllowedExplore|StopAfterPlan'` passes, and the pre-commit hook (build, `lint/go`, `lint/go/new-code`) passed. ## Stack This stack splits #28757, as requested in review. It moves the code that prepares a chat turn into one file, `turn_environment.go`, and moves the tool policy tests from full turns to a table test. chatd behaves as on `main`, except for the debug log in #30865. 1. #30864 Remove the no-op `allowedBehaviorToolNames` filter 2. #30865 Log workspace agent lookup errors in turn preparation 3. #30866 Rename `prepareGeneration` to `buildTurnEnvironment` 4. #30867 Cover the turn tool policy at `buildTurnEnvironment` 5. #30868 Remove seven end-to-end tool policy tests 6. #30869 Name the `*Server` receiver `p` in `turn_environment.go` 7. #30870 Move the turn preparation helpers into `turn_environment.go` > Xum, an AI agent, opened this PR for @ibetitsmike and wrote this description.
1a00517 to
6475b29
Compare
…aration (#30865) Logs the error from `getWorkspaceAgent` at debug level in turn preparation. `prepareGeneration` discarded it. The turn still goes on without the agent, as before. The agent only adds its OS and working directory to the instruction header. Validation: build and vet pass, and the pre-commit hook passed. ## Stack This stack splits #28757, as requested in review. It moves the code that prepares a chat turn into one file, `turn_environment.go`, and moves the tool policy tests from full turns to a table test. chatd behaves as on `main`, except for the debug log in #30865. 1. #30864 Remove the no-op `allowedBehaviorToolNames` filter 2. #30865 Log workspace agent lookup errors in turn preparation 3. #30866 Rename `prepareGeneration` to `buildTurnEnvironment` 4. #30867 Cover the turn tool policy at `buildTurnEnvironment` 5. #30868 Remove seven end-to-end tool policy tests 6. #30869 Name the `*Server` receiver `p` in `turn_environment.go` 7. #30870 Move the turn preparation helpers into `turn_environment.go` > Xum, an AI agent, opened this PR for @ibetitsmike and wrote this description.
6475b29 to
4b5a295
Compare
TestBuildTurnEnvironmentToolPolicy checks the tools, active tools, provider tools, and system prompt overlays that buildTurnEnvironment returns for root, plan mode, and Explore chats. All five cases share one fixture with approved, unapproved, allowlisted, and Force On MCP servers, an inline and a workspace MCP server, and a model with the web_search provider tool. Also cover the pending user row split for compaction and the project instructions in the memory test.
Review of #30867 found regressions that only the end-to-end tests removed in #30868 caught. TestBuildTurnEnvironmentToolPolicy now: - rejects edit_files in Explore chats - requires MCP tools without plan approval in Explore children - rejects an MCP server the chat does not select - sends a SendMessage that asks to replace an Explore child's MCP servers, checks the stored selection, and builds the next turn - searches the full system prompt, so a plan-mode child must not get the root-only propose_plan instruction
An audit of every check in the seven tests that #30868 removes found four regressions that only those tests caught: - a plan-mode root chat does not get spawn_agent - a general sub-agent of a plan-mode chat loses the plan mode - POST /chats drops the requested plan mode - Server.CreateChat drops the MCP selection of a root Explore chat PlanRoot now requires spawn_agent and runs it to check the child's plan mode. TestCreateExploreChatPreservesMCPSelection checks the stored selection, and TestPostChats/Success creates a plan-mode chat.
Children load inline MCP servers from the root chat, so the inline server that the table seeded on each child chat never loaded and the child inline checks passed for any policy. The fixture now seeds one inline server without allow_in_subagents on the parent, and ExploreChild rejects inline__echo.
Renames the turn preparation step after what it returns: the environment for one turn. Only names change, plus one small change to how callers release the environment. - `generation_preparer.go` becomes `turn_environment.go`, and its test file becomes `turn_environment_internal_test.go`. GitHub shows both as renames. A comment in `compactionTriggers.test.ts` that names the test file follows the rename. - `prepareGeneration` becomes `buildTurnEnvironment`, and its result type `generationPrepared` becomes `turnEnvironment`. Four `TestPrepareGeneration*` tests become `TestBuildTurnEnvironment*`. - The `Cleanup` field becomes the unexported `cleanup`, and callers use `Close()`. `Close()` does nothing on a zero value. The type stays in `generation.go` for now. #30870 moves it next to `buildTurnEnvironment`. Validation: build and vet pass, the renamed tests and the `toolinput` tests pass, and the pre-commit hook passed. ## Stack This stack splits #28757, as requested in review. It moves the code that prepares a chat turn into one file, `turn_environment.go`, and moves the tool policy tests from full turns to a table test. chatd behaves as on `main`, except for the debug log in #30865. 1. #30864 Remove the no-op `allowedBehaviorToolNames` filter 2. #30865 Log workspace agent lookup errors in turn preparation 3. #30866 Rename `prepareGeneration` to `buildTurnEnvironment` 4. #30867 Cover the turn tool policy at `buildTurnEnvironment` 5. #30868 Remove seven end-to-end tool policy tests 6. #30869 Name the `*Server` receiver `p` in `turn_environment.go` 7. #30870 Move the turn preparation helpers into `turn_environment.go` > Xum, an AI agent, opened this PR for @ibetitsmike and wrote this description.
4b5a295 to
3c97175
Compare
Adds
TestBuildTurnEnvironmentToolPolicy, a table test of the tool policy at the place chatd applies it: the result ofbuildTurnEnvironment. #30868 then removes seven end-to-end tests that checked the same policy through a full turn.The six cases (
Root,PlanRoot,PlanChild,ExploreRoot,ExploreChild,ExploreChildAfterMCPReplacement) share one fixture:allow_in_subagentson the parent of the child chats, and one workspace MCP serverweb_searchprovider toolEach case builds the tool definitions with
chatloop.BuildToolDefinitions, the same function the chat loop uses for the model request. It then compares the tool names, active tools, and provider tools with the case. It also searches the full system prompt for the plan mode and Explore mode overlays and for the root-onlypropose_planinstruction.ExploreChildAfterMCPReplacementfirst callsSendMessagewith a request to replace the Explore child's MCP servers with the unselected one. The test checks that the stored selection does not change, then builds the next turn from the stored chat and messages.PlanRootruns itsspawn_agenttool and checks that the general sub-agent keeps the plan mode.The PR also adds:
TestCreateExploreChatPreservesMCPSelection:Server.CreateChatstores the MCP servers that a root Explore chat selectsTestPostChats/Success:POST /chatsstores the requested plan modeTestBuildTurnEnvironmentSplitsPendingUserRowsFromCompactionPromptand a project instructions assertion inTestBuildTurnEnvironmentMemoryValidation
chatd.gofailsExploreRootandExploreChild.maincaught. Two examples: root Explore chats get MCP tools, and Explore chats get workspace MCP tools. That run used an earliermainand the refactor(coderd/x/chatd): move turn preparation into turn_environment.go #28757 branch, which has the same test code.spawn_agentcall, and tried a mutant for each check without an obvious replacement. A mutant counts as a gap only if a removed test fails with it and the rest ofcoderd/x/chatd/...plus the chat, MCP, Explore, and plan tests incoderdandenterprise/coderdpass. The audits found the last two rows. They also found one gap that this PR leaves open; test(coderd/x/chatd): remove end-to-end tool policy tests #30868 explains why.coderd/x/chatd/...that only the seven removed tests run. All 4 are error returns.edit_filesExploreRoot,ExploreChildExploreChild,ExploreChildAfterMCPReplacementSendMessagereplaces an Explore child's MCP serversExploreChildAfterMCPReplacementRootpropose_planinstructionPlanChildspawn_agentPlanRootServer.CreateChatdrops the MCP servers of a root Explore chatTestCreateExploreChatPreservesMCPSelectionPlanRootPOST /chatsdrops the requested plan modeTestPostChats/SuccessStack
This stack splits #28757, as requested in review. It moves the code that prepares a chat turn into one file,
turn_environment.go, and moves the tool policy tests from full turns to a table test. chatd behaves as onmain, except for the debug log in #30865.allowedBehaviorToolNamesfilterprepareGenerationtobuildTurnEnvironmentbuildTurnEnvironment*Serverreceiverpinturn_environment.goturn_environment.go