feat: add computer-use provider selection for AI agents by ibetitsmike · Pull Request #24772 · coder/coder

ibetitsmike · 2026-04-28T10:56:54Z

Adds a deployment-wide setting to select the computer-use provider (Anthropic or OpenAI) for AI agents, plus the OpenAI computer-use runner needed to honor that selection.

The setting is stored in site_configs under agents_computer_use_provider, defaults to Anthropic when unset, and is exposed via experimental GET/PUT endpoints under /api/experimental/chats/config/computer-use-provider. The chatd computer-use tool now dispatches to either runAnthropicComputerUse or runOpenAIComputerUse based on the resolved provider, with provider-specific result metadata for OpenAI screenshots.

Frontend adds a provider dropdown to the Agents Experiments settings page nested under the virtual desktop toggle, with disabled state handling while virtual desktop is off and skeleton loaders while config queries are in flight.

Hugo and Codex review follow-up:

Uses shared provider validation and clearer computer-use constant names.
Removes stale OpenAI pending-safety-checks commentary.
Documents why provider result metadata is needed for OpenAI screenshots.
Keeps the computer-use subagent visible when provider credentials are missing, then returns a clear spawn-time configuration error.
Uses OpenAI's recommended 1600x900 screenshot geometry to preserve the native 16:9 aspect ratio.
Moves OpenAI-specific computer-use helpers into coderd/x/chatd/chatopenai/computeruse after rebasing onto the provider package refactor in main.
Converts OpenAI pixel scroll deltas to Coder desktop wheel-click amounts.
Preserves OpenAI pointer modifiers with key down/up desktop actions and rejects unsupported non-left double-click buttons explicitly.
Maps OpenAI back/forward side-button clicks to browser navigation key actions.
Defaults omitted OpenAI click buttons to left-click.
Retries mouse release cleanup if the final OpenAI drag release fails.
Keeps computer-use subagent availability messages stable when provider config cannot be loaded, while logging the backend error.
Releases remaining OpenAI modifier keys if a synthetic key-up cleanup action fails.
Updates Storybook interaction stories so provider snapshots show the selected final provider.

Mux updated this PR description on behalf of Mike.

ibetitsmike · 2026-04-28T10:57:04Z

/coder-agents-review

coder-agents-review

Solid work. The provider abstraction is well-structured, the tool dispatch switches are consistent across all five enumeration sites, and the OpenAI runner test suite is thorough (10 action types, 16 desktop actions, key normalization table tests, mock clock for wait). The chatloop ResultProviderMetadata plumbing is cleanly integrated. The coordinate-system pinning test is a nice touch.

Severity breakdown: 2 P2, 4 P3, 2 Nit. The P2s are both in the OpenAI runner: a button-mapping gap where OpenAI sends "wheel" but the code maps "middle", and a key normalization whitelist that rejects punctuation characters needed for common shortcuts.

"An admin seeing configured provider "openai" is not configured will stare at this, then open the source." (Hisoka)

site/src/pages/AgentsPage/components/VirtualDesktopSettings.tsx:109

P3 [DEREM-5] The help text still says "the Anthropic provider to be configured," but this PR adds OpenAI as a selectable alternative. An admin who selects OpenAI reads conflicting guidance.

Fix: "and the selected computer use provider to be configured," or omit the provider name entirely. (Mafuuu P3, Nami P3)

🤖

coderd/x/chatd/chattool/computeruse_test.go:462

P3 [DEREM-8] TestComputerUseTool_Run_ConnError and TestComputerUseTool_Run_InvalidInput only exercise the Anthropic code path. The OpenAI path uses a different parser (fantasyopenai.ParseComputerUseInput) with potentially different failure modes. Could we add a parallel TestComputerUseTool_Run_OpenAI_InvalidInput with malformed JSON to cover the OpenAI branch? (Bisky P3)

🤖

🤖 This review was automatically generated with Coder Agents.

ibetitsmike · 2026-04-28T11:44:56Z

/coder-agents-review

coder-agents-review

Round 2. All 9 prior findings addressed cleanly. DEREM-10 (PUT without key check) was contested by the author; the panel evaluated the defense and closed it unanimously (4/4). The runtime now surfaces a specific, actionable error at spawn time, which is consistent with how other admin-toggleable settings work.

The fix commits introduced one regression and one gap: 1 P2, 1 P3.

"The Switch is fully replaced, removing it from the DOM" (Mafuuu, on the story breakage)

🤖 This review was automatically generated with Coder Agents.

ibetitsmike · 2026-04-28T12:14:12Z

/coder-agents-review

coder-agents-review

Round 3. DEREM-16 and DEREM-17 both addressed in a2968a7. All P0-P2 findings across three rounds are resolved.

Two minor items from the final panel scan (1 P3, 1 Nit) noted inline. Neither blocks merge.

Across three rounds: 18 findings raised, 11 fixed by author, 1 closed by panel vote (4/4), 6 dropped by orchestrator. Responsive fixes with no regressions in the final pass.

"Clean decomposition of the OpenAI runner. The drag error-recovery path (mouse_up cleanup on mid-drag failure) is a nice detail." (Kite)

🤖 This review was automatically generated with Coder Agents.

ibetitsmike · 2026-04-28T12:42:38Z

/coder-agents-review

coder-agents-review

Round 4. DEREM-20 addressed (comment added in 34c17cc). DEREM-19 acknowledged by author with substantive reasoning (OpenAI trailing screenshot is intentional UX; graceful fallback bounds consequence).

One P4 from the final panel scan noted inline (navigation key gap, sibling of DEREM-3). Does not block merge.

Full ledger across four rounds: 21 findings raised, 12 fixed by author, 1 closed by panel vote (4/4), 1 acknowledged by author, 7 dropped by orchestrator.

🤖 This review was automatically generated with Coder Agents.

…back

ibetitsmike · 2026-05-04T10:47:20Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: afdf1c0c47

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

hugodutka

Approved, pending 1 comment

ibetitsmike · 2026-05-04T10:58:17Z

@codex review

chatgpt-codex-connector · 2026-05-04T11:07:37Z

Codex Review: Didn't find any major issues. Chef's kiss.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ibetitsmike · 2026-05-04T11:12:45Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 91a4dfa428

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ibetitsmike · 2026-05-04T11:34:16Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 69f22accca

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ibetitsmike · 2026-05-04T12:00:04Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 651f3ff219

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ibetitsmike · 2026-05-04T12:22:19Z

@codex review

chatgpt-codex-connector · 2026-05-04T12:30:55Z

Codex Review: Didn't find any major issues. 🎉

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ibetitsmike · 2026-05-04T12:45:29Z

@codex review

chatgpt-codex-connector · 2026-05-04T12:53:53Z

Codex Review: Didn't find any major issues. Nice work!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ibetitsmike · 2026-05-04T14:30:09Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e4daf47f26

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ibetitsmike · 2026-05-04T15:07:33Z

@codex review

github-actions Bot assigned ibetitsmike Apr 28, 2026

coder-agents-review Bot reviewed Apr 28, 2026

View reviewed changes

Comment thread site/src/pages/AgentsPage/AgentSettingsExperimentsPageView.stories.tsx Outdated

Comment thread coderd/x/chatd/chatd.go Outdated

coder-agents-review Bot approved these changes Apr 28, 2026

View reviewed changes

Comment thread coderd/x/chatd/chattool/computeruse.go

Comment thread coderd/x/chatd/chattool/computeruse.go Outdated

coder-agents-review Bot approved these changes Apr 28, 2026

View reviewed changes

Comment thread coderd/x/chatd/chattool/computeruse.go Outdated

ibetitsmike requested a review from hugodutka April 30, 2026 11:07

hugodutka reviewed Apr 30, 2026

View reviewed changes

ibetitsmike force-pushed the mike/agents-computer-use-provider branch from a122ca1 to f100df5 Compare April 30, 2026 19:04

ibetitsmike marked this pull request as ready for review May 4, 2026 09:11

ibetitsmike added 17 commits May 4, 2026 09:19

feat: add agents computer use provider setting

4bf9ec4

chore(deps): bump fantasy to include openai computer use

b49feb1

test(codersdk/workspacesdk): pin computer use desktop geometry

346d831

feat(site): add computer use provider dropdown

2fc4057

feat(coderd/x/chatd/chatloop): add provider tool result metadata call…

d97c922

…back

feat(coderd/x/chatd/chattool): add openai computer use runner

09e9f3b

feat(coderd/x/chatd): select computer use provider

3bf8fef

refactor(coderd/x/chatd/chattool): unify computer use API

6692e0e

feat(coderd/x/chatd/chattool): bump OpenAI computer use default

90ea9de

fix(coderd): address computer-use review findings

ac9f345

feat(site/src/pages/AgentsPage): show agent settings loading states

3f4b331

fix(coderd): address computer-use review findings

e190cac

fix(site/src/pages/AgentsPage): clarify settings loading state

373050c

fix: address agents review computer-use feedback

cff8db0

chore(coderd/x/chatd/chattool): explain OpenAI screenshot continue

a91e122

fix(site/src/pages/AgentsPage): address provider toggle review

952846f

fix: use OpenAI computer-use resolution from review feedback

6c15c9d

fix(coderd/x/chatd): handle OpenAI side-button clicks

afdf1c0

chatgpt-codex-connector Bot reviewed May 4, 2026

View reviewed changes

Comment thread coderd/x/chatd/chatopenai/computeruse/computeruse.go

fix(coderd/x/chatd): default OpenAI click button

2eff832

hugodutka approved these changes May 4, 2026

View reviewed changes

Comment thread codersdk/workspacesdk/display.go Outdated

fix(codersdk/workspacesdk): preserve OpenAI desktop aspect ratio

91a4dfa

chatgpt-codex-connector Bot reviewed May 4, 2026

View reviewed changes

Comment thread coderd/x/chatd/chatopenai/computeruse/computeruse.go

fix(coderd/x/chatd): retry OpenAI drag cleanup

69f22ac

chatgpt-codex-connector Bot reviewed May 4, 2026

View reviewed changes

Comment thread coderd/x/chatd/chattool/computeruse.go

fix(coderd/x/chatd): document OpenAI tool geometry handling

651f3ff

chatgpt-codex-connector Bot reviewed May 4, 2026

View reviewed changes

Comment thread coderd/x/chatd/chatopenai/computeruse/computeruse.go

Comment thread coderd/x/chatd/chatopenai/computeruse/computeruse.go

fix(coderd/x/chatd): release OpenAI input state on errors

4f07e70

fix(site/src/pages/AgentsPage): update provider selection stories

608e887

fix(site/src/pages/AgentsPage): align provider story assertions

e4daf47

chatgpt-codex-connector Bot reviewed May 4, 2026

View reviewed changes

Comment thread coderd/x/chatd/chatopenai/computeruse/computeruse.go Outdated

Comment thread coderd/x/chatd/subagent_catalog.go Outdated

fix(coderd/x/chatd): harden computer-use cleanup

53dc56f

ibetitsmike merged commit 0bb0993 into main May 4, 2026
52 of 54 checks passed

ibetitsmike deleted the mike/agents-computer-use-provider branch May 4, 2026 18:30

github-actions Bot locked and limited conversation to collaborators May 4, 2026

Conversation

ibetitsmike commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ibetitsmike commented Apr 28, 2026

Uh oh!

coder-agents-review Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ibetitsmike commented Apr 28, 2026

Uh oh!

coder-agents-review Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ibetitsmike commented Apr 28, 2026

Uh oh!

coder-agents-review Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ibetitsmike commented Apr 28, 2026

Uh oh!

coder-agents-review Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ibetitsmike commented May 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

hugodutka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ibetitsmike commented May 4, 2026

Uh oh!

chatgpt-codex-connector Bot commented May 4, 2026

Uh oh!

ibetitsmike commented May 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

ibetitsmike commented May 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

ibetitsmike commented May 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

ibetitsmike commented Apr 28, 2026 •

edited

Loading