feat(computer): a browser computer's page tools: navigate, history, tabs, dialogs and uploads, and every frame's page - #473

What and why

An agent on a browser computer can now do what a desktop browser's UI would let it do:

• go to a URL;
• go back, go forward or reload;
• list, open, switch and close tabs;
• answer a JavaScript dialog;
• pick files in a file chooser.

Before this, a pixel agent on #471's page computer couldn't navigate at all, and a dialog froze the page where the agent couldn't see it. This is PR 2 of 3 for the browser as a computer, part of FD-5116. Its five decisions are in the design note, under "PR 2, the page tools".

The page group, in the protocol (decision 1). The protocol reserves computer_* names, so the tools are an optional operation group, like the pointer group: @navigate, @history, @tabs, @dialog, @upload and @page_state. A driver that implements all six serves:

• computer_navigate {url};
• computer_history {direction: back | forward | reload};
• computer_tabs {action: list | open | switch | close, tab?, url?}, which answers with the tabs after it;
• computer_dialog {accept, text?};
• computer_upload {paths}, files in ~/Uploads.

The driver also declares computer-page/v1. Desktop drivers don't change.

Every frame names its page (decision 2). Every frame such a driver returns carries an optional page field:

• tab, url and title;
• dialog, while one is open, with its type, message and default_text;
• file_chooser, while one is open, with multiple.

This covers screenshots, and the input and wait results that return a frame. The page is the one the screenshot was captured on, read in the same call, so a tab switch between the two can't mislabel it. The summary says the same in a line.

General browser/v1 operations behind the tools (decision 3). They're all framework-only, for page drivers and agentenv's own steps alike, and M5.11 reuses the tab operations:

• navigate and history;
• tabs/list, tabs/open, tabs/activate and tabs/close;
• dialog/answer and files/choose;
• page/state.

Tabs are named by their CDP target ids (decision 4), as the browser record has them.

Input while a dialog is open (decision 5). While a dialog is open on the page, the input it blocks is refused at once with a new dialog_open error and applied: "no". That covers click, drag, scroll, type, key, the pointer and keyboard tools, and also navigate and history. Tabs, the dialog answer and uploads still go through. Screenshots still work: a tab whose page a dialog keeps from painting shows its last capture rather than white.

Also in the browser's control connection:

• Input that opens a dialog returns as soon as it opens, when the dialog is left to the agent. The page doesn't answer that input until the dialog is answered, so otherwise a click on a confirm() button would hold the input queue, with computer_dialog queued behind it. A dialog the browser answers itself doesn't stop the input: the rest of it is sent.
• How the active tab is chosen:
  • the tab a dialog is open on, whose page it keeps from answering;
  • else the one whose page has focus;
  • else the one agentenv last brought to the front.
• File choosers: headless, so on a page computer, they're intercepted and noted for computer_upload. On a desktop the browser's own chooser still opens.
• The record: each dialog in it now has its default_text. A dialog answered through computer_dialog is recorded as the agent's answer.
• The base: ubuntu-24.04-headless gains ~/Uploads.

Not in this PR:

• the HTML Fixture Desk for gate 3's input checks on a page (PR 3);
• cookies, storage and downloads over CDP (M5.11).

Size: 1,012 lines added, about 470 of them tests.

How it was tested

• ruff check: clean.
• Protocol tests: 1015 passed. The new ones cover:
  • the page group's card and tools, which desktop drivers don't get;
  • page on screenshots and on input results' frames, with the summary;
  • the tools reaching the driver, and computer_tabs's tabs;
  • input refused dialog_open until computer_dialog, with computer_tabs still served;
  • the control's navigation and history, and why it can't;
  • tabs brought to the front and closed;
  • a dialog's tab being the active one, its state, and the agent's answer;
  • the last capture while a dialog blocks painting;
  • input that opens a dialog returning once it opens, and going on past one answered automatically;
  • a frame naming the page its capture came with;
  • the file chooser taking files only from ~/Uploads, as many as it allows.
• Unit tests: 11,792 passed. The page driver's new tests cover its page operations' calls, its page state, and the browser's refusals passed on as invalid_argument. test_a_token_signs_the_clone was left out, because it waits on git credential fill on this laptop, on main too.
• Integration, slow tier, on agent-environments-admin VMs (amd64 c6i.2xlarge, arm64 c7g.2xlarge): 13 passed on each, at e7f89b2.
  • browser_computer_test.py, 4: the new journey has an agent use only its MCP tools on a page computer serving a small test site. It navigates, goes back and forward, answers a confirm() its click opened (refused dialog_open before it does), opens, switches and closes a tab, and uploads a file from ~/Uploads to a file input. Each step is checked in its screenshot's page, and the browser's record has the agent's answer and the tab agentenv closed.
  • chromium_app_test.py, 7, and website_browser_test.py, 2, gate 3 among them. The second's methods check now lists the new operations; it failed on both VMs and in CI before that.
• Greptile: 3/5 on the first commit, with two P1s: a dialog the browser answered itself cut the rest of the input short, and a screenshot could name a tab other than the one captured. Both are fixed in 23d3eeb, with tests, and the re-review is 5/5.

🤖 Generated with Claude Code

Confidence Score: 5/5

Summary

Adds browser navigation, history, tabs, dialog answers, and uploads to page computers. Frames now include the page they show.

• Browser agents can navigate pages and manage tabs with page tools.
• Each browser frame names the page and open controls it shows.
• Agents can see and answer page dialogs without leaving input stuck.
• Agents can provide files from ~/Uploads to a page’s open chooser.

_{Reviews (3) · Last reviewed commit: "Expect the browser's new page operations..." · Reviewed by Greptile}