Skip to content

remove LLM calls to detect prompts for input, add fg terminal support for send_to_terminal/get_terminal_output#308587

Merged
meganrogge merged 8 commits intomainfrom
merogge/revamp-terminal-tool-2
Apr 9, 2026
Merged

remove LLM calls to detect prompts for input, add fg terminal support for send_to_terminal/get_terminal_output#308587
meganrogge merged 8 commits intomainfrom
merogge/revamp-terminal-tool-2

Conversation

@meganrogge
Copy link
Copy Markdown
Collaborator

@meganrogge meganrogge commented Apr 8, 2026

fixes #308296
fixes #308030
fixes #288570

Now that the agent has send_to_terminal and get_terminal_output at its disposal, there's no need for the terminal tool to make extra LLM calls or show elicitation UI when a command prompts for input — the agent can detect the prompt, decide what to send, and send it itself. This branch leans into that by replacing ~600 lines of LLM-powered elicitation code with a simple regex-based onDidDetectInputNeeded event that hands control back to the agent, cutting unnecessary latency and token spend.

This also extends send_to_terminal and get_terminal_output with a terminalId parameter so the agent can interact with foreground terminals too, not just persistent background ones.

We still use a lightweight regex check to quickly determine whether user input is likely needed. If so, we immediately return control to the agent and indicate that input may be required. We also surface this signal when a terminal command times out, prompting the agent to assess whether input was expected. The agent can then use the ask_questions_tool to confirm (unless running in autopilot mode), followed by the send_to_terminal tool to execute.

if (isAutoApproved) {
resultText.push(`Note: The command is running in terminal ID ${termId} and may be waiting for input. Evaluate the terminal output to determine if the command is actually waiting for input (e.g. a password prompt, confirmation, or interactive question). A normal shell prompt does NOT count as waiting for input. If it IS waiting for input, determine the appropriate response from context and immediately call ${TerminalToolId.SendToTerminal} with id "${termId}" to provide it.\n\n`);
} else {
resultText.push(`Note: The command is running in terminal ID ${termId} and may be waiting for input. Evaluate the terminal output to determine if the command is actually waiting for input (e.g. a password prompt, confirmation, or interactive question). A normal shell prompt does NOT count as waiting for input. If it IS waiting for input, call the askQuestions tool to ask the user what input to provide, then call ${TerminalToolId.SendToTerminal} with id "${termId}" to send their response.\n\n`);
}

Before

image

After

When in Default Approvals

Screenshot 2026-04-08 at 4 59 42 PM Screenshot 2026-04-08 at 4 59 52 PM Screenshot 2026-04-08 at 5 01 13 PM Screenshot 2026-04-08 at 5 01 30 PM

When in Autopilot

after.mov

Before vs After:

All Scenarios

Output Monitor — Input Detection

Mode Scenario Before After
Foreground Input prompt detected (Password:, [Y/n]) No detection — would eventually timeout New inputNeeded race candidate fires, converts to background, returns output + terminalId to agent
Foreground "Press any key" detected Showed elicitation UI with free-form input dialog, waited for user to type Fires onDidDetectInputNeeded, stops polling — agent handles via send_to_terminal
Async Input prompt detected Made LLM call to classify, then showed elicitation UI ("Terminal is awaiting input" + system-initiated chat request) Regex-only detection, fires onDidDetectInputNeeded, stops polling — agent handles it
Async "Press any key" detected Showed elicitation UI Fires onDidDetectInputNeeded, stops polling — agent handles it
Foreground Command completes normally Race resolves completed Same
Foreground User clicks "Continue in Background" Race resolves background Same
Foreground Timeout Race resolves timeout, extended polling Same
Async Command completes normally Monitor detects idle Same
Any Non-interactive help (press h for help) Detected, stopped Same
Any VS Code task finish message Detected, stopped Same

run_in_terminal — Foreground Race

Aspect Before After
Race candidates 3: completed, background, timeout 4: completed, background, timeout, inputNeeded
inputNeeded handling N/A Converts to background, returns output with terminalId so agent can use send_to_terminal
toolMetadata response Returns id (UUID) Returns id + terminalId (numeric instanceId) in all code paths
Listener cleanup N/A Uses DisposableStore for onDidDetectInputNeeded subscription, disposed after race

send_to_terminal — New Foreground Support

Aspect Before After
Parameters id (UUID, required) id (UUID) or terminalId (number) — one required
Foreground terminals Not supported Sends via instance.sendText(), returns guidance to use get_terminal_output with terminalId
Confirmation message "Run {cmd} in background terminal {uuid}" "Run {cmd} in terminal {title}" with "Focus Terminal" command link
Auto-approve analysis Only for persistent terminals Also resolves CWD and shell for foreground terminals

get_terminal_output — New Foreground Support

Aspect Before After
Parameters id (UUID, required) id (UUID) or terminalId (number) — one required
Foreground terminals Not supported Reads xterm buffer via getOutput(instance) (truncated to 16K chars, tail-biased)
Constructor No injected services Injects ITerminalService

outputMonitor.ts — Code Removed (~600 lines)

Removed Description
_requestFreeFormTerminalInput() Elicitation UI for free-form terminal input
_createElicitationPart() Generic elicitation part builder with system-initiated request support
_showInstance() Terminal focus helper for elicitation "Focus terminal" button
_isAutopilotMode() Checked widget/request permission level to skip elicitation
_lastPromptMarker, _promptPart Tracking fields for elicitation state
IChatService, IChatWidgetService, IConfigurationService, ITerminalService Constructor dependencies only needed for elicitation
All localize() strings for elicitation "Terminal is awaiting input", "Focus terminal", etc.
LLM-related imports ChatElicitationRequestPart, ChatModel, ElicitationState, ChatRequestTextPart, OffsetRange, ChatPermissionLevel, etc.

Copilot AI review requested due to automatic review settings April 8, 2026 18:43
@meganrogge meganrogge marked this pull request as draft April 8, 2026 18:43
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 8, 2026

Screenshot Changes

Base: 27a9b8ee Current: cf4e8e08

Changed (1)

editor/inlineChatAffordance/InlineChatOverlay/Light
Before After
before after

Removed (2)

chat/aiCustomizations/aiCustomizationManagementEditor/ClaudeHarness/Dark

baseline

chat/aiCustomizations/aiCustomizationManagementEditor/ClaudeHarness/Light

baseline

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR streamlines terminal prompt detection by removing LLM-based prompt analysis from the output monitor, and extends the terminal chat tools to support targeting foreground terminals via terminalId in addition to background/persistent terminal UUIDs.

Changes:

  • Remove LLM-driven “prompt/options” detection in OutputMonitor, replacing it with regex-based input-needed signaling.
  • Add foreground terminal support to send_to_terminal and get_terminal_output via terminalId and surface terminalId in run_in_terminal metadata.
  • Update/replace unit tests to reflect the new input-needed event behavior and new parameter validation.
Show a summary per file
File Description
src/vs/workbench/contrib/terminalContrib/chatAgentTools/test/browser/outputMonitor.test.ts Updates tests to validate the new onDidDetectInputNeeded event behavior and removal of LLM prompt selection flows.
src/vs/workbench/contrib/terminalContrib/chatAgentTools/test/browser/getTerminalOutputTool.test.ts Adapts tests to DI-based construction and adds coverage for missing-parameter error handling.
src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/sendToTerminalTool.ts Adds terminalId support, a focus-terminal command link in confirmations, and updated tool schema/description.
src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/runInTerminalTool.ts Races foreground execution against an input-needed signal and emits terminalId in tool metadata.
src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/monitoring/outputMonitor.ts Removes LLM prompt detection and introduces onDidDetectInputNeeded signaling based on regex detection.
src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/getTerminalOutputTool.ts Adds terminalId support, DI for ITerminalService, and updated tool schema/description.

Copilot's findings

Comments suppressed due to low confidence (4)

src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/sendToTerminalTool.ts:57

  • The input schema no longer requires id, but it also doesn’t express that either id or terminalId must be provided (and not both). As-is, schema validation/model guidance can produce { command } or { id, terminalId, command }, which contradicts the parameter docs and the implementation’s intent. Consider expressing this in the JSON schema via oneOf branches (one requiring id, the other requiring terminalId) and a not branch (or oneOf-only) to disallow both at once.
	inputSchema: {
		type: 'object',
		properties: {
			id: {
				type: 'string',
				description: `The ID of a persistent terminal session to send a command to (returned by ${TerminalToolId.RunInTerminal} in async mode). Provide either 'id' or 'terminalId', not both.`,
				pattern: '^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-5][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}$'
			},
			terminalId: {
				type: 'number',
				description: 'The numeric instanceId of a terminal. Use this to send input to terminals not started by the agent (e.g., user-created terminals or terminals that need interactive input). Provide either \'id\' or \'terminalId\', not both.'
			},
			command: {
				type: 'string',
				description: 'The command to send to the terminal. The text will be sent followed by Enter to execute it.'
			},
		},
		required: [
			'command',
		]
	}

src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/sendToTerminalTool.ts:140

  • SendToTerminalTool accepts both id and terminalId, but doesn’t reject the case where both are provided. That’s problematic because prepareToolInvocation’s label resolution prefers id, while invoke will take the terminalId path first, so the confirmation UI can describe/focus the wrong terminal. Add an explicit validation/error when both are set (or define a clear precedence and apply it consistently in both prepare+invoke).
	async prepareToolInvocation(context: IToolInvocationPreparationContext, _token: CancellationToken): Promise<IPreparedToolInvocation | undefined> {
		const args = context.parameters as ISendToTerminalInputParams;
		const displayCommand = buildCommandDisplayText(args.command);
		const safeInlineCode = toMarkdownInlineCode(displayCommand);

		// Resolve a human-friendly terminal label from the instance title
		const terminalLabel = this._getTerminalLabel(args);

		const invocationMessage = new MarkdownString();
		invocationMessage.appendMarkdown(localize('send.progressive', "Sending {0} to terminal", safeInlineCode));

		const pastTenseMessage = new MarkdownString();
		pastTenseMessage.appendMarkdown(localize('send.past', "Sent {0} to terminal", safeInlineCode));

		// Build the confirmation message with a "Focus Terminal" command link
		const instanceId = this._getTerminalInstanceId(args);
		const confirmationMessage = new MarkdownString('', { isTrusted: { enabledCommands: [FocusTerminalByIdCommandId] } });
		const baseMessage = localize('send.confirm.message', "Run {0} in terminal {1}", displayCommand, toMarkdownInlineCode(terminalLabel));
		if (instanceId !== undefined) {
			const focusUri = createCommandUri(FocusTerminalByIdCommandId, instanceId);
			confirmationMessage.appendMarkdown(`${baseMessage} — [$(terminal) ${localize('focusTerminal', "Focus Terminal")}](${focusUri})`);
		} else {
			confirmationMessage.appendMarkdown(baseMessage);
		}

src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/getTerminalOutputTool.ts:75

  • GetTerminalOutputTool validates the “neither provided” case, but it doesn’t reject when both id and terminalId are provided, even though the schema/docs say “Provide either … not both.” This makes the API ambiguous and can hide mistakes (it will silently prefer terminalId). Add a check to return an explicit error when both are set, and consider encoding this mutual exclusivity in the JSON schema (e.g. oneOf).
	async invoke(invocation: IToolInvocation, _countTokens: CountTokensCallback, _progress: ToolProgress, token: CancellationToken): Promise<IToolResult> {
		const args = invocation.parameters as IGetTerminalOutputInputParams;

		if (!args.id && args.terminalId === undefined) {
			return {
				content: [{
					kind: 'text',
					value: 'Error: Either \'id\' (persistent terminal UUID) or \'terminalId\' (foreground terminal instanceId) must be provided.'
				}]
			};
		}

		// Foreground terminal path
		if (args.terminalId !== undefined) {
			const instance = this._terminalService.getInstanceFromId(args.terminalId);
			if (!instance) {

src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/sendToTerminalTool.ts:35

  • The modelDescription mentions using get_terminal_output “for persistent terminals”, but this tool now also supports sending to foreground terminals via terminalId, and get_terminal_output also supports terminalId. Consider updating the description to mention checking output for both target types to avoid steering the model away from the new foreground flow.
	id: TerminalToolId.SendToTerminal,
	toolReferenceName: 'sendToTerminal',
	displayName: localize('sendToTerminalTool.displayName', 'Send to Terminal'),
	modelDescription: `Send a command to a terminal session. This can target either a persistent terminal started with ${TerminalToolId.RunInTerminal} in async mode (using 'id') or any foreground terminal visible in the terminal panel (using 'terminalId'). After sending, use ${TerminalToolId.GetTerminalOutput} to check updated output for persistent terminals.`,
	icon: Codicon.terminal,
  • Files reviewed: 6/6 changed files
  • Comments generated: 3

@meganrogge meganrogge self-assigned this Apr 8, 2026
@meganrogge meganrogge added this to the 1.116.0 milestone Apr 8, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the chat terminal toolchain to avoid LLM-driven “terminal awaiting input” classification/elicitation, and instead surfaces a deterministic onDidDetectInputNeeded signal so the agent can respond autonomously via send_to_terminal/get_terminal_output. It also extends send_to_terminal and get_terminal_output to support foreground terminals via terminalId.

Changes:

  • Remove LLM + elicitation UI paths from OutputMonitor and replace with an onDidDetectInputNeeded event based on regex detection.
  • Extend send_to_terminal/get_terminal_output to accept either persistent id (UUID) or foreground terminalId (instanceId).
  • Update run_in_terminal foreground execution to race inputNeeded and return early with terminalId metadata.
Show a summary per file
File Description
src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/sendToTerminalTool.ts Adds terminalId targeting, “Focus Terminal” link, and foreground send path
src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/getTerminalOutputTool.ts Adds terminalId targeting and foreground output retrieval via xterm buffer
src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/runInTerminalTool.ts Adds inputNeeded as a foreground race candidate and returns terminalId in metadata
src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/monitoring/outputMonitor.ts Removes LLM-based prompt detection/elicitation; emits onDidDetectInputNeeded
src/vs/workbench/contrib/terminalContrib/chatAgentTools/test/browser/sendToTerminalTool.test.ts Updates tool description assertion for the new wording
src/vs/workbench/contrib/terminalContrib/chatAgentTools/test/browser/getTerminalOutputTool.test.ts Updates instantiation to inject ITerminalService; adds parameter validation test
src/vs/workbench/contrib/terminalContrib/chatAgentTools/test/browser/outputMonitor.test.ts Replaces auto-reply/LLM tests with input-needed event tests

Copilot's findings

  • Files reviewed: 7/7 changed files
  • Comments generated: 5

@meganrogge meganrogge requested a review from roblourens April 8, 2026 20:28
@meganrogge meganrogge changed the title remove extra LLM calls to detect prompts for input, add fg terminal support for send_to_terminal/get_terminal_output remove LLM calls to detect prompts for input, add fg terminal support for send_to_terminal/get_terminal_output Apr 8, 2026
@meganrogge meganrogge marked this pull request as ready for review April 8, 2026 21:19
@meganrogge meganrogge enabled auto-merge (squash) April 8, 2026 21:19
@meganrogge
Copy link
Copy Markdown
Collaborator Author

Screenshot 2026-04-08 at 5 23 41 PM

@meganrogge meganrogge requested a review from karthiknadig April 8, 2026 21:41
@meganrogge
Copy link
Copy Markdown
Collaborator Author

cc @anthonykim1

@meganrogge
Copy link
Copy Markdown
Collaborator Author

meganrogge commented Apr 8, 2026

after.mov

@meganrogge meganrogge merged commit edd5c96 into main Apr 9, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants