remove LLM calls to detect prompts for input, add fg terminal support for send_to_terminal/get_terminal_output#308587
Conversation
There was a problem hiding this comment.
Pull request overview
This PR streamlines terminal prompt detection by removing LLM-based prompt analysis from the output monitor, and extends the terminal chat tools to support targeting foreground terminals via terminalId in addition to background/persistent terminal UUIDs.
Changes:
- Remove LLM-driven “prompt/options” detection in
OutputMonitor, replacing it with regex-based input-needed signaling. - Add foreground terminal support to
send_to_terminalandget_terminal_outputviaterminalIdand surfaceterminalIdinrun_in_terminalmetadata. - Update/replace unit tests to reflect the new input-needed event behavior and new parameter validation.
Show a summary per file
| File | Description |
|---|---|
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/test/browser/outputMonitor.test.ts | Updates tests to validate the new onDidDetectInputNeeded event behavior and removal of LLM prompt selection flows. |
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/test/browser/getTerminalOutputTool.test.ts | Adapts tests to DI-based construction and adds coverage for missing-parameter error handling. |
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/sendToTerminalTool.ts | Adds terminalId support, a focus-terminal command link in confirmations, and updated tool schema/description. |
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/runInTerminalTool.ts | Races foreground execution against an input-needed signal and emits terminalId in tool metadata. |
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/monitoring/outputMonitor.ts | Removes LLM prompt detection and introduces onDidDetectInputNeeded signaling based on regex detection. |
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/getTerminalOutputTool.ts | Adds terminalId support, DI for ITerminalService, and updated tool schema/description. |
Copilot's findings
Comments suppressed due to low confidence (4)
src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/sendToTerminalTool.ts:57
- The input schema no longer requires
id, but it also doesn’t express that eitheridorterminalIdmust be provided (and not both). As-is, schema validation/model guidance can produce{ command }or{ id, terminalId, command }, which contradicts the parameter docs and the implementation’s intent. Consider expressing this in the JSON schema viaoneOfbranches (one requiringid, the other requiringterminalId) and anotbranch (oroneOf-only) to disallow both at once.
inputSchema: {
type: 'object',
properties: {
id: {
type: 'string',
description: `The ID of a persistent terminal session to send a command to (returned by ${TerminalToolId.RunInTerminal} in async mode). Provide either 'id' or 'terminalId', not both.`,
pattern: '^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-5][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}$'
},
terminalId: {
type: 'number',
description: 'The numeric instanceId of a terminal. Use this to send input to terminals not started by the agent (e.g., user-created terminals or terminals that need interactive input). Provide either \'id\' or \'terminalId\', not both.'
},
command: {
type: 'string',
description: 'The command to send to the terminal. The text will be sent followed by Enter to execute it.'
},
},
required: [
'command',
]
}
src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/sendToTerminalTool.ts:140
SendToTerminalToolaccepts bothidandterminalId, but doesn’t reject the case where both are provided. That’s problematic becauseprepareToolInvocation’s label resolution prefersid, whileinvokewill take theterminalIdpath first, so the confirmation UI can describe/focus the wrong terminal. Add an explicit validation/error when both are set (or define a clear precedence and apply it consistently in both prepare+invoke).
async prepareToolInvocation(context: IToolInvocationPreparationContext, _token: CancellationToken): Promise<IPreparedToolInvocation | undefined> {
const args = context.parameters as ISendToTerminalInputParams;
const displayCommand = buildCommandDisplayText(args.command);
const safeInlineCode = toMarkdownInlineCode(displayCommand);
// Resolve a human-friendly terminal label from the instance title
const terminalLabel = this._getTerminalLabel(args);
const invocationMessage = new MarkdownString();
invocationMessage.appendMarkdown(localize('send.progressive', "Sending {0} to terminal", safeInlineCode));
const pastTenseMessage = new MarkdownString();
pastTenseMessage.appendMarkdown(localize('send.past', "Sent {0} to terminal", safeInlineCode));
// Build the confirmation message with a "Focus Terminal" command link
const instanceId = this._getTerminalInstanceId(args);
const confirmationMessage = new MarkdownString('', { isTrusted: { enabledCommands: [FocusTerminalByIdCommandId] } });
const baseMessage = localize('send.confirm.message', "Run {0} in terminal {1}", displayCommand, toMarkdownInlineCode(terminalLabel));
if (instanceId !== undefined) {
const focusUri = createCommandUri(FocusTerminalByIdCommandId, instanceId);
confirmationMessage.appendMarkdown(`${baseMessage} — [$(terminal) ${localize('focusTerminal', "Focus Terminal")}](${focusUri})`);
} else {
confirmationMessage.appendMarkdown(baseMessage);
}
src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/getTerminalOutputTool.ts:75
GetTerminalOutputToolvalidates the “neither provided” case, but it doesn’t reject when bothidandterminalIdare provided, even though the schema/docs say “Provide either … not both.” This makes the API ambiguous and can hide mistakes (it will silently preferterminalId). Add a check to return an explicit error when both are set, and consider encoding this mutual exclusivity in the JSON schema (e.g.oneOf).
async invoke(invocation: IToolInvocation, _countTokens: CountTokensCallback, _progress: ToolProgress, token: CancellationToken): Promise<IToolResult> {
const args = invocation.parameters as IGetTerminalOutputInputParams;
if (!args.id && args.terminalId === undefined) {
return {
content: [{
kind: 'text',
value: 'Error: Either \'id\' (persistent terminal UUID) or \'terminalId\' (foreground terminal instanceId) must be provided.'
}]
};
}
// Foreground terminal path
if (args.terminalId !== undefined) {
const instance = this._terminalService.getInstanceFromId(args.terminalId);
if (!instance) {
src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/sendToTerminalTool.ts:35
- The
modelDescriptionmentions using get_terminal_output “for persistent terminals”, but this tool now also supports sending to foreground terminals viaterminalId, and get_terminal_output also supportsterminalId. Consider updating the description to mention checking output for both target types to avoid steering the model away from the new foreground flow.
id: TerminalToolId.SendToTerminal,
toolReferenceName: 'sendToTerminal',
displayName: localize('sendToTerminalTool.displayName', 'Send to Terminal'),
modelDescription: `Send a command to a terminal session. This can target either a persistent terminal started with ${TerminalToolId.RunInTerminal} in async mode (using 'id') or any foreground terminal visible in the terminal panel (using 'terminalId'). After sending, use ${TerminalToolId.GetTerminalOutput} to check updated output for persistent terminals.`,
icon: Codicon.terminal,
- Files reviewed: 6/6 changed files
- Comments generated: 3
There was a problem hiding this comment.
Pull request overview
This PR updates the chat terminal toolchain to avoid LLM-driven “terminal awaiting input” classification/elicitation, and instead surfaces a deterministic onDidDetectInputNeeded signal so the agent can respond autonomously via send_to_terminal/get_terminal_output. It also extends send_to_terminal and get_terminal_output to support foreground terminals via terminalId.
Changes:
- Remove LLM + elicitation UI paths from
OutputMonitorand replace with anonDidDetectInputNeededevent based on regex detection. - Extend
send_to_terminal/get_terminal_outputto accept either persistentid(UUID) or foregroundterminalId(instanceId). - Update
run_in_terminalforeground execution to raceinputNeededand return early withterminalIdmetadata.
Show a summary per file
| File | Description |
|---|---|
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/sendToTerminalTool.ts | Adds terminalId targeting, “Focus Terminal” link, and foreground send path |
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/getTerminalOutputTool.ts | Adds terminalId targeting and foreground output retrieval via xterm buffer |
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/runInTerminalTool.ts | Adds inputNeeded as a foreground race candidate and returns terminalId in metadata |
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/monitoring/outputMonitor.ts | Removes LLM-based prompt detection/elicitation; emits onDidDetectInputNeeded |
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/test/browser/sendToTerminalTool.test.ts | Updates tool description assertion for the new wording |
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/test/browser/getTerminalOutputTool.test.ts | Updates instantiation to inject ITerminalService; adds parameter validation test |
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/test/browser/outputMonitor.test.ts | Replaces auto-reply/LLM tests with input-needed event tests |
Copilot's findings
- Files reviewed: 7/7 changed files
- Comments generated: 5
…move stale jsdoc, fix test
send_to_terminal/get_terminal_outputsend_to_terminal/get_terminal_output
|
cc @anthonykim1 |
after.mov |

fixes #308296
fixes #308030
fixes #288570
Now that the agent has
send_to_terminalandget_terminal_outputat its disposal, there's no need for the terminal tool to make extra LLM calls or show elicitation UI when a command prompts for input — the agent can detect the prompt, decide what to send, and send it itself. This branch leans into that by replacing ~600 lines of LLM-powered elicitation code with a simple regex-basedonDidDetectInputNeededevent that hands control back to the agent, cutting unnecessary latency and token spend.This also extends
send_to_terminalandget_terminal_outputwith aterminalIdparameter so the agent can interact with foreground terminals too, not just persistent background ones.We still use a lightweight regex check to quickly determine whether user input is likely needed. If so, we immediately return control to the agent and indicate that input may be required. We also surface this signal when a terminal command times out, prompting the agent to assess whether input was expected. The agent can then use the
ask_questions_toolto confirm (unless running in autopilot mode), followed by thesend_to_terminaltool to execute.vscode/src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/runInTerminalTool.ts
Lines 1458 to 1462 in 35a23cd
Before
After
When in
Default ApprovalsWhen in
Autopilotafter.mov
Before vs After:
All Scenarios
Output Monitor — Input Detection
Password:,[Y/n])inputNeededrace candidate fires, converts to background, returns output +terminalIdto agentonDidDetectInputNeeded, stops polling — agent handles viasend_to_terminalonDidDetectInputNeeded, stops polling — agent handles itonDidDetectInputNeeded, stops polling — agent handles itcompletedbackgroundtimeout, extended pollingpress h for help)run_in_terminal— Foreground Racecompleted,background,timeoutcompleted,background,timeout,inputNeededinputNeededhandlingterminalIdso agent can usesend_to_terminaltoolMetadataresponseid(UUID)id+terminalId(numeric instanceId) in all code pathsDisposableStoreforonDidDetectInputNeededsubscription, disposed after racesend_to_terminal— New Foreground Supportid(UUID, required)id(UUID) orterminalId(number) — one requiredinstance.sendText(), returns guidance to useget_terminal_outputwithterminalId"Run {cmd} in background terminal {uuid}""Run {cmd} in terminal {title}"with "Focus Terminal" command linkget_terminal_output— New Foreground Supportid(UUID, required)id(UUID) orterminalId(number) — one requiredgetOutput(instance)(truncated to 16K chars, tail-biased)ITerminalServiceoutputMonitor.ts — Code Removed (~600 lines)
_requestFreeFormTerminalInput()_createElicitationPart()_showInstance()_isAutopilotMode()_lastPromptMarker,_promptPartIChatService,IChatWidgetService,IConfigurationService,ITerminalServicelocalize()strings for elicitationChatElicitationRequestPart,ChatModel,ElicitationState,ChatRequestTextPart,OffsetRange,ChatPermissionLevel, etc.