Skip to content

feat(tui): attach images to question answers - #56651

Open
hearnadam wants to merge 1 commit into
anomalyco:devfrom
hearnadam:question-images
Open

hearnadam wants to merge 1 commit into
anomalyco:devfrom
hearnadam:question-images

Conversation

@hearnadam

@hearnadam hearnadam commented Oct 18, 2026 •

Copy link
Copy Markdown

Issue for this PR

Supersedes #56650, which was automatically closed for targeting the retired v2 branch.

Refs #31791 (auto-closed as stale). Follows Kit's suggestion to keep attachments separate from scalar answers. This covers the full TUI, not Web or Mini.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Allow dropped image paths and clipboard images in terminal question answers. Form replies carry optional field-keyed attachments without changing Form.Answer; the question tool sends images through existing tool-image normalization with text identifying their question.

Users can remove images or submit an image-only answer. Adding images preserves selected answers, and pending clipboard reads survive session-tab navigation. Updates generated clients, OpenAPI documents, and usage docs.

How did you verify your code works?

  • 61 focused tests pass under the pinned Bun 1.4.2: Core form/question tool (20), TUI form/helpers (40), generated-client HTTP round trip (1).
  • All six affected packages typecheck.
  • Website typecheck, build, and generated-document checks passed during implementation.
  • Source TUI exercised in a 120×35 PTY; submitted image bytes matched the original file. The new selection/navigation regressions failed before the fixes and pass afterward.
  • Full pre-push bun run check passed in a clean checkout under Bun 1.4.2 (36 typecheck tasks; existing lint warnings only). The initial Vite errors came from stale local dependency copies, not the PR.

Screenshots / recordings

Text capture from the source TUI's PTY smoke test (no graphical screenshot attached):

Questions

Attach a screenshot to this terminal question.

1. Type your own answer

[Image 1] terminal-question-test.png · remove

enter submit  esc close  esc dismiss

Before this change, pasting the same PNG path into the question editor submitted a plain answer string; the normal prompt already recognized it as an image.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Added by the opencode bot (AI-generated).

Related: #31791

Public API: adds optional attachments to form string/multiselect fields, Form.Reply (HTTP form reply body), the answered Form.State and the form.replied event, plus new Form.Attachment/Form.Attachments schemas (backwards compatible)

@github-actions

Copy link
Copy Markdown
Contributor

The following comment was made by an LLM, it may be inaccurate:

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core and server parts look sound: attachments are checked against the field's attachments flag and whether the field is active, the scalar Form.Answer is unchanged, and the question tool reuses the existing tool file content. The new tests pass (core 20, server 1, TUI 36), and core, schema, tui and server typecheck. I couldn't trigger a real bracketed paste or clipboard image through the TUI driver, so I haven't seen the paste path work end to end. A short recording or screenshot of the [Image N] list and the submitted result would help.

Public API (all additive; existing callers keep working):

  • Form.StringField / Form.MultiselectField: new optional attachments: boolean
  • New schemas Form.Attachment { data: base64, mime: image/*, name? } and Form.Attachments (a record from field key to Attachment[])
  • Form.Reply and POST …/form/{formID}/reply: new optional attachments
  • Form.State answered variant: new optional attachments
  • form.replied event: new optional attachments. Subscribers now receive the full base64 image data.
  • Generated promise and effect clients and the OpenAPI documents are updated to match. The plugin question tool result content becomes an array (text plus files) when images are attached.

Concerns (details inline):

  • The server sets no limit on attachment size or count; only the TUI's local-file read is bounded.
  • The full image bytes are broadcast in form.replied and kept in the form state cache.
  • The q${index} key is duplicated outside toField.
  • Every paste into a question field, including plain text, now goes through the async attachment path and blocks submit and tab navigation until it finishes. Please confirm that typing and pasting text in question answers still feels instant.

About 60 lines of form.tsx are formatting-only JSX reflows unrelated to the feature. Please drop them to keep the diff focused.

Related: #31791

export const Answer = Schema.Record(Schema.String, Value).annotate({ identifier: "Form.Answer" })
export type Answer = typeof Answer.Type

export const Attachment = Schema.Struct({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing limits the size or number of attachments here or in Form.reply. The TUI caps local file reads at 20 MB, but any HTTP client can post arbitrarily large base64 payloads. Those payloads are then kept in the form state, broadcast on the bus and sent to the model. Could you add a server-side limit on size and count, e.g. reusing the prompt attachment limits?

Comment thread packages/core/src/form.ts
answer: input.answer,
...(input.attachments === undefined ? {} : { attachments: input.attachments }),
}
yield* bus.publish(Form.Event.Replied, {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now publishes the full base64 image data on form.replied to every event subscriber (all SSE clients, notifications, ACP translation, Mini). Existing consumers only need to know that the form was answered. Could the event carry only answer, plus perhaps an attachment count, with the bytes left to the deferred result the question tool already reads?

? [
{ type: "text", text: toModelContent(input.questions, output.answers, state.attachments) },
...input.questions.flatMap((question, index): Tool.Content[] => {
const images = state.attachments?.[`q${index}`]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

q${index} repeats the key format that toField defines, here and in toModelContent. If the key scheme changes, images will silently stop being associated with their questions. Please derive the key from one helper shared with toField.

if (!active()) return
const value = stripAnsiSequences(decodePasteBytes(event.bytes)).replace(/\r\n?/g, "\n")
const current = answerField()
if (!confirm() && current && allowsAttachments(current)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every question field has attachments: true, so every paste into a question answer, including plain text, now goes through enqueuePaste and resolvePastedAttachments. While that runs, store.pasting blocks reply, selectTab, selectOption and commitInput. A quick paste followed by Enter can therefore be dropped silently. Could text that clearly isn't a file path skip the queue, or could submit wait for the pending paste instead of returning early?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant