fix(llm): surface attached image filename in model context - #41798
fix(llm): surface attached image filename in model context#41798patil2001 wants to merge 1 commit into
Conversation
Provider lowerings (OpenAI Chat/Responses, Anthropic, Gemini) dropped the MediaPart.filename that survives attachment ingestion, so the model could see the image content but never the original file name. Emit the filename as a caption text part alongside the image so agents can reference the attached file. Fixes anomalyco#41454
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
|
The following comment was made by an LLM, it may be inaccurate: Potential related PR found:
However, this is not a duplicate. PR #41798 is focused on adding filename captions in the LLM protocols when passing images to models, while PR #41789 likely handles exposing attachment paths at the core level. These are complementary changes addressing different layers of the same feature. No duplicate PRs found. |
|
This pull request has been automatically closed because it was not updated to meet our contributing guidelines within the 2-hour window. Feel free to open a new pull request that follows our guidelines. |
Summary
When a user attaches an image in the TUI, the attachment chip shows the original filename (e.g.
IMG_3480.JPG), and the filename survives all the way intoMediaPart.filename— but every provider lowering drops it before building the request. The model receives only the image content, so it cannot identify or reference the attached file.This PR emits the filename as a caption text part alongside the image in the user message for the four image-capable protocols:
openai-chat)openai-responses)anthropic-messages)gemini)Bedrock Converse already preserves
filenamefor document blocks, so it is left unchanged.Changes
packages/llm/src/protocols/shared.ts: newmediaCaption(part)helper producing[Image: <filename>]when a filename is present.Test plan
bun testinpackages/llm: 302 pass, 0 fail.tsgo --noEmitinpackages/llm: clean.Fixes #41454