feat(app, ui): render local images in markdown via /file/content API#30722
Open
Howardzhangdqs wants to merge 4 commits into
Open
feat(app, ui): render local images in markdown via /file/content API#30722Howardzhangdqs wants to merge 4 commits into
Howardzhangdqs wants to merge 4 commits into
Conversation
Contributor
|
Hey! Your PR title Please update it to start with one of:
Where See CONTRIBUTING.md for details. |
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds support for rendering “local” Markdown images by tagging non-URL image paths during Markdown parsing and resolving them to data URIs at runtime via a context-provided resolver in the app.
Changes:
- Extend the Markdown renderer to emit
data-local-imagefor non-URL image paths. - Introduce a
LocalImageResolvercontext and wire it into the app viaServerSDKto fetch file content as base64 data URIs. - Thread
directorythrough message rendering into the Markdown component so resolution can be directory-aware.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/ui/src/context/marked.tsx | Adds custom Marked image() renderer that marks local image paths for later resolution. |
| packages/ui/src/context/local-image.tsx | Introduces a Solid context to provide a local-image resolver function. |
| packages/ui/src/context/index.ts | Re-exports the new local-image context module. |
| packages/ui/src/components/message-part.tsx | Threads directory into Markdown rendering (paced + non-paced). |
| packages/ui/src/components/markdown.tsx | Resolves data-local-image images post-render via the resolver context and directory prop. |
| packages/app/src/app.tsx | Provides a resolver implementation backed by serverSDK.file.read() and wraps the router with the provider. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+5
to
+8
| const ctx = createContext<LocalImageResolver>() | ||
|
|
||
| export const LocalImageProvider = ctx.Provider | ||
| export const useLocalImageResolver = () => useContext(ctx) |
Comment on lines
+478
to
+484
| image({ href, title, text }) { | ||
| const titleAttr = title ? ` title="${title}"` : "" | ||
| if (href && !/^(https?:|data:|#|\/\/)/.test(href)) { | ||
| return `<img data-local-image="${href}" alt="${text}"${titleAttr}>` | ||
| } | ||
| return `<img src="${href ?? ""}" alt="${text}"${titleAttr}>` | ||
| }, |
| if (href && !/^(https?:|data:|#|\/\/)/.test(href)) { | ||
| return `<img data-local-image="${href}" alt="${text}"${titleAttr}>` | ||
| } | ||
| return `<img src="${href ?? ""}" alt="${text}"${titleAttr}>` |
Comment on lines
+336
to
+347
| const resolver = useLocalImageResolver() | ||
| if (resolver && local.directory) { | ||
| for (const img of container.querySelectorAll<HTMLImageElement>("[data-local-image]")) { | ||
| const path = img.getAttribute("data-local-image") | ||
| if (!path || img.src) continue | ||
| resolver(path, local.directory) | ||
| .then((dataUri) => { | ||
| if (dataUri) img.src = dataUri | ||
| }) | ||
| .catch(() => {}) | ||
| } | ||
| } |
Comment on lines
+338
to
+341
| for (const img of container.querySelectorAll<HTMLImageElement>("[data-local-image]")) { | ||
| const path = img.getAttribute("data-local-image") | ||
| if (!path || img.src) continue | ||
| resolver(path, local.directory) |
Comment on lines
+72
to
+82
| const resolver: LocalImageResolver = async (rawPath, directory) => { | ||
| try { | ||
| if (!directory) return undefined | ||
| const client = serverSDK.createClient({ directory, throwOnError: true }) | ||
| const resp = await client.file.read({ path: rawPath }) | ||
| if (resp.data?.type === "binary" && resp.data.encoding === "base64" && resp.data.mimeType) { | ||
| return `data:${resp.data.mimeType};base64,${resp.data.content}` | ||
| } | ||
| } catch {} | ||
| return undefined | ||
| } |
| if (resp.data?.type === "binary" && resp.data.encoding === "base64" && resp.data.mimeType) { | ||
| return `data:${resp.data.mimeType};base64,${resp.data.content}` | ||
| } | ||
| } catch {} |
Contributor
|
Thanks for updating your PR! It now meets our contributing guidelines. 👍 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Closes #16234
Type of change
What does this PR do?
Enable rendering of local images (e.g.
) in agent markdown responses in the WebUI.When an LLM returns markdown with relative image paths, the browser cannot resolve them. This PR adds a marked image renderer that outputs
<img data-local-image="...">placeholders, then resolves them asynchronously by callingGET /file/content?path=...with the session directory and converting the base64 binary response todata:URIs.How did you verify your code works?
bun run --conditions=browser ./src/index.ts serve --port 4096) and frontend dev server (bun dev -- --port 4444)tsc --noEmitpasses for bothpackages/appandpackages/uirender the image in the browserScreenshots / recordings
N/A — feature renders local images inline in markdown.
Checklist