Skip to content

feat(core): replace webfetch markdown renderer - #42229

Open
kitlangton wants to merge 1 commit into
devfrom
webfetch-markdown
Open

feat(core): replace webfetch markdown renderer#42229
kitlangton wants to merge 1 commit into
devfrom
webfetch-markdown

Conversation

@kitlangton

Copy link
Copy Markdown
Contributor

What

Replace V2 Core webfetch's Turndown/Domino HTML-to-Markdown conversion with a purpose-built htmlparser2 event renderer. The renderer preserves model-readable document structure without constructing a browser DOM and removes Core's turndown, @mixmark-io/domino, and @types/turndown closure.

Measurement Before After Change
322 KB article throughput, median 4.23 MiB/s 10.12 MiB/s 2.39x
Conversion time, 1 warmup + 9 measured 72.55 ms (66.93-109.75) 30.32 ms (22.29-83.55) -58.2%
Isolated minified bundle 311,557 B 61,293 B -250,264 B
Isolated gzip bundle 98,680 B 26,922 B -71,758 B
macOS arm64 CLI executable 87,338,978 B 87,091,298 B -247,680 B
Installed Turndown + Domino closure 9,028 KiB 0 KiB in Core -9,028 KiB

How

  • packages/core/src/tool/html-markdown.ts implements an iterative semantic frame renderer for headings, paragraphs, inline formatting, links/images, blockquotes, nested lists, inline/fenced code, and readable tables.
  • Active/fallback elements are suppressed, entities decode through htmlparser2, code fences expand around embedded backticks, and preformatted content bypasses whitespace normalization.
  • List/quote indentation is capped and pathological open-tag depth takes a deterministic text-only path, bounding amplification for webfetch's 5 MiB response limit.
  • packages/core/test/tool-webfetch.test.ts covers the public helper and tool path with malformed/deep input, Unicode/entities, active content, whitespace, code, nested lists/tables, and output-growth regressions.
  • packages/core/perf/html-markdown.md records the autoresearch experiments and measurements.

Scope

  • V2 only: Core and the V2 CLI artifact were changed/evaluated.
  • packages/opencode is untouched.
  • No public package or API was added; WebFetchTool.convertHTMLToMarkdown remains the public helper.
  • Base Turndown flattened table cells; the replacement intentionally emits readable GFM-style tables.
  • Malformed overlapping inline/block tags may drop formatting delimiters while preserving visible text.
  • Documents deeper than 10,000 open tags fall back to deterministic visible-text extraction to protect memory.
  • This repository does not use Changesets, so no changeset is included.

Testing

  • bun run test tool-webfetch.test.ts from packages/core: 26 passed.
  • bun typecheck from packages/core: passed.
  • bun typecheck from packages/cli: passed.
  • Push hook bun turbo typecheck: 29/29 package tasks passed.
  • MODELS_DEV_API_JSON=../core/test/plugin/fixtures/models-dev.json bun run build --single --skip-install from packages/cli: passed.
  • Temporary real-site differential corpus: Example Domain, MDN table reference, Python asyncio docs, RFC 9110, and W3C forms tutorial. Snapshots were not committed.
  • Full Core suite: 1,032/1,033 passed. The unrelated macOS watcher test reports .git/HEAD as add instead of expected change; it reproduces in isolation.
  • Four independent review rounds found and verified fixes for depth/output amplification, fence crashes, code whitespace mutation, Markdown escaping, quote/table containment, emphasis whitespace, and nested tables.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant