docs: translated documentation sites and the tooling behind them (slim) - #3265
Draft
maxisbey wants to merge 2 commits into
Draft
docs: translated documentation sites and the tooling behind them (slim)#3265maxisbey wants to merge 2 commits into
maxisbey wants to merge 2 commits into
Conversation
Ship preview translations of the docs in Simplified Chinese, Japanese, Korean and Brazilian Portuguese, served at /zh-CN/, /ja/, /ko/ and /pt-BR/ next to the English site (three pages each to start). English stays the single source of truth; everything else is generated and never hand-edited. scripts/docs/translations.py drives it (status / translate / stage / languages). `translate` re-translates only the `##` sections whose English changed and carries the rest of the page over byte-for-byte from the previous translation, then gates the result: heading anchors and code blocks are re-imposed from the English, the structure (sections, headings, fences, links, inline code, admonitions, glossary rules) is checked with the findings fed back for another attempt, and a stronger model reviews the meaning against the English. Per-language inputs — instructions.md (register, voice, typography) and glossary.json (termbase, keep-in-English list, banned renderings) — are the human-edited part; the pages and state.json under i18n/<code>/ are generated, so a correction goes into the inputs and holds across every future run. The build stages each language over the English tree (translations overlaid, notices stamped in — machine-translated, may be behind English, not translated — and links into the API reference pointed at the English one) and builds it strictly into site/<code>/. Heading ids are pinned into the translations from the ids the English site renders, so #fragment links resolve on every language site without touching the English pages. An outdated translation is served with a warning unless the English page's links or anchors moved under it, so an English edit can never break a language build. Every config carries the same language switcher.
Contributor
📚 Documentation preview
|
The test fixture wrote its throwaway repo tree with the platform default encoding, which cannot represent the Japanese fixtures on Windows; write it (and the tool's own generated pages and state) explicitly as UTF-8 with LF newlines so the output is identical everywhere.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A leaner redo of #3231: machine-translated preview sites for Simplified
Chinese, Japanese, Korean and Brazilian Portuguese (a few pages each to start),
served at
/zh-CN/,/ja/,/ko/and/pt-BR/next to the English site.English stays the single source of truth; everything else is generated and
never hand-edited. Compared to #3231 this is roughly half the diff and about
a fifth of the code, with the English pages untouched.
scripts/docs/translations.py(status,translate,stage,languages).translatere-translates only the##sectionswhose English changed and carries the rest of the page over byte-for-byte
from the previous translation, then gates the result: heading anchors and
code blocks are re-imposed from the English rather than validated, the
structure (sections, headings, fences, links, inline code, admonitions,
glossary keep/ban rules, abridgement markers, untranslated headings) is
checked with the findings fed back for up to two more attempts, and a
stronger model reviews the meaning against the English (one correction
round; a surviving blocker fails the page).
instructions.md(register, voice,typography) and
glossary.json(termbase, keep-in-English list, bannedrenderings) are the human-authored part; the pages and
state.jsonaregenerated, so a correction goes into the inputs and holds across every
future run. Sidebar labels and page notices are translated the same way (one
JSON call per language) and follow the same correction loop.
docs/— instead of pinning{#anchor}on every Englishheading, each heading's id is computed exactly as the site renders it and
pinned into the translation, so
#fragmentlinks resolve on every languagesite. (Verified against a real build: 511/511 heading ids across 50 pages.)
scripts/docs/build.shbuilds English intosite/, thenfor each language stages the tree (English pages overlaid with that
language's translations, notices stamped in — machine-translated / may be
behind the English / shown in English — and links into the API reference
pointed at the single English one) and builds it strictly into
site/<code>/. Every config carries the sameextra.alternateswitcher; alanguage site drops mkdocstrings. Four languages add ~10s to the build.
section hashes and the fingerprint of the prompt inputs it was made from;
statusreports missing / outdated / current. An outdated translation isserved with a warning notice unless the English page's links or heading
anchors moved under it — then the English page is served until the next
translaterun refreshes it, so an English edit can never break a languagebuild.
docs/translations.mdexplains all this to readers, with an issueform for reports.
Motivation and Context
Same as #3231 (make the docs useful to more people, with translations that stay
correctable at the source), redone from scratch to minimise the code the repo
carries: no anchor churn across the English pages, no offline
checker/validator commands duplicating what the build already enforces, and
generated state kept to what the pipeline actually reads.
How Has This Been Tested?
pages plus the four languages' UI strings, and exercised the staleness
contract with real edits (prose edit → outdated notice; heading reword →
translation withdrawn; a one-section English edit re-translates only that
section and carries the rest over byte-for-byte).
--strict) and browsed it: switcher on every page,hreflangalternates,translated sidebars and notices, English heading anchors resolving on the
language sites, API Reference nav entry linking the English reference.
English and docs: translated documentation sites and the tooling behind them #3231's translations of the same pages: no meaning blockers;
the tool's gates were extended (untranslated-heading and inline-code checks)
for the two structural slips found, and those pages regenerated.
tests/test_docs_i18n.pycovers the pure logic offline (mechanical repair,section carry-forward, the structural gate, the reviewer round-trip with a
scripted model, staging/withdrawal, status, UI strings).
Breaking Changes
None to the SDK. Docs builds now also produce the four language sites.
Types of changes
Checklist
Additional context
i18n/general-prompt.md→i18n/ja/instructions.md→docs/translations.md→i18n/README.md→ thesample pages under
i18n/*/pages/→scripts/docs/translations.py(top tobottom) →
tests/test_docs_i18n.py.translationlabel the issue form applies.translateneedsANTHROPIC_API_KEY(or an authtoken); nothing in CI calls the API.
translate --lang <code> --freshredoes a whole language from scratch (e.g. after changing the model).
served in English with a notice. Translating the full corpus is one command
per language whenever we want it.
hreflangalternates pointat each site root rather than the equivalent page, and CJK search has no
word segmentation.
AI Disclaimer