Pin text I/O to UTF-8 and fail CI on locale-dependent reads/writes - #3296
Draft
maxisbey wants to merge 2 commits into
Draft
Pin text I/O to UTF-8 and fail CI on locale-dependent reads/writes#3296maxisbey wants to merge 2 commits into
maxisbey wants to merge 2 commits into
Conversation
Path.read_text() without an encoding argument uses the locale preferred encoding, which on Windows is the ANSI code page (e.g. GBK on Chinese-locale systems). manifest.toml and the story sources are UTF-8, so tests/examples failed to collect or run there. Fixes #3244 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Text-mode open()/Path.read_text()/write_text() without encoding= use the process locale, which is the ANSI code page on Windows. CI can't see the difference (ubuntu is UTF-8, windows-latest is cp1252 and happens to decode our em dashes), but a cp936/cp932/cp949 machine raises, and `mcp install` would read claude_desktop_config.json with the wrong codec and write any pre-existing non-ASCII entries back as mojibake. - Pass encoding="utf-8" at every remaining call site (cli, stories harness, scripts, tests, examples); subprocess captures decode as UTF-8 with errors="replace"; drop a non-ASCII glyph from a script's piped output. - Run pytest with PYTHONWARNDEFAULTENCODING=1 in CI and scripts/test so any future omission raises EncodingWarning under the existing "error" filter, run the two repo scripts with -X warn_default_encoding, and enable ruff's PLW1514 as the edit-time counterpart. - test_other_servers_preserved now seeds non-ASCII entries so the config round-trip is covered. Follows #3245.
Contributor
📚 Documentation preview
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3245 (its commit is included here; this PR shrinks to one commit once that merges). Review the second commit only.
Motivation and Context
#3244 / #3245 fix the two
read_text()calls the test suite trips over on a Windows machine whose ANSI code page isn't cp1252. The underlying problem is wider than those two lines: any text-modeopen()/Path.read_text()/write_text()/tempfile/subprocess(text=True)withoutencoding=uses the process locale, and CI structurally can't notice — ubuntu is UTF-8, andwindows-latestis cp1252, which happens to decode every non-ASCII byte we currently have (em dashes, box drawing, arrows) as mojibake rather than raising. On cp936/cp932/cp949/cp950 the same bytes are illegal sequences.The one place this reaches users is
mcp install:src/mcp/cli/claude.pyreadclaude_desktop_config.jsonwith the locale codec and wrote it back throughjson.dumps, so on a stock cp1252 or a CJK Windows box, pre-existing entries containing non-ASCII (aC:\Users\书清\…path, a server namedmétéo) were either rejected or silently persisted as mojibake while the command reported success. The stories harness (examples/stories/_harness.py) also reads the same manifest #3245 fixes in the tests, so everypython -m stories.<name>.clientREADME command failed on such a machine, as did the two repo scripts.This PR:
encoding="utf-8"at every remaining call site (cli, stories harness, scripts, tests, simple-chatbot, onedocs_srcsnippet); the two diagnosticsubprocess.run(..., text=True)captures becomeencoding="utf-8", errors="replace"; and drops a✓fromupdate_readme_snippets.py's output, which raisedUnicodeEncodeErrorwhenever stdout was a non-UTF-8 pipe (e.g. under pre-commit on Windows).scripts/testsetPYTHONWARNDEFAULTENCODING=1(PEP 597), so an omittedencoding=raisesEncodingWarningat the call site on every OS, which our existingfilterwarnings = ["error"]makes fatal (env var rather than-Xso pytest-xdist workers inherit it; one scoped ignore forpytest_examples' ownPopen). The two script steps run with-X warn_default_encoding -W error::EncodingWarning. ruff'sPLW1514is enabled as the edit-time counterpart (preview rule, hencepreview = true+explicit-preview-rules = trueso nothing else changes); on its own it only recognises obviousPath(...)receivers and misses e.g.(a / "b").read_text(), which is why the runtime guard is the primary mechanism.test_other_servers_preservedto seed non-ASCII entries, so themcp installround-trip is covered directly.How Has This Been Tested?
mainwith only the env var set fails deterministically attests/examples/conftest.py:41(EncodingWarning: 'encoding' argument not specified) on Linux and onwindows-latest; with this branch./scripts/testis green (5582 passed, 100% coverage, strict-no-cover clean).PYTHONWARNDEFAULTENCODING=1onwindows-latest(3.10 and 3.12), both at the stock code page 1252 and with the runner switched to 936: green, no additional warning sources.zh_CN.GBK/en_US.CP1252glibc locales (PYTHONUTF8=0): drovemcp installagainst a config seeded with Chinese and accented entries — before: cp1252 rewrites them as文件系统/José, GBK fails with'gbk' codec can't decode byte 0x94; after: entries byte-identical, new server added, re-install merges env vars.python -m stories.tools.clientand both scripts'--checkgo fromUnicodeDecodeErrorto OK under GBK.claude.pyunder cp1252, under GBK, and under the warning guard on a UTF-8 box.Breaking Changes
None.
mcp installwrites the same bytes as before (json.dumpsstill ASCII-escapes); it just decodes the existing file correctly first.Types of changes
Checklist
Additional context
Set-WinSystemLocale zh-CNtakes effect for new processes on the hosted image without a reboot). It would cover third-party code and spawned children, which the warning can't reach; happy to add it if we think it earns its keep.AI Disclaimer