Skip to content

Pin text I/O to UTF-8 and fail CI on locale-dependent reads/writes - #3296

Draft
maxisbey wants to merge 2 commits into
mainfrom
encoding-guard
Draft

Pin text I/O to UTF-8 and fail CI on locale-dependent reads/writes#3296
maxisbey wants to merge 2 commits into
mainfrom
encoding-guard

Conversation

@maxisbey

Copy link
Copy Markdown
Contributor

Stacked on #3245 (its commit is included here; this PR shrinks to one commit once that merges). Review the second commit only.

Motivation and Context

#3244 / #3245 fix the two read_text() calls the test suite trips over on a Windows machine whose ANSI code page isn't cp1252. The underlying problem is wider than those two lines: any text-mode open() / Path.read_text() / write_text() / tempfile / subprocess(text=True) without encoding= uses the process locale, and CI structurally can't notice — ubuntu is UTF-8, and windows-latest is cp1252, which happens to decode every non-ASCII byte we currently have (em dashes, box drawing, arrows) as mojibake rather than raising. On cp936/cp932/cp949/cp950 the same bytes are illegal sequences.

The one place this reaches users is mcp install: src/mcp/cli/claude.py read claude_desktop_config.json with the locale codec and wrote it back through json.dumps, so on a stock cp1252 or a CJK Windows box, pre-existing entries containing non-ASCII (a C:\Users\书清\… path, a server named météo) were either rejected or silently persisted as mojibake while the command reported success. The stories harness (examples/stories/_harness.py) also reads the same manifest #3245 fixes in the tests, so every python -m stories.<name>.client README command failed on such a machine, as did the two repo scripts.

This PR:

  • passes encoding="utf-8" at every remaining call site (cli, stories harness, scripts, tests, simple-chatbot, one docs_src snippet); the two diagnostic subprocess.run(..., text=True) captures become encoding="utf-8", errors="replace"; and drops a from update_readme_snippets.py's output, which raised UnicodeEncodeError whenever stdout was a non-UTF-8 pipe (e.g. under pre-commit on Windows).
  • makes the class fail CI from now on: the pytest step and scripts/test set PYTHONWARNDEFAULTENCODING=1 (PEP 597), so an omitted encoding= raises EncodingWarning at the call site on every OS, which our existing filterwarnings = ["error"] makes fatal (env var rather than -X so pytest-xdist workers inherit it; one scoped ignore for pytest_examples' own Popen). The two script steps run with -X warn_default_encoding -W error::EncodingWarning. ruff's PLW1514 is enabled as the edit-time counterpart (preview rule, hence preview = true + explicit-preview-rules = true so nothing else changes); on its own it only recognises obvious Path(...) receivers and misses e.g. (a / "b").read_text(), which is why the runtime guard is the primary mechanism.
  • upgrades test_other_servers_preserved to seed non-ASCII entries, so the mcp install round-trip is covered directly.

How Has This Been Tested?

  • Unmodified main with only the env var set fails deterministically at tests/examples/conftest.py:41 (EncodingWarning: 'encoding' argument not specified) on Linux and on windows-latest; with this branch ./scripts/test is green (5582 passed, 100% coverage, strict-no-cover clean).
  • Full suite with this change under PYTHONWARNDEFAULTENCODING=1 on windows-latest (3.10 and 3.12), both at the stock code page 1252 and with the runner switched to 936: green, no additional warning sources.
  • Under generated zh_CN.GBK / en_US.CP1252 glibc locales (PYTHONUTF8=0): drove mcp install against a config seeded with Chinese and accented entries — before: cp1252 rewrites them as 文件系统 / José, GBK fails with 'gbk' codec can't decode byte 0x94; after: entries byte-identical, new server added, re-install merges env vars. python -m stories.tools.client and both scripts' --check go from UnicodeDecodeError to OK under GBK.
  • The upgraded test fails on the old claude.py under cp1252, under GBK, and under the warning guard on a UTF-8 box.

Breaking Changes

None. mcp install writes the same bytes as before (json.dumps still ASCII-escapes); it just decodes the existing file correctly first.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

Additional context

  • Python 3.15 makes UTF-8 mode the default (PEP 686), which retires the failure mode for users on 3.15+, but 3.10–3.14 stay supported for a long while and the explicit argument remains correct after that.
  • Not done here: running one Windows matrix cell under code page 936 (Set-WinSystemLocale zh-CN takes effect for new processes on the hosted image without a reboot). It would cover third-party code and spawned children, which the warning can't reach; happy to add it if we think it earns its keep.

AI Disclaimer

ShuQingDollarVoyager and others added 2 commits August 11, 2026 15:07
Path.read_text() without an encoding argument uses the locale
preferred encoding, which on Windows is the ANSI code page (e.g.
GBK on Chinese-locale systems). manifest.toml and the story sources
are UTF-8, so tests/examples failed to collect or run there.

Fixes #3244

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Text-mode open()/Path.read_text()/write_text() without encoding= use the
process locale, which is the ANSI code page on Windows. CI can't see the
difference (ubuntu is UTF-8, windows-latest is cp1252 and happens to decode
our em dashes), but a cp936/cp932/cp949 machine raises, and `mcp install`
would read claude_desktop_config.json with the wrong codec and write any
pre-existing non-ASCII entries back as mojibake.

- Pass encoding="utf-8" at every remaining call site (cli, stories harness,
  scripts, tests, examples); subprocess captures decode as UTF-8 with
  errors="replace"; drop a non-ASCII glyph from a script's piped output.
- Run pytest with PYTHONWARNDEFAULTENCODING=1 in CI and scripts/test so any
  future omission raises EncodingWarning under the existing "error" filter,
  run the two repo scripts with -X warn_default_encoding, and enable ruff's
  PLW1514 as the edit-time counterpart.
- test_other_servers_preserved now seeds non-ASCII entries so the config
  round-trip is covered.

Follows #3245.
@github-actions

Copy link
Copy Markdown
Contributor

📚 Documentation preview

Preview https://b678632e.mcp-python-docs.pages.dev
Deployment https://b678632e.mcp-python-docs.pages.dev
Commit 6c27385
Triggered by @maxisbey
Updated 2026-08-12 09:47:22 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants