Skip to content

Fix malformed Unicode error constructors - #8497

Draft
youknowone wants to merge 2 commits into
RustPython:mainfrom
youknowone:agent/fix-unicode-error-constructors
Draft

Fix malformed Unicode error constructors#8497
youknowone wants to merge 2 commits into
RustPython:mainfrom
youknowone:agent/fix-unicode-error-constructors

Conversation

@youknowone

@youknowone youknowone commented Aug 11, 2026

Copy link
Copy Markdown
Member

Fixes #8352.

Summary

  • replace every remaining message-only UnicodeDecodeError and UnicodeEncodeError construction with fully initialized exceptions
  • preserve the encoding, source object, failure range, and reason at CSV, socket, filesystem, array, tkinter, Windows codec, and POSIX call sites
  • remove the malformed helpers and rename the correct decode constructor to new_unicode_decode_error
  • preserve Windows filename surrogates through WTF-8 instead of rejecting them as invalid UTF-8

Root cause and impact

The old macro-generated helpers bypassed Unicode error initialization and created one-argument exceptions. Those exceptions had no encoding, object, start, end, or reason attributes, and their string representation could be empty. All affected paths now produce structurally valid exceptions that error handlers and user code can inspect.

Validation

  • cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher --exclude rustpython-capi
  • cargo check -p rustpython-stdlib --features tkinter
  • cargo check from crates/capi
  • targeted Windows runtime assertions for array UTF-16 reconstruction, mbcs/oem, nt path decoding, and the original CSV reproduction
  • pre-commit hooks (rustfmt, cspell, merge-conflict and patch checks)

The standalone C-API cargo test reaches the link step but cannot run in this checkout because its Windows configuration references the unavailable placeholder library pythonXY.lib; its cargo check passes.

AI assistance

Implementation and validation were assisted by Codex (GPT-5) and reviewed interactively by the contributor.

@youknowone
Fix malformed Unicode error constructors
46c9004 
Replace message-only Unicode decode and encode errors with fully initialized exceptions, preserve source objects and failure ranges, and remove the obsolete constructors.

Assisted-by: Codex:gpt-5
@coderabbitai

coderabbitai Bot commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: fc69120a-3676-48a2-8e1f-79a9ea12ece7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@youknowone
Fix Windows Unicode CI regressions
60c4100 
Assisted-by: Codex:gpt-5
@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[x] lib: cpython/Lib/codecs.py
[x] test: cpython/Lib/test/test_charmapcodec.py
[ ] test: cpython/Lib/test/test_codeccallbacks.py (TODO: 9)
[x] test: cpython/Lib/test/test_codecencodings_cn.py (TODO: 4)
[x] test: cpython/Lib/test/test_codecencodings_hk.py (TODO: 1)
[x] test: cpython/Lib/test/test_codecencodings_iso2022.py (TODO: 5)
[x] test: cpython/Lib/test/test_codecencodings_jp.py (TODO: 7)
[x] test: cpython/Lib/test/test_codecencodings_kr.py (TODO: 3)
[x] test: cpython/Lib/test/test_codecencodings_tw.py (TODO: 1)
[ ] test: cpython/Lib/test/test_codecmaps_cn.py (TODO: 3)
[ ] test: cpython/Lib/test/test_codecmaps_hk.py (TODO: 1)
[ ] test: cpython/Lib/test/test_codecmaps_jp.py (TODO: 6)
[ ] test: cpython/Lib/test/test_codecmaps_kr.py (TODO: 3)
[ ] test: cpython/Lib/test/test_codecmaps_tw.py (TODO: 3)
[ ] test: cpython/Lib/test/test_codecs.py (TODO: 12)
[ ] test: cpython/Lib/test/test_multibytecodec.py (TODO: 1)
[x] test: cpython/Lib/test/testcodec.py

dependencies:

  • codecs

dependent tests: (161 tests)

  • codecs: test_charmapcodec test_codeccallbacks test_codecs test_eof test_exceptions test_importlib test_inspect test_io test_json test_locale test_logging test_multibytecodec test_os test_pdb test_plistlib test_sax test_str test_sys
    • encodings: test_pydoc
      • locale: test__locale test_builtin test_c_locale_coercion test_calendar test_decimal test_float test_format test_re test_regrtest test_strftime test_strptime test_types test_utf8_mode
    • json: test_embed test_pyrepl test_subprocess test_sysconfig test_tomllib test_tools test_traceback test_zoneinfo
      • importlib.metadata: test_importlib
      • multiprocessing.resource_tracker: test_concurrent_futures
    • pickle: test_annotationlib test_argparse test_array test_ast test_asyncio test_bool test_bytes test_bz2 test_collections test_concurrent_futures test_configparser test_coroutines test_csv test_ctypes test_defaultdict test_deque test_descr test_dict test_dictviews test_email test_enum test_enumerate test_fractions test_functools test_generators test_genericalias test_http_cookies test_ipaddress test_iter test_itertools test_list test_lzma test_memoryio test_memoryview test_minidom test_opcache test_operator test_ordered_dict test_pathlib test_pickle test_picklebuffer test_pickletools test_platform test_positional_only_arg test_posix test_random test_range test_set test_shelve test_slice test_socket test_statistics test_string test_structseq test_super test_time test_trace test_tuple test_turtle test_type_aliases test_type_params test_typing test_unittest test_uuid test_xml_dom_minicompat test_xml_etree test_xpickle test_zipfile test_zlib test_zoneinfo
      • tracemalloc: test_tracemalloc
    • plistlib:
      • platform: test__osx_support test_asyncio test_baseexception test_cmath test_ctypes test_fcntl test_math test_mimetypes test_shutil test_ssl test_winreg test_wsgiref
    • tokenize: test_linecache test_peg_generator test_tabnanny test_tokenize test_unparse
      • inspect: test_abc test_asyncgen test_buffer test_clinic test_code test_grammar test_monitoring test_ntpath test_patma test_posixpath test_signal test_sqlite3 test_type_annotations test_yield_from test_zipimport test_zipimport_support
      • linecache: test_bdb
      • traceback: test_asyncio test_code_module test_contextlib test_contextlib_async test_dictcomps test_http_cookiejar test_importlib test_listcomps test_pyexpat test_setcomps test_threadedtempfile test_threading test_unittest test_with

Legend:

  • [+] path exists in CPython
  • [x] up-to-date, [ ] outdated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Replace malformed new_unicode_decode_error / new_unicode_encode_error helpers with the _real variants

1 participant

@youknowone