Skip to content

Fix StringIO newline splitting and character positions#8351

Open
doma17 wants to merge 3 commits into
RustPython:mainfrom
doma17:fix-stringio-read-semantics
Open

Fix StringIO newline splitting and character positions#8351
doma17 wants to merge 3 commits into
RustPython:mainfrom
doma17:fix-stringio-read-semantics

Conversation

@doma17

@doma17 doma17 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

AI assistance: codex gpt-5.6-sol

Summary

Fix StringIO(newline='') so readline() recognizes CR, LF, and CRLF boundaries. Also use character-based positions and sizes for StringIO.
This restores CR-separated email header parsing and prevents read(n) from splitting multibyte characters.
Remove expected-failure markers from the inherited test_memoryio and test_email cases that now pass.

Testing

Summary by CodeRabbit

  • Bug Fixes

    • String-based streams now measure stream positions, lengths, reads, and truncation in characters (not UTF-8 bytes), with correct handling for multibyte text.
    • write() now reports the number of characters written.
    • readline()/line slicing now respect the stream’s newline behavior accurately.
    • Restored stream state validates and reconstructs the cursor position and newline mode correctly.
  • New Features

    • Stream construction supports a configurable newline setting, which is preserved across save/restore.

doma17 added 2 commits July 23, 2026 11:51
Constraint: email.feedparser depends on StringIO(newline='') recognizing CR, LF, and CRLF boundaries.
Rejected: an email-specific parser workaround | StringIO is the shared root cause.
Confidence: high
Scope-risk: moderate
Directive: Keep internal storage byte-based; convert only at StringIO API boundaries.
Tested: prek run --all-files; test_memoryio; test_email; full workspace verification by contributor
Assisted-by: Codex:gpt-5.6-sol
Constraint: upstream StringIO tests already cover the repaired behavior.
Rejected: new RustPython-only regression tests | existing CPython coverage is sufficient.
Confidence: high
Scope-risk: narrow
Directive: Remove expected-failure markers only while the inherited tests pass.
Tested: prek run --all-files; test_memoryio; test_email
Assisted-by: Codex:gpt-5.6-sol
Copilot AI review requested due to automatic review settings July 23, 2026 02:53
@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: 54560721-0253-42af-be2b-7e05b7f306d1

📥 Commits

Reviewing files that changed from the base of the PR and between b87b789 and 7c097b5.

⛔ Files ignored due to path filters (2)
  • Lib/test/test_memoryio.py is excluded by !Lib/**
  • Lib/test/test_shlex.py is excluded by !Lib/**
📒 Files selected for processing (1)
  • crates/vm/src/stdlib/_io.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/vm/src/stdlib/_io.rs

📝 Walkthrough

Walkthrough

StringIO now stores newline configuration, converts between character and byte offsets, performs character-based I/O, and preserves newline mode and character position during pickling.

Changes

StringIO character offsets

Layer / File(s) Summary
Newline and offset model
crates/vm/src/stdlib/_io.rs
StringIO accepts and stores newline modes, translates content, fixes CRLF scanning, and adds UTF-8 character/byte offset helpers.
Character-based I/O operations
crates/vm/src/stdlib/_io.rs
Read, write, seek, tell, readline, and truncate use character counts with newline-aware byte sizing.
Pickle state restoration
crates/vm/src/stdlib/_io.rs
Pickle state records newline mode and character position, validates restored positions, and converts positions for cursor seeking.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant StringIO
  participant UnderlyingByteBuffer
  Caller->>StringIO: request character count
  StringIO->>UnderlyingByteBuffer: calculate and read byte range
  UnderlyingByteBuffer-->>StringIO: return UTF-8 bytes
  StringIO-->>Caller: return character data
Loading

Suggested reviewers: copilot

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: fixing StringIO newline splitting and character-based positions.
Linked Issues check ✅ Passed StringIO now uses character-based reads, seeks, and newline-aware boundaries, which addresses the email.parser CR/LF/CRLF parsing and large-read failure.
Out of Scope Changes check ✅ Passed Changes stay focused on StringIO newline and position handling plus related pickle state, with no unrelated edits evident.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[x] lib: cpython/Lib/email
[ ] test: cpython/Lib/test/test_email (TODO: 8)

dependencies:

  • email

dependent tests: (53 tests)

  • email: test_email test_http_cookiejar test_httpservers test_mailbox test_smtplib test_urllib test_urllib2 test_urllib2_localnet test_urllibnet test_zipfile
    • http.client: test_docxmlrpc test_hashlib test_ssl test_ucn test_unicodedata test_wsgiref test_xmlrpc
      • logging.handlers: test_concurrent_futures test_logging test_pkgutil
      • urllib.request: test_pathlib test_pydoc test_sax test_site test_urllib2net
    • http.server: test_robotparser
      • pydoc: test_enum
    • importlib.metadata: test_importlib test_zoneinfo
    • mailbox: test_genericalias
    • pydoc:
      • pdb: test_pdb
    • smtplib: test_smtpnet
    • urllib.request:
      • pathlib: test_ast test_dbm_sqlite3 test_ensurepip test_importlib test_json test_launcher test_pathlib test_peg_generator test_pyrepl test_runpy test_tarfile test_tempfile test_tomllib test_tools test_traceback test_unparse test_venv test_winapi test_zipapp test_zipfile test_zstd

[x] lib: cpython/Lib/shlex.py
[x] test: cpython/Lib/test/test_shlex.py

dependencies:

  • shlex

dependent tests: (10 tests)

  • shlex: test_mimetypes test_random test_regrtest test_shlex test_venv test_webbrowser
    • pdb: test_pdb
    • webbrowser:
      • pydoc: test_enum test_pydoc
      • wsgiref.simple_server: test_wsgiref

[x] lib: cpython/Lib/io.py
[x] lib: cpython/Lib/_pyio.py
[ ] test: cpython/Lib/test/test_io.py (TODO: 13)
[x] test: cpython/Lib/test/test_bufio.py
[x] test: cpython/Lib/test/test_fileio.py (TODO: 1)
[ ] test: cpython/Lib/test/test_memoryio.py (TODO: 11)

dependencies:

  • io

dependent tests: (108 tests)

  • io: test__colorize test_android test_argparse test_ast test_asyncio test_base64 test_buffer test_bufio test_builtin test_bz2 test_calendar test_cmd test_cmd_line_script test_codecs test_compile test_compileall test_compiler_assemble test_concurrent_futures test_configparser test_contextlib test_csv test_dbm_dumb test_descr test_dis test_email test_enum test_file test_fileinput test_fileio test_ftplib test_generated_cases test_getpass test_gzip test_hashlib test_http_cookiejar test_httplib test_httpservers test_importlib test_inspect test_io test_json test_largefile test_logging test_lzma test_mailbox test_marshal test_memoryio test_memoryview test_mimetypes test_minidom test_multibytecodec test_optparse test_pathlib test_pdb test_peg_generator test_pickle test_pickletools test_platform test_plistlib test_pprint test_print test_profile test_pstats test_pty test_pulldom test_pydoc test_pyexpat test_pyrepl test_quopri test_regrtest test_robotparser test_sax test_shlex test_shutil test_site test_smtplib test_socket test_socketserver test_subprocess test_support test_sys test_tarfile test_tempfile test_threadedtempfile test_timeit test_tokenize test_traceback test_types test_typing test_unittest test_univnewlines test_urllib test_urllib2 test_uuid test_wave test_webbrowser test_winconsoleio test_wsgiref test_xml_dom_xmlbuilder test_xml_etree test_xml_etree_c test_xmlrpc test_xpickle test_zipapp test_zipfile test_zipimport test_zoneinfo test_zstd

Legend:

  • [+] path exists in CPython
  • [x] up-to-date, [ ] outdated

@doma17
doma17 marked this pull request as ready for review July 23, 2026 02:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes RustPython’s io.StringIO behavior to better match CPython for newline handling (notably newline='') and to use character-based offsets/sizes so reads don’t split multi-byte characters. This addresses email parsing edge cases (e.g., CR-separated headers) and removes several expectedFailure markers from CPython-derived tests that now pass.

Changes:

  • Implement character-based seek/tell/read/truncate behavior for StringIO using WTF-8 boundary-aware conversions.
  • Track and serialize the newline setting for StringIO, and adjust readline() sizing to avoid splitting characters and to recognize CR/LF/CRLF when newline=''.
  • Un-xfail affected test_memoryio and test_email cases that now pass.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
crates/vm/src/stdlib/_io.rs Updates StringIO to store newline, use char-based positions/sizes, and improve readline()/read() behavior around UTF-8/WTF-8 boundaries.
Lib/test/test_memoryio.py Removes RustPython-specific expectedFailure markers for StringIO behaviors that now pass.
Lib/test/test_email/test_email.py Removes RustPython-specific expectedFailure markers for feedparser/newline-related email tests that now pass.
Comments suppressed due to low confidence (1)

Lib/test/test_memoryio.py:1041

  • Similarly, CStringIOPickleTest.test_newline_empty was removed rather than just un-xfailing it. Re-adding the override (without @expectedFailure) keeps the upstream test layout consistent and makes future syncs/conflict resolution easier.
    @unittest.expectedFailure  # TODO: RUSTPYTHON; +
    def test_issue5265(self):
        return super().test_issue5265()

    @unittest.expectedFailure  # TODO: RUSTPYTHON; ?                   ^^^^^

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread crates/vm/src/stdlib/_io.rs Outdated
Comment thread Lib/test/test_memoryio.py Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
crates/vm/src/stdlib/_io.rs (2)

4467-4486: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift

read_size decodes the entire remaining buffer on every call — quadratic cost for line-by-line reads.

bytes spans from the current position to the end of the whole buffer, and Wtf8::from_bytes(bytes) validates/decodes all of it just to answer a bounded "how many bytes for size chars / until the next newline" question. Since read() and readline() both route through this, a common pattern like for line in string_io: degrades from ~O(total length) to ~O(total length²) on a large buffer, because each readline() call re-decodes everything after the current position instead of only the bytes needed to find the next boundary or satisfy size.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/vm/src/stdlib/_io.rs` around lines 4467 - 4486, Update read_size to
avoid constructing and decoding a Wtf8 value for the entire buffer remainder on
every call. Limit scanning to the bytes required to satisfy the requested size
or locate the next newline, while preserving the existing byte-count result and
handling invalid Wtf8 consistently. Keep the change scoped to read_size, which
serves both read and readline.

4440-4453: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

char_offset_to_byte always scans the whole buffer, even when the target is in range.

text.code_points().count() runs over the entire bytes slice unconditionally, before checking whether char_offset is actually in range. For the common case (char_offset < char_len), this makes every call O(buffer length) instead of O(char_offset). This function is on the hot path for seek(), truncate(), and __setstate__(), so seeking near the start of a large StringIO still costs a full-buffer scan.

⚡ Proposed fix: try the bounded conversion first
 fn char_offset_to_byte(
     bytes: &[u8],
     char_offset: usize,
     vm: &VirtualMachine,
 ) -> PyResult<usize> {
     let text = Wtf8::from_bytes(bytes)
         .ok_or_else(|| vm.new_value_error("Error Retrieving Value"))?;
-    let char_len = text.code_points().count();
-    if char_offset >= char_len {
-        return Ok(bytes.len() + (char_offset - char_len));
-    }
-    Ok(crate::common::str::codepoint_range_end(text, char_offset)
-        .expect("char_offset is within the string"))
+    match crate::common::str::codepoint_range_end(text, char_offset) {
+        Some(byte_pos) => Ok(byte_pos),
+        None => {
+            let char_len = text.code_points().count();
+            Ok(bytes.len() + (char_offset - char_len))
+        }
+    }
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/vm/src/stdlib/_io.rs` around lines 4440 - 4453, Update
char_offset_to_byte to attempt codepoint_range_end(text, char_offset) before
scanning for the total code-point count, returning the bounded result when the
offset is in range. Only count code points after that attempt fails, then
preserve the existing bytes.len() plus overflow-offset behavior for out-of-range
offsets.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/vm/src/stdlib/_io.rs`:
- Around line 4541-4582: Update the seek implementation around the how/offset
calculation to acquire self.buffer(vm)? once before the match and retain that
guard through byte-offset conversion and the final seek. For how=1 and how=2
with zero offset, extract the differing byte position from the existing buffer,
then perform a single shared byte_offset_to_char call after the match; preserve
all existing validation and error behavior.

---

Nitpick comments:
In `@crates/vm/src/stdlib/_io.rs`:
- Around line 4467-4486: Update read_size to avoid constructing and decoding a
Wtf8 value for the entire buffer remainder on every call. Limit scanning to the
bytes required to satisfy the requested size or locate the next newline, while
preserving the existing byte-count result and handling invalid Wtf8
consistently. Keep the change scoped to read_size, which serves both read and
readline.
- Around line 4440-4453: Update char_offset_to_byte to attempt
codepoint_range_end(text, char_offset) before scanning for the total code-point
count, returning the bounded result when the offset is in range. Only count code
points after that attempt fails, then preserve the existing bytes.len() plus
overflow-offset behavior for out-of-range offsets.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: a4baea76-da38-4196-b743-3aec340a0a93

📥 Commits

Reviewing files that changed from the base of the PR and between dd9bce5 and b87b789.

⛔ Files ignored due to path filters (2)
  • Lib/test/test_email/test_email.py is excluded by !Lib/**
  • Lib/test/test_memoryio.py is excluded by !Lib/**
📒 Files selected for processing (1)
  • crates/vm/src/stdlib/_io.rs

Comment thread crates/vm/src/stdlib/_io.rs
Constraint: StringIO must apply its newline mode consistently when constructing, writing, and reading text.\nRejected: a readline-only fix | it leaves CR and CRLF modes internally inconsistent.\nConfidence: high\nScope-risk: moderate\nDirective: Keep StringIO buffer contents valid WTF-8 before using unchecked views.\nTested: prek run --all-files; cargo clippy -p rustpython-vm -- -D warnings; test_memoryio; test_shlex; test_email; workspace excluding rustpython-capi; full workspace manually verified by contributor\nNot-tested: local rustpython-capi workspace test crashes with a pre-existing macOS SIGSEGV\nAssisted-by: Codex:gpt-5.6-sol
Copilot AI review requested due to automatic review settings July 23, 2026 05:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inconsistent behaviors for Parser().parsestr(t).keys() in email.parser

2 participants