Fix StringIO newline splitting and character positions#8351
Conversation
Constraint: email.feedparser depends on StringIO(newline='') recognizing CR, LF, and CRLF boundaries. Rejected: an email-specific parser workaround | StringIO is the shared root cause. Confidence: high Scope-risk: moderate Directive: Keep internal storage byte-based; convert only at StringIO API boundaries. Tested: prek run --all-files; test_memoryio; test_email; full workspace verification by contributor Assisted-by: Codex:gpt-5.6-sol
Constraint: upstream StringIO tests already cover the repaired behavior. Rejected: new RustPython-only regression tests | existing CPython coverage is sufficient. Confidence: high Scope-risk: narrow Directive: Remove expected-failure markers only while the inherited tests pass. Tested: prek run --all-files; test_memoryio; test_email Assisted-by: Codex:gpt-5.6-sol
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughStringIO now stores newline configuration, converts between character and byte offsets, performs character-based I/O, and preserves newline mode and character position during pickling. ChangesStringIO character offsets
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant StringIO
participant UnderlyingByteBuffer
Caller->>StringIO: request character count
StringIO->>UnderlyingByteBuffer: calculate and read byte range
UnderlyingByteBuffer-->>StringIO: return UTF-8 bytes
StringIO-->>Caller: return character data
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
📦 Library DependenciesThe following Lib/ modules were modified. Here are their dependencies: [x] lib: cpython/Lib/email dependencies: dependent tests: (53 tests)
[x] lib: cpython/Lib/shlex.py dependencies:
dependent tests: (10 tests)
[x] lib: cpython/Lib/io.py dependencies:
dependent tests: (108 tests)
Legend:
|
There was a problem hiding this comment.
Pull request overview
Fixes RustPython’s io.StringIO behavior to better match CPython for newline handling (notably newline='') and to use character-based offsets/sizes so reads don’t split multi-byte characters. This addresses email parsing edge cases (e.g., CR-separated headers) and removes several expectedFailure markers from CPython-derived tests that now pass.
Changes:
- Implement character-based
seek/tell/read/truncatebehavior forStringIOusing WTF-8 boundary-aware conversions. - Track and serialize the
newlinesetting forStringIO, and adjustreadline()sizing to avoid splitting characters and to recognize CR/LF/CRLF whennewline=''. - Un-xfail affected
test_memoryioandtest_emailcases that now pass.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
crates/vm/src/stdlib/_io.rs |
Updates StringIO to store newline, use char-based positions/sizes, and improve readline()/read() behavior around UTF-8/WTF-8 boundaries. |
Lib/test/test_memoryio.py |
Removes RustPython-specific expectedFailure markers for StringIO behaviors that now pass. |
Lib/test/test_email/test_email.py |
Removes RustPython-specific expectedFailure markers for feedparser/newline-related email tests that now pass. |
Comments suppressed due to low confidence (1)
Lib/test/test_memoryio.py:1041
- Similarly,
CStringIOPickleTest.test_newline_emptywas removed rather than just un-xfailing it. Re-adding the override (without@expectedFailure) keeps the upstream test layout consistent and makes future syncs/conflict resolution easier.
@unittest.expectedFailure # TODO: RUSTPYTHON; +
def test_issue5265(self):
return super().test_issue5265()
@unittest.expectedFailure # TODO: RUSTPYTHON; ? ^^^^^
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
crates/vm/src/stdlib/_io.rs (2)
4467-4486: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift
read_sizedecodes the entire remaining buffer on every call — quadratic cost for line-by-line reads.
bytesspans from the current position to the end of the whole buffer, andWtf8::from_bytes(bytes)validates/decodes all of it just to answer a bounded "how many bytes forsizechars / until the next newline" question. Sinceread()andreadline()both route through this, a common pattern likefor line in string_io:degrades from ~O(total length) to ~O(total length²) on a large buffer, because eachreadline()call re-decodes everything after the current position instead of only the bytes needed to find the next boundary or satisfysize.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/vm/src/stdlib/_io.rs` around lines 4467 - 4486, Update read_size to avoid constructing and decoding a Wtf8 value for the entire buffer remainder on every call. Limit scanning to the bytes required to satisfy the requested size or locate the next newline, while preserving the existing byte-count result and handling invalid Wtf8 consistently. Keep the change scoped to read_size, which serves both read and readline.
4440-4453: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win
char_offset_to_bytealways scans the whole buffer, even when the target is in range.
text.code_points().count()runs over the entirebytesslice unconditionally, before checking whetherchar_offsetis actually in range. For the common case (char_offset < char_len), this makes every call O(buffer length) instead of O(char_offset). This function is on the hot path forseek(),truncate(), and__setstate__(), so seeking near the start of a largeStringIOstill costs a full-buffer scan.⚡ Proposed fix: try the bounded conversion first
fn char_offset_to_byte( bytes: &[u8], char_offset: usize, vm: &VirtualMachine, ) -> PyResult<usize> { let text = Wtf8::from_bytes(bytes) .ok_or_else(|| vm.new_value_error("Error Retrieving Value"))?; - let char_len = text.code_points().count(); - if char_offset >= char_len { - return Ok(bytes.len() + (char_offset - char_len)); - } - Ok(crate::common::str::codepoint_range_end(text, char_offset) - .expect("char_offset is within the string")) + match crate::common::str::codepoint_range_end(text, char_offset) { + Some(byte_pos) => Ok(byte_pos), + None => { + let char_len = text.code_points().count(); + Ok(bytes.len() + (char_offset - char_len)) + } + } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/vm/src/stdlib/_io.rs` around lines 4440 - 4453, Update char_offset_to_byte to attempt codepoint_range_end(text, char_offset) before scanning for the total code-point count, returning the bounded result when the offset is in range. Only count code points after that attempt fails, then preserve the existing bytes.len() plus overflow-offset behavior for out-of-range offsets.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/vm/src/stdlib/_io.rs`:
- Around line 4541-4582: Update the seek implementation around the how/offset
calculation to acquire self.buffer(vm)? once before the match and retain that
guard through byte-offset conversion and the final seek. For how=1 and how=2
with zero offset, extract the differing byte position from the existing buffer,
then perform a single shared byte_offset_to_char call after the match; preserve
all existing validation and error behavior.
---
Nitpick comments:
In `@crates/vm/src/stdlib/_io.rs`:
- Around line 4467-4486: Update read_size to avoid constructing and decoding a
Wtf8 value for the entire buffer remainder on every call. Limit scanning to the
bytes required to satisfy the requested size or locate the next newline, while
preserving the existing byte-count result and handling invalid Wtf8
consistently. Keep the change scoped to read_size, which serves both read and
readline.
- Around line 4440-4453: Update char_offset_to_byte to attempt
codepoint_range_end(text, char_offset) before scanning for the total code-point
count, returning the bounded result when the offset is in range. Only count code
points after that attempt fails, then preserve the existing bytes.len() plus
overflow-offset behavior for out-of-range offsets.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro Plus
Run ID: a4baea76-da38-4196-b743-3aec340a0a93
⛔ Files ignored due to path filters (2)
Lib/test/test_email/test_email.pyis excluded by!Lib/**Lib/test/test_memoryio.pyis excluded by!Lib/**
📒 Files selected for processing (1)
crates/vm/src/stdlib/_io.rs
Constraint: StringIO must apply its newline mode consistently when constructing, writing, and reading text.\nRejected: a readline-only fix | it leaves CR and CRLF modes internally inconsistent.\nConfidence: high\nScope-risk: moderate\nDirective: Keep StringIO buffer contents valid WTF-8 before using unchecked views.\nTested: prek run --all-files; cargo clippy -p rustpython-vm -- -D warnings; test_memoryio; test_shlex; test_email; workspace excluding rustpython-capi; full workspace manually verified by contributor\nNot-tested: local rustpython-capi workspace test crashes with a pre-existing macOS SIGSEGV\nAssisted-by: Codex:gpt-5.6-sol
AI assistance: codex gpt-5.6-sol
Summary
Fix
StringIO(newline='')soreadline()recognizes CR, LF, and CRLF boundaries. Also use character-based positions and sizes forStringIO.This restores CR-separated email header parsing and prevents
read(n)from splitting multibyte characters.Remove expected-failure markers from the inherited
test_memoryioandtest_emailcases that now pass.Testing
prek run --all-filescargo run --release -- -m test test_memoryiocargo run --release -- -m test test_emailSummary by CodeRabbit
Bug Fixes
write()now reports the number of characters written.readline()/line slicing now respect the stream’s newline behavior accurately.New Features