Summary

Recent CSV compatibility work suggests that RustPython is reaching the architectural boundary of the current csv-core adapter.

This is not intended to argue that the recent incremental fixes were wrong. In particular:

• csv: fix csv escape fieldsep #8260 fixed a real compatibility gap for QUOTE_NONE with escapechar.
• csv: handle empty fields with skipinitialspace #8304 fixes incorrect and unsafe skipinitialspace preprocessing and needs quote provenance for QUOTE_NOTNULL / QUOTE_STRINGS.

However, the implementation has started to grow a second CSV parser alongside csv-core. Features that are still missing—multiline records across iterator items, strict, complete escape handling, Unicode dialect characters, empty-line behavior, and the remaining quote modes—will require more persistent parser state and more CPython-specific semantics.

At this point, it seems useful to compare the contracts of CPython, csv-core, and RustPython before the custom parser grows further.

This issue:

1. summarizes the current implementation differences;
2. explains why an unchanged csv-core wrapper cannot provide full CPython compatibility without duplicating much of the grammar;
3. compares the main long-term architecture options without selecting one.

My initial preference is to preserve the performance model of csv-core, potentially through a RustPython-specific adaptation or fork. However, Unicode and strict error semantics may make that a much larger redesign than it first appears, so this issue intentionally leaves the final choice open.

Related work

The issues are individually valid and can continue to receive narrow fixes. The architectural concern is that each fix currently has to decide independently whether its semantics belong in FormatOptions, preprocessing, csv-core, a custom parser, or post-processing.

Scope

The primary subject is the reader parser architecture.

Dialect representation and validation are included only where they constrain the parser architecture, especially Unicode dialect characters and the conversion to csv-core types.

Writer implementation is not proposed to share the reader state machine. The writer issues above are included as evidence that the current dialect abstraction also affects writer compatibility.

This issue is not intended to block small compatibility fixes. Its purpose is to make the long-term cost of each parser direction visible before larger changes accumulate around the current hybrid structure.

Current implementation layers

At a high level, RustPython currently has these layers:

Python arguments / dialect object
        |
        v
FormatOptions + PyDialect
        |
        +-----------------------+
        |                       |
        v                       v
csv_core::Reader         custom Rust parser paths
(common quote modes)     (currently selected cases)
        |                       |
        +-----------+-----------+
                    |
                    v
          Python field conversion

This layering has allowed incremental progress, but parser semantics are now distributed across:

• argument parsing and dialect conversion;
• csv-core::ReaderBuilder;
• raw-input preprocessing such as skipinitialspace;
• custom quote-mode paths;
• post-processing for QUOTE_NONNUMERIC, QUOTE_STRINGS, and QUOTE_NOTNULL;
• outer iterator lifecycle and line_num.

The same input may be quote-scanned by RustPython and then parsed again by csv-core.

1. CPython 3.14 contract

Reference:

• Modules/_csv.c at v3.14.6
• Lib/test/test_csv.py at v3.14.6

Dialect representation

CPython stores the following as Python Unicode or Unicode code points:

• delimiter: one Unicode character;
• quotechar: one Unicode character or None;
• escapechar: one Unicode character or None;
• lineterminator: an arbitrary string.

It validates the dialect once when constructing the internal dialect object. Validation includes:

• exact accepted types;
• exactly one character where required;
• valid quoting enum values;
• requiring quotechar when quoting is enabled;
• conflicts among delimiter, quotechar, escapechar, and lineterminator;
• special restrictions involving CR, LF, spaces, and skipinitialspace.

The reader does not use lineterminator to recognize input records. CPython's reader recognizes CR and LF directly. lineterminator is primarily a writer setting.

Reader state machine

CPython's reader uses a persistent nine-state parser:

START_RECORD
START_FIELD
ESCAPED_CHAR
IN_FIELD
IN_QUOTED_FIELD
ESCAPE_IN_QUOTED_FIELD
QUOTE_IN_QUOTED_FIELD
EAT_CRNL
AFTER_ESCAPED_CRNL

The parser consumes Python Unicode code points, not encoded bytes.

Important properties:

• One reader.__next__() may pull multiple strings from the input iterator until a complete CSV record is produced.
• After each iterator item, CPython injects a virtual EOL sentinel.
• Virtual EOL means “this iterator item ended”; it is not true iterator EOF and is not identical to CR or LF.
• True input EOF occurs only when the input iterator raises StopIteration.
• Parser state persists across iterator items.
• A quoted record may therefore span multiple items even when an item does not end with a physical newline.
• An escape at an iterator-item boundary has defined behavior.
• strict=True can reject invalid characters after a closing quote and unexpected true EOF.
• The parser retains whether the field was quoted. That metadata is required by QUOTE_NOTNULL, QUOTE_STRINGS, and numeric conversion.
• Empty physical lines and empty iterator items have defined behavior.
• line_num counts successfully obtained input iterator items, not returned CSV records.

Output conversion

Parsing and Python-level conversion are related but separable:

• quoted/unquoted provenance determines None conversion;
• unquoted non-empty fields may be converted with float() for QUOTE_NONNUMERIC and QUOTE_STRINGS;
• field-size checks occur while building a field;
• the parser produces one Python record only after reaching START_RECORD again.

2. csv-core contract

References:

• csv-core reader documentation
• csv-core/src/reader.rs

RustPython declares csv-core = "0.1.11" as a caret dependency; the current lockfile resolves 0.1.13.

Performance model

csv-core is intentionally optimized around byte input:

• parser syntax characters are u8;
• input is expected to be at least ASCII-compatible;
• the implementation contains an explicit NFA and generates a compact DFA transition table from it;
• parser configuration is compiled into the transition table;
• read_record writes unescaped field data into caller-provided output buffers and writes field end offsets into another buffer;
• the core parser does not allocate.

This is a strong implementation for its intended contract.

Semantic differences from CPython

csv-core deliberately prefers a parse over rejecting malformed input.

Relevant differences include:

• special dialect characters are bytes, not arbitrary Python Unicode characters;
• an empty byte slice is the parser's true EOF signal;
• there is no Python iterator-item boundary event;
• parsing never returns a malformed-CSV error;
• open quoted fields are finalized at EOF instead of producing CPython's strict error;
• escape handling is implemented inside quoted fields, not in unquoted fields;
• skipinitialspace is not part of the reader state machine;
• quoted/unquoted field provenance is not returned;
• empty input-line behavior differs;
• internal NFA/DFA state is private, so an adapter cannot inspect whether InputEmpty occurred in an unquoted field, quoted field, escape state, or closing-quote state;
• the reader's Terminator abstraction is byte-oriented and differs from CPython's input model.

These are not necessarily bugs in csv-core; they reflect a different public contract and design philosophy.

3. RustPython's current contract

Reference:

• crates/stdlib/src/csv.rs
• csv: fix csv escape fieldsep #8260
• csv: handle empty fields with skipinitialspace #8304

Dialect model

RustPython currently stores:

struct PyDialect {
    delimiter: u8,
    quotechar: Option<u8>,
    escapechar: Option<u8>,
    doublequote: bool,
    skipinitialspace: bool,
    lineterminator: csv_core::Terminator,
    quoting: QuoteStyle,
    strict: bool,
}

Consequences:

• “one Python character” is currently implemented as “exactly one encoded byte” in several paths;
• non-ASCII delimiter, quotechar, or escapechar cannot be represented correctly;
• a full Python lineterminator string cannot be represented;
• reader and writer configuration are constrained by csv-core types before the Python dialect contract has been fully resolved;
• validation and dialect override behavior are spread across TryFromObject, FromArgs, result, to_reader, and to_writer;
• invalid-type behavior can differ depending on whether a setting comes from a dialect object or a keyword argument, as seen in csv: reader allows escapechar=1 #8284.

Reader lifecycle

The current outer Reader.next() fetches one Python iterator item.

The csv-core path may call read_record repeatedly to resize buffers or consume the remainder of that same item, but it does not fetch another Python item for the same record. When the current byte slice is exhausted, passing an empty slice to csv-core also acts as csv-core EOF and can finalize an open quote.

This differs from CPython, where:

• iterator-item end is a virtual EOL event;
• true EOF is a separate event;
• one record may consume multiple iterator items.

Custom parser growth

#8260 added a small parser for QUOTE_NONE + escapechar, because an escaped delimiter in an unquoted field cannot be expressed through the current csv-core reader API.

#8304 generalizes this into quote-aware scanning and uses custom parsing for:

• QUOTE_NONE with an escape character;
• QUOTE_NOTNULL;
• QUOTE_STRINGS.

It also uses the scanner as a preprocessing pass for skipinitialspace before the common csv-core path.

This is the point where the custom implementation becomes more than a local exception:

• it recognizes field starts;
• it tracks quoted/unquoted state;
• it handles quote closing;
• it handles doubled quotes;
• it handles escapes;
• it recognizes delimiters and record terminators;
• it records quoted-field provenance.

Those are the core responsibilities of a CSV parser.

The custom path still has local, per-item state, so extending it to multiline records and strict EOF behavior will require moving parser state into the reader object and changing the outer iterator lifecycle.

Performance uncertainty

The common path retains csv-core's DFA, but skipinitialspace in #8304 first quote-scans and copies the complete input before csv-core parses it again.

The custom path allocates per-field buffers instead of using csv-core's contiguous output buffer and field-end offsets.

This does not prove that the end-to-end implementation is slow. Python object creation may dominate total cost. It does mean that the performance benefit of keeping csv-core should be measured at the RustPython API level rather than assumed from csv-core microbenchmarks.

Compatibility matrix

The “RustPython proposed” column refers to the architecture direction exposed by #8304, not necessarily merged main.

Why an unchanged thin wrapper is insufficient

Some missing behavior can be implemented around csv-core:

• dialect validation;
• Python object conversion;
• line counting;
• some empty-line handling;
• quote provenance, if a separate scanner tracks it.

The difficult cases are those where csv-core consumes syntax before the wrapper can reinterpret it.

Unquoted escape

In CPython, escape is recognized in both START_FIELD and IN_FIELD.

If an escaped byte is a delimiter, CR, LF, quote, or another special byte, unchanged csv-core will still interpret that byte according to its grammar. A wrapper would need to:

• replace escaped syntax with collision-free placeholders and restore it later;
• bypass csv-core output for parts of a field;
• or dispatch those dialects to another parser.

All of these add a second grammar and difficult state synchronization.

Strict parsing

csv-core does not expose enough internal state to distinguish:

• a normal unquoted field;
• an open quoted field;
• an escape-pending state;
• the state immediately after a closing quote.

A wrapper can run a shadow CPython-like FSM over the raw input, but then the wrapper is implementing most of the parser grammar solely to validate another parser.

Iterator boundaries

ReadRecordResult::InputEmpty does not tell the caller whether more Python iterator items should be fetched as part of the same record. An empty input slice means true EOF to csv-core, while CPython requires a distinct item-boundary event.

A shadow state machine can decide whether to fetch another item, but again this duplicates the parser state.

Therefore, a complete wrapper is technically possible, but it is no longer a thin adapter and may be harder to reason about than a single compatibility parser.

Architecture options

Option A: Continue the current hybrid approach

Keep csv-core for common cases and add custom paths or preprocessing for each unsupported feature.

Pros

• smallest incremental changes;
• existing working paths remain stable;
• preserves csv-core throughput for inputs that avoid preprocessing and custom dispatch;
• individual contributors can fix isolated compatibility failures.

Cons

• quote modes and dialect combinations can select different parsers;
• parser semantics are distributed across preprocessing, core parsing, custom parsing, and conversion;
• multiline and strict handling require persistent state in every applicable path;
• each new feature increases the cross-product of dialect options and parser paths;
• duplicate grammars can diverge on malformed input and boundaries;
• performance can include double scanning and extra allocation;
• Unicode still requires a separate design.

Assessment

Reasonable for narrow fixes, but not a good long-term architecture if CPython compatibility remains the goal.

Option B: Keep unchanged csv-core plus a comprehensive wrapper/shadow FSM

Use csv-core for output generation while a RustPython-owned FSM tracks CPython state, validates strict mode, handles iterator boundaries, and records provenance.

Pros

• continues using the current dependency without a fork;
• retains the optimized DFA and contiguous output buffers;
• may preserve good performance for ASCII data;
• can be introduced incrementally around the current backend.

Cons

• the shadow FSM duplicates most of the CSV grammar;
• two parsers must consume exactly the same logical input and remain synchronized;
• unquoted escape cannot be represented cleanly through the existing public API;
• placeholder or output-bypass schemes complicate field offsets and correctness;
• private csv-core state prevents direct synchronization checks;
• debugging requires understanding both parsers;
• Unicode remains outside the core backend.

Assessment

Possible, but the complexity is hidden rather than removed. A prototype should demonstrate that it is materially simpler or faster than an independent parser before choosing this option.

Option C: Implement one RustPython-owned CPython-compatible Unicode parser

Model the CPython state machine directly in Rust, operating on a Python-character representation rather than u8.

Pros

• one source of truth for quote, escape, delimiter, strict, multiline, empty-line, and boundary semantics;
• parser states map directly to CPython tests and source;
• arbitrary Python delimiter, quotechar, and escapechar become representable;
• virtual item end and true EOF can be first-class events;
• quote provenance and conversion metadata are natural outputs;
• no backend dispatch by quote style;
• easier differential testing against CPython.

Cons

• RustPython owns parser correctness and maintenance;
• initial implementation and review scope are larger;
• careful buffer design is needed to avoid unnecessary allocation;
• the existing csv-core reader performance path may be lost;
• performance needs to be recovered through profiling and optimization;
• CPython's behavior includes subtle permissive cases, not just the obvious nine states.

Assessment

The clearest route to compatibility and maintainability, but potentially the largest short-term implementation and performance risk.

Option D: Adapt, fork, or vendor csv-core for RustPython

Add the missing states and metadata to the csv-core NFA/DFA design:

• skipinitialspace;
• unquoted escape states;
• iterator-item boundary versus true EOF;
• strict error transitions;
• quoted/unquoted provenance;
• CPython empty-line behavior;
• Unicode-capable dialect syntax.

The practical form could be a RustPython-specific fork, vendored code, or a substantial adapter-owned variant. An upstream contribution would only be realistic where the required changes are sufficiently general; it should not be assumed as the primary path.

Pros

• preserves the NFA-to-DFA approach and configuration-specific transition tables;
• can retain contiguous caller-provided output buffers;
• unsupported semantics become real parser transitions instead of preprocessing;
• one parser can potentially serve all quote modes;
• offers the strongest chance of combining compatibility with the current performance model.

Cons

• csv-core intentionally uses bytes and intentionally never reports malformed-CSV errors; CPython compatibility conflicts with both design choices;
• Unicode support is not a small type substitution because APIs, input indexing, output buffers, and equivalence-class construction are byte-oriented;
• a RustPython-specific fork introduces dependency maintenance and rebase burden;
• the result could become a mostly independent parser while retaining complex DFA machinery;
• implementation and review cost are high;
• the required work may exceed the value of preserving the existing backend.

Important note

Unicode does not make a DFA impossible. A parser can classify each Python character into a small set such as delimiter, quote, escape, space, CR, LF, item-end, and other. The difficulty is that the current csv-core API and buffer model are byte-based, so reaching that design requires a substantial refactor.

Assessment

Potentially the best performance/compatibility combination, but also the highest architectural and maintenance ambition. Its feasibility is uncertain enough that it should remain an option rather than the conclusion of this issue.

Option E: Use two explicit backends

Use a fast ASCII backend when a dialect is fully representable and a compatibility backend otherwise.

For example:

ResolvedDialect
      |
      +-- Fast backend: csv-core or an extended csv-core
      |
      +-- Compatibility backend: RustPython Unicode FSM

Pros

• preserves optimized performance for common ASCII CSV;
• provides a direct path to full Unicode and CPython semantics;
• makes fallback conditions explicit instead of growing quote-mode-specific exceptions;
• backend performance can evolve independently.

Cons

• two parser implementations must remain semantically equivalent in their overlapping domain;
• differential testing becomes mandatory;
• dispatch criteria can become another source of bugs;
• malformed-input behavior may drift;
• maintenance and code size are higher than a unified parser;
• fast-path wins may be small after Python object conversion.

Assessment

A practical compromise if benchmarks show a meaningful end-to-end benefit. It should use a shared lifecycle and output contract so that only the transition engine differs.

Note on compatibility fixes during this discussion

Narrow fixes with focused tests can still be useful. The concern here is not the existence of custom code by itself, but whether larger additions continue to deepen several partially overlapping parser paths without an agreed long-term direction.

Discussion goal

The main questions for this issue are:

1. Is the current hybrid structure a sustainable long-term architecture, or should it be treated as transitional?
2. Among Options B–E, which direction best balances CPython compatibility, implementation complexity, maintenance cost, and the performance value of csv-core?

This issue does not need to choose an implementation plan immediately. Reaching a shared understanding of the architectural trade-offs would already provide guidance for subsequent CSV work.

Appendix: backend-independent constraints

The following points are relevant to any replacement or dual-backend design, but they are secondary to the architecture comparison above.

• The internal dialect model must be able to represent CPython's Unicode delimiter, quotechar, and escapechar before converting to backend-specific types.
• A Python iterator-item boundary must remain distinct from true input EOF.
• Parser state must be able to persist across iterator items for multiline records and boundary-sensitive escape handling.
• The parser must retain quoted/unquoted provenance for QUOTE_NOTNULL, QUOTE_STRINGS, and numeric conversion.
• Syntax parsing and Python object conversion should remain distinguishable responsibilities.
• skipinitialspace is safer as a parser transition than as a preprocessing pass that rewrites the input before another parser sees it.

These constraints do not imply one specific backend. They describe compatibility requirements that Options C, D, and E would need to address, and that make Option B difficult to keep thin.

References

RustPython

• Current csv.rs
• #8253: csv.writer ignores a dialect's quoting and lineterminator
• #8260: csv: fix csv escape fieldsep
• #8284: csv: reader allows escapechar=1
• #8302: embedded CR/LF not quoted with custom lineterminator
• #8304: csv: handle empty fields with skipinitialspace

CPython

• Modules/_csv.c at v3.14.6
• Lib/test/test_csv.py at v3.14.6
• csv documentation

rust-csv

• csv-core documentation
• csv-core/src/reader.rs
• csv-core/src/writer.rs