[connectors] Support all CSV options for input connector#6517
Conversation
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
| headers: bool, | ||
|
|
||
| /// Quote character (as a byte), used for record splitting. | ||
| quote: u8, |
There was a problem hiding this comment.
Apparently this is the design of the Rust csv crate, which apparently matches the definition of the CSV format, which is byte-oriented, and not Unicode: https://www.rfc-editor.org/info/rfc4180/
So I don't think we should diverge here
| ); | ||
| } | ||
|
|
||
| // ---- CsvSplitter option tests ---- |
mythical-fred
left a comment
There was a problem hiding this comment.
Solid expansion of CsvParserConfig. The new fields (quote, escape, double_quote, quoting, comment, flexible, trim) mirror the underlying csv crate's ReaderBuilder faithfully, defaults are backward-compatible, the csv_reader_builder helper is centralized so the reader and the deserializer can't drift, and end-to-end coverage actually exercises a live CsvParser via mock_parser_pipeline. Test density is excellent — splitter unit tests for every interaction (headers + comment, custom quote, quoting=false, comment interleaved) plus E2E for each field.
A few non-blocking observations:
quote: charvs ASCII byte (carryover of gz's nit at line 111). Thecsvcrate takes au8, and the doc-comment correctly says "Must be an ASCII character." Butcharis a Unicode scalar (up to 4 bytes), and theas u8truncations atCsvParser::new/CsvSplitter/csv_reader_buildersilently mangle any non-ASCII config the user provides — e.g.quote: 'ä'will deserialize, truncate to0xE4, and behave unpredictably. Two options, equally fine: (a) add avalidate()-style check inCsvParserConfig(similar todelimiter(), which already returns aCsvDelimiterwrapper) that rejects non-ASCII at config-load time with a clear error pointing at the offending field; or (b) use the sameCsvDelimiter-style wrapper forquote/escape/comment. Option (a) is the smaller change. Thedelimiterfield already does this; the other ASCII fields should too, or we end up with one connector config that validates one way and the other six fields that don't.Splitter::clearresetsquoted,in_comment,at_line_startbut notheaders. Aclear()mid-stream after the header has been consumed will keepheaders = false(correct), but ifclear()is called before the first newline ever arrives, the next call toinput()will keep treating the next line as the header — which may or may not be what callers want. Worth a one-line code comment stating the intended semantics so future callers don't have to reverse-engineer it. (Carryover of the same kind of "what does reset mean" question as the original splitter.)flexible = trueas a default. This matches the existing behavior (the deserializer was unconditionally.flexible(true)before), but it does mean malformed CSV silently lands in the pipeline with missing fields. With the newflexible=falsetest in place, it would be worth a one-line note in the docs that the default is permissive and that users who care about data integrity should consider settingflexible: false. (Doc note only, not a code change.)- End-to-end test
csv_e2e_double_quote_disabled. Theassert_ne!(rows[0].first, "foo\"bar")correctly asserts the non-collapse, but it doesn't pin what the field should contain (csv-core'squote_escapes_no_doublesays"a""b"→a"b"). Either spell out the expected value (assert_eq!(rows[0].first, "foo\"bar\"")modulo trailing quote) or expand the comment so a future reader knows the assert isn't lazy. Minor. is_commentonly checks the first byte. This matches thecsvcrate'scommentsemantics, but the doc-comment onCsvParserConfig::commentsays "lines whose first byte matches this character are treated as comments and skipped" — good — and the splitter handles continuation lines correctly (comment ends at the next newline). One sub-case worth a test: a quoted field whose opening quote lands on a line that begins with the comment character — does that line get skipped? The current code says yes (the comment check happens before quote tracking), which is correct CSV semantics but a sharp edge worth pinning with a test.
Docs in csv.md are great — the per-field bullet list is exactly the right level of detail. Manual test plan box ticked, doc-checkbox ticked. gz already approved. From my side: APPROVE once the quote: char ASCII-validation question is at least filed as a follow-up issue if you don't want to address it in this PR.
|
@gz I put this back in the merge queue, since the design follows clearly the corresponding Rust crate |
This PR exposes all the options supported by the csv parser crate as input connector options.
I have tried to also enhance the output encoder, but ran into some problems, so I filed #6516.
Fixes #2424
Describe Manual Test Plan
Ran the new unit tests manually.
Checklist