Skip to content

[connectors] Support all CSV options for input connector#6517

Merged
mihaibudiu merged 1 commit into
feldera:mainfrom
mihaibudiu:issue2424
Jun 24, 2026
Merged

[connectors] Support all CSV options for input connector#6517
mihaibudiu merged 1 commit into
feldera:mainfrom
mihaibudiu:issue2424

Conversation

@mihaibudiu

@mihaibudiu mihaibudiu commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

This PR exposes all the options supported by the csv parser crate as input connector options.

I have tried to also enhance the output encoder, but ran into some problems, so I filed #6516.

Fixes #2424

Describe Manual Test Plan

Ran the new unit tests manually.

Checklist

  • Unit tests added/updated
  • Documentation updated

@mihaibudiu
mihaibudiu requested a review from gz June 22, 2026 01:19
@mihaibudiu
mihaibudiu enabled auto-merge June 22, 2026 01:20
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
headers: bool,

/// Quote character (as a byte), used for record splitting.
quote: u8,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

id expected this to be 'char'

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently this is the design of the Rust csv crate, which apparently matches the definition of the CSV format, which is byte-oriented, and not Unicode: https://www.rfc-editor.org/info/rfc4180/

So I don't think we should diverge here

);
}

// ---- CsvSplitter option tests ----

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whatsup with the ----

@mihaibudiu
mihaibudiu added this pull request to the merge queue Jun 22, 2026
@gz
gz removed this pull request from the merge queue due to a manual request Jun 22, 2026

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid expansion of CsvParserConfig. The new fields (quote, escape, double_quote, quoting, comment, flexible, trim) mirror the underlying csv crate's ReaderBuilder faithfully, defaults are backward-compatible, the csv_reader_builder helper is centralized so the reader and the deserializer can't drift, and end-to-end coverage actually exercises a live CsvParser via mock_parser_pipeline. Test density is excellent — splitter unit tests for every interaction (headers + comment, custom quote, quoting=false, comment interleaved) plus E2E for each field.

A few non-blocking observations:

  • quote: char vs ASCII byte (carryover of gz's nit at line 111). The csv crate takes a u8, and the doc-comment correctly says "Must be an ASCII character." But char is a Unicode scalar (up to 4 bytes), and the as u8 truncations at CsvParser::new / CsvSplitter / csv_reader_builder silently mangle any non-ASCII config the user provides — e.g. quote: 'ä' will deserialize, truncate to 0xE4, and behave unpredictably. Two options, equally fine: (a) add a validate()-style check in CsvParserConfig (similar to delimiter(), which already returns a CsvDelimiter wrapper) that rejects non-ASCII at config-load time with a clear error pointing at the offending field; or (b) use the same CsvDelimiter-style wrapper for quote / escape / comment. Option (a) is the smaller change. The delimiter field already does this; the other ASCII fields should too, or we end up with one connector config that validates one way and the other six fields that don't.
  • Splitter::clear resets quoted, in_comment, at_line_start but not headers. A clear() mid-stream after the header has been consumed will keep headers = false (correct), but if clear() is called before the first newline ever arrives, the next call to input() will keep treating the next line as the header — which may or may not be what callers want. Worth a one-line code comment stating the intended semantics so future callers don't have to reverse-engineer it. (Carryover of the same kind of "what does reset mean" question as the original splitter.)
  • flexible = true as a default. This matches the existing behavior (the deserializer was unconditionally .flexible(true) before), but it does mean malformed CSV silently lands in the pipeline with missing fields. With the new flexible=false test in place, it would be worth a one-line note in the docs that the default is permissive and that users who care about data integrity should consider setting flexible: false. (Doc note only, not a code change.)
  • End-to-end test csv_e2e_double_quote_disabled. The assert_ne!(rows[0].first, "foo\"bar") correctly asserts the non-collapse, but it doesn't pin what the field should contain (csv-core's quote_escapes_no_double says "a""b"a"b"). Either spell out the expected value (assert_eq!(rows[0].first, "foo\"bar\"") modulo trailing quote) or expand the comment so a future reader knows the assert isn't lazy. Minor.
  • is_comment only checks the first byte. This matches the csv crate's comment semantics, but the doc-comment on CsvParserConfig::comment says "lines whose first byte matches this character are treated as comments and skipped" — good — and the splitter handles continuation lines correctly (comment ends at the next newline). One sub-case worth a test: a quoted field whose opening quote lands on a line that begins with the comment character — does that line get skipped? The current code says yes (the comment check happens before quote tracking), which is correct CSV semantics but a sharp edge worth pinning with a test.

Docs in csv.md are great — the per-field bullet list is exactly the right level of detail. Manual test plan box ticked, doc-checkbox ticked. gz already approved. From my side: APPROVE once the quote: char ASCII-validation question is at least filed as a follow-up issue if you don't want to address it in this PR.

@mihaibudiu
mihaibudiu added this pull request to the merge queue Jun 24, 2026
@mihaibudiu

Copy link
Copy Markdown
Contributor Author

@gz I put this back in the merge queue, since the design follows clearly the corresponding Rust crate

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 24, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Jun 24, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 24, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Jun 24, 2026
Merged via the queue into feldera:main with commit 31558f6 Jun 24, 2026
1 check passed
@mihaibudiu
mihaibudiu deleted the issue2424 branch June 24, 2026 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[connectors] CSV encoder/decoder connectors do not expose any configuration

3 participants