Skip to content

[connectors] Rename CsvParserConfig to CsvFormatConfig #6537

Merged
mihaibudiu merged 1 commit into
feldera:mainfrom
mihaibudiu:issue6516
Jun 25, 2026
Merged

[connectors] Rename CsvParserConfig to CsvFormatConfig #6537
mihaibudiu merged 1 commit into
feldera:mainfrom
mihaibudiu:issue6516

Conversation

@mihaibudiu

Copy link
Copy Markdown
Contributor

Rename CsvParserConfig to CsvFormatConfig to reflect the fact that it's used both for parsing and encoding

Fixes #6516

We can see whether this change can be merged, if it cannot, it is a sign that the change is not backwards-compatible.
This is in preparation for adding some output formatting options.

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mechanical rename of CsvParserConfig to CsvFormatConfig, plus per-field doc comments tagging each option as input-only or input+output. Good direction — the type is shared between parser and encoder, the old name was misleading, and clarifying which fields apply to encoding is exactly what's needed before adding more output options.

But the rename is incomplete in one place, and the new doc tags reveal a latent semantic question worth answering before this lands.

Blocker: missed reference in OtherTests.java

sql-to-dbsp-compiler/SQL-compiler/src/test/java/org/dbsp/sqlCompiler/compiler/sql/OtherTests.java still references CsvParserConfig inside the Rust string literal used by testCompilerExample:

line 439:     use feldera_types::format::csv::CsvParserConfig;
line 473:         .cursor(RecordFormat::Csv(CsvParserConfig::default()))

That string is appended to a generated Rust file and compiled by BaseSQLTests.compileAndTestRust(true). With the type renamed, this test will fail to compile. The PR description ("we can see whether this change can be merged, if it cannot, it is a sign that the change is not backwards-compatible") suggests the intent is to use CI to catch breakage — fine for the public API surface, but for internal test fixtures embedded in Java strings the rename has to be done by hand. Same fix for sql-to-dbsp-compiler/using.md, which you already touched in this PR.

Pre-existing typo worth fixing while you're in using.md

-    use use feldera_types::format::csv::CsvParserConfig;
+    use use feldera_types::format::csv::CsvFormatConfig;

The doubled use use is wrong in both the before and after of the patch — it was a pre-existing typo. While you're touching these two lines, drop the duplicate use so the example actually compiles.

Architectural: the "Used by:" tags expose a leak

Tagging every field "Used by: input only" except delimiter ("Used by: input and output") admits in the docs what the struct itself does not: this is really an input config plus one output-relevant field. The encoder path in csv.rs already discards everything except delimiter:

CursorWithPolarity::new(batch.cursor(RecordFormat::Csv(CsvFormatConfig {
    delimiter: self.config.delimiter,
    ..Default::default()
}))?)

— so the other eight fields on the output side are silently ignored, not honored. If a user sets quote: '\'' on a CSV output connector and the encoder happily uses " anyway, that is a confusing UX regardless of what the doc string says. Two cleaner options:

  1. Split the type now, while you're renaming. CsvParserConfig keeps everything; a smaller CsvEncoderConfig (which already exists in crates/adapters/src/format/csv.rs!) is the only thing the encoder sees. The RecordFormat::Csv variant in catalog.rs takes whichever makes sense at each call site (it's parser context, so CsvParserConfig is fine there). This avoids the "shared struct where most fields are silently dropped" trap and keeps the type names honest.

  2. If you genuinely want one struct and intend to start honoring more fields on output (which the body suggests — "preparation for adding some output formatting options"), say which fields will be added on the output side, and make the field-level docs forward-compatible (Used by: input only (output support planned)) so we don't end up with three flavours of stale doc later.

Right now the rename + per-field tags lands the worst of both worlds: one type with mostly-ignored fields on the output side, plus documentation that admits the asymmetry without resolving it.

Verdict

REQUEST_CHANGES on the missed OtherTests.java reference — that's a concrete CI breakage. The using.md typo is a one-line bonus. The CsvFormatConfig vs CsvEncoderConfig split is a design call I'd push back on but is not a hard blocker for this PR — you can address it in the follow-up "add output formatting options" PR if you prefer to land the rename now.

… fact that it's used both for parsing and encoding

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
@mihaibudiu
mihaibudiu enabled auto-merge June 25, 2026 18:42
@mihaibudiu
mihaibudiu added this pull request to the merge queue Jun 25, 2026

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename now reaches the OtherTests.java Rust string literal and the using.md snippet (which also picks up the pre-existing use use typo) — earlier blockers cleared.

The "Used by" doc still flags that this struct is really an input config plus delimiter; renaming to CsvFormatConfig is honest about that current state, but the encoder-ignores-most-fields asymmetry is a real footgun once the output side grows more options. Worth splitting into CsvParserConfig / extending CsvEncoderConfig in a follow-up before any new output knob lands.

Merged via the queue into feldera:main with commit 1f5f758 Jun 25, 2026
1 check passed
@mihaibudiu
mihaibudiu deleted the issue6516 branch June 25, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[connectors] RecordFormat is used for both encoding and decoding

3 participants