[connectors] Rename CsvParserConfig to CsvFormatConfig #6537
Conversation
mythical-fred
left a comment
There was a problem hiding this comment.
Mechanical rename of CsvParserConfig to CsvFormatConfig, plus per-field doc comments tagging each option as input-only or input+output. Good direction — the type is shared between parser and encoder, the old name was misleading, and clarifying which fields apply to encoding is exactly what's needed before adding more output options.
But the rename is incomplete in one place, and the new doc tags reveal a latent semantic question worth answering before this lands.
Blocker: missed reference in OtherTests.java
sql-to-dbsp-compiler/SQL-compiler/src/test/java/org/dbsp/sqlCompiler/compiler/sql/OtherTests.java still references CsvParserConfig inside the Rust string literal used by testCompilerExample:
line 439: use feldera_types::format::csv::CsvParserConfig;
line 473: .cursor(RecordFormat::Csv(CsvParserConfig::default()))
That string is appended to a generated Rust file and compiled by BaseSQLTests.compileAndTestRust(true). With the type renamed, this test will fail to compile. The PR description ("we can see whether this change can be merged, if it cannot, it is a sign that the change is not backwards-compatible") suggests the intent is to use CI to catch breakage — fine for the public API surface, but for internal test fixtures embedded in Java strings the rename has to be done by hand. Same fix for sql-to-dbsp-compiler/using.md, which you already touched in this PR.
Pre-existing typo worth fixing while you're in using.md
- use use feldera_types::format::csv::CsvParserConfig;
+ use use feldera_types::format::csv::CsvFormatConfig;
The doubled use use is wrong in both the before and after of the patch — it was a pre-existing typo. While you're touching these two lines, drop the duplicate use so the example actually compiles.
Architectural: the "Used by:" tags expose a leak
Tagging every field "Used by: input only" except delimiter ("Used by: input and output") admits in the docs what the struct itself does not: this is really an input config plus one output-relevant field. The encoder path in csv.rs already discards everything except delimiter:
CursorWithPolarity::new(batch.cursor(RecordFormat::Csv(CsvFormatConfig {
delimiter: self.config.delimiter,
..Default::default()
}))?)— so the other eight fields on the output side are silently ignored, not honored. If a user sets quote: '\'' on a CSV output connector and the encoder happily uses " anyway, that is a confusing UX regardless of what the doc string says. Two cleaner options:
-
Split the type now, while you're renaming.
CsvParserConfigkeeps everything; a smallerCsvEncoderConfig(which already exists incrates/adapters/src/format/csv.rs!) is the only thing the encoder sees. TheRecordFormat::Csvvariant in catalog.rs takes whichever makes sense at each call site (it's parser context, soCsvParserConfigis fine there). This avoids the "shared struct where most fields are silently dropped" trap and keeps the type names honest. -
If you genuinely want one struct and intend to start honoring more fields on output (which the body suggests — "preparation for adding some output formatting options"), say which fields will be added on the output side, and make the field-level docs forward-compatible (
Used by: input only (output support planned)) so we don't end up with three flavours of stale doc later.
Right now the rename + per-field tags lands the worst of both worlds: one type with mostly-ignored fields on the output side, plus documentation that admits the asymmetry without resolving it.
Verdict
REQUEST_CHANGES on the missed OtherTests.java reference — that's a concrete CI breakage. The using.md typo is a one-line bonus. The CsvFormatConfig vs CsvEncoderConfig split is a design call I'd push back on but is not a hard blocker for this PR — you can address it in the follow-up "add output formatting options" PR if you prefer to land the rename now.
… fact that it's used both for parsing and encoding Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
mythical-fred
left a comment
There was a problem hiding this comment.
Rename now reaches the OtherTests.java Rust string literal and the using.md snippet (which also picks up the pre-existing use use typo) — earlier blockers cleared.
The "Used by" doc still flags that this struct is really an input config plus delimiter; renaming to CsvFormatConfig is honest about that current state, but the encoder-ignores-most-fields asymmetry is a real footgun once the output side grows more options. Worth splitting into CsvParserConfig / extending CsvEncoderConfig in a follow-up before any new output knob lands.
Rename CsvParserConfig to CsvFormatConfig to reflect the fact that it's used both for parsing and encoding
Fixes #6516
We can see whether this change can be merged, if it cannot, it is a sign that the change is not backwards-compatible.
This is in preparation for adding some output formatting options.