Skip to content

connectors: make transport configs factory parsed#6529

Open
Dnreikronos wants to merge 6 commits into
feldera:mainfrom
Dnreikronos:connectors/opaque_transport_config
Open

connectors: make transport configs factory parsed#6529
Dnreikronos wants to merge 6 commits into
feldera:mainfrom
Dnreikronos:connectors/opaque_transport_config

Conversation

@Dnreikronos

Copy link
Copy Markdown
Contributor

refs #6446

imo this is the second slice after #6522. normal connector configs are open { name, config } values now, and the built-in transport factories parse their own payloads instead of making feldera-types know every transport.

i kept the old json shape lgtm: existing { name, config } configs still deserialize, no-config transports can omit config, and sql property generation still catches known wrong-direction built-ins while letting unknown dynamic names through. idk if integrated connector registries belong in this stack, since they need a different factory shape around schemas, controller handles, and runtime state.

@Dnreikronos
Dnreikronos force-pushed the connectors/opaque_transport_config branch 2 times, most recently from d2082e3 to dc3c74c Compare June 23, 2026 17:08
@Dnreikronos

Copy link
Copy Markdown
Contributor Author

small note on the registry key change: imo it belongs in this slice, not a later one.

with transport config becoming { name, config }, connector names are no longer limited to variants known by feldera-types. keeping the transport registries keyed by &'static str would still force runtime-discovered connectors to leak strings or predeclare static names somewhere. switching only the transport registries to own String keeps the same local pattern (BTreeMap, Arc<dyn factory>, get(&str)), but removes that static-name constraint.

built-ins behave the same. they still register string constants, and lookups still use borrowed &str. idk that preprocess/postprocess need the same treatment in this pr, since #6446 is about connector transports and changing those too would widen the diff for no immediate benefit.

@Dnreikronos
Dnreikronos marked this pull request as draft June 23, 2026 17:31
@Dnreikronos
Dnreikronos force-pushed the connectors/opaque_transport_config branch from 4086edf to dc3c74c Compare June 23, 2026 17:36
@Dnreikronos
Dnreikronos marked this pull request as ready for review June 23, 2026 20:57

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the real prize from the #6446 thread: TransportConfig is now { name: Cow<'static, str>, config: JsonValue }, factories own their parsing, feldera-types no longer has to know every transport at compile time, and the SQL-side connector validation still catches the obvious wrong-direction mistakes (null_output on an input relation, datagen on an output relation) while letting genuinely unknown names through. The backward-compatibility tests (transport_config_deserializes_existing_config_payload_shape, transport_config_deserializes_existing_unit_variant_shape, transport_config_serializes_unit_transports_without_config_field) pin the migration: existing {name, config} payloads still deserialize, no-config transports may omit config, and serialization skips config: null. +1063/-340 even after touching 20 files is a reasonable size for this kind of architectural extraction.

A few real points:

  • known_transport_direction vs sql_connector_transport_direction divergence (crates/pipeline-manager/src/db/types/program.rs). Two near-identical tables with one delta: the SQL-side rejects HTTP_INPUT / ADHOC_INPUT / CLOCK / HTTP_OUTPUT as "runtime-only" while known_transport_direction accepts them. That's exactly right semantically (you can't declare an ad-hoc connector in SQL CONNECTORS), but the two matches are easy to drift. A test asserts the SQL-rejection list today (generate_program_info_rejects_runtime_only_input_transports), but the relationship between the two functions — sql_connector_transport_direction(name).is_some() implies known_transport_direction(name).is_some() and the direction agrees — isn't pinned. A one-line property-style test that walks every TransportConfig::* constant would catch a future addition that forgets one of the tables. The maintenance hazard is real because the canonical list of names lives in feldera-types::config::TransportConfig and the SQL filter lives in pipeline-manager, so adding a new transport now requires three coordinated edits (constant + registry + SQL filter) instead of two.
  • The boolean logic sql_direction == Some(Output) || (known_transport_direction(name).is_some() && sql_direction.is_none()) is right but harder to read than it needs to be. The intent is "reject if the transport is known and is not a valid input transport for SQL declarations." A small helper — fn rejects_as_input(name) -> bool — would compress the two call sites into a single boolean and document the rule.
  • registered: BTreeMap<&'static str, ...> carryover from #6522. Same comment as on #6522: a third-party / dynamic factory can't easily inject a non-'static name today. With the { name: Cow<'static, str>, config } shape now stable, this is the last impedance-mismatch piece. Recommend widening to String (or at least Cow<'static, str>) before this lands so the dynamic-name path that motivates moving config parsing into factories is actually usable. Reading your own issue comment "small note on the registry key change: imo it belongs in this slice, not a later one" — agreed, fold it in.
  • TransportConfig::is_transient() still hard-codes the runtime-only list (ADHOC_INPUT | HTTP_INPUT | HTTP_OUTPUT | CLOCK). That's the third copy of the same partition — is_transient, known_transport_direction, sql_connector_transport_direction — and they'll drift over time. Consider centralizing as a TransportClass { Input, Output, Transient } lookup table on TransportConfig itself, then is_transient / direction queries derive from it. Out of scope for this PR if you want, but worth filing a follow-up.
  • integrated.rs removals look right at a glance, and the open-coded match for integrated connectors stays exactly where you'd expect (they still need schema/controller-handle wiring that's distinct from the factory shape). Worth a one-line comment in integrated.rs noting that integrated connectors are deliberately not in the registry because their factories need richer args.
  • secret_resolver.rs -19 lines. The simplification (using the opaque JSON shape) is nice. One thing to double-check: every transport-specific secret-resolution rule that lived in the old match arms is now handled in the factory or by the generic JSON walker, right? A regression here is silent (secret reference left unresolved → connector gets a literal ${secret:...} string and fails opaquely at runtime). Worth one integration test per direction (input + output) that uses a ${secret:...} reference and asserts the resolved value reaches the factory.

Direction is right. Direct the registry-key widening here and merge the two slices into a coherent story rather than carrying it into a third PR.

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every blocker from my earlier pass is addressed cleanly:

  • Drift guard between the two transport tables (Guard SQL transport direction drift): the new rejects_sql_connector_transport(name, expected_direction) helper centralizes the boolean logic at both call sites — same intent ("reject if the transport is known and is not a valid SQL connector in this direction"), now one source of truth. Plus builtin_transport_names() lists all 24 built-in transports and the new known_transport_direction_covers_* / sql_connector_transport_direction_is_subset_of_known property tests pin (a) known_transport_direction covers every TransportConfig::* constant and (b) sql_connector_transport_direction(name).is_some() implies known_transport_direction(name).is_some() with matching direction. Adding a new transport now fails CI at the table walk, not silently at runtime.
  • Registry key widening (folded into #6522): BTreeMap<String, ...> on both registries, register(impl Into<String>, ...), no behavior change for builtins.
  • Secret-resolution regression test (Test transport secret resolution before parsing): two new tests (input_transport_secrets_are_resolved_before_factory_parsing, output_transport_secrets_are_resolved_before_factory_parsing) register a custom factory that asserts config.config["path"] equals the resolved value, then route a ${secret:kubernetes:transport/...} reference through *_transport_config_to_endpoint_with_registry. Catches exactly the silent-failure mode I worried about (factory receives the unresolved ${secret:...} literal). Nice extraction of the *_with_registry shim so tests don't need to mutate the global builtin registry.
  • Integrated connector doc (Document integrated transport registry boundary): two-line comment on create_integrated_output_endpoint saying integrated connectors stay out of the registry because they need schema + controller lifecycle state at construction time. Future readers will not wonder.

is_transient() centralization is correctly deferred — I called it out as out-of-scope and the comment thread agrees. Worth a follow-up issue when you have a moment.

LGTM.

@blp
blp force-pushed the connectors/opaque_transport_config branch from 348e24e to 8f99d9d Compare June 24, 2026 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants