Today the connectors are loaded using a function like transport.input_transport_config_to_endpoint which has a big match statement:
match config {
TransportConfig::FileInput(config) => Box::new(FileInputEndpoint::new(config)),
#[cfg(feature = "with-kafka")]
TransportConfig::KafkaInput(config) => Box::new(KafkaFtInputEndpoint::new(config)?),
#[cfg(not(feature = "with-kafka"))]
TransportConfig::KafkaInput(_) => return Ok(None),
#[cfg(feature = "with-nats")]
...
This means that adding new connectors types requires recompiling this file and requires a platform update.
preprocessors however are implemented differently: there's a dynamic map from string to implementation which is looked up. preprocessor factories are registered dynamically in this map and looked up by name.
I think a similar structure could be used for connectors as well. Configuration parsing would be moved into the factory.
This step may be a prerequisite for eventually linking connectors dynamically.
Today the connectors are loaded using a function like
transport.input_transport_config_to_endpointwhich has a big match statement:This means that adding new connectors types requires recompiling this file and requires a platform update.
preprocessors however are implemented differently: there's a dynamic map from string to implementation which is looked up. preprocessor factories are registered dynamically in this map and looked up by name.
I think a similar structure could be used for connectors as well. Configuration parsing would be moved into the factory.
This step may be a prerequisite for eventually linking connectors dynamically.