Skip to content

[connectors] Connectors should use a dynamic registry #6446

Description

@mihaibudiu

Today the connectors are loaded using a function like transport.input_transport_config_to_endpoint which has a big match statement:

match config {
        TransportConfig::FileInput(config) => Box::new(FileInputEndpoint::new(config)),
        #[cfg(feature = "with-kafka")]
        TransportConfig::KafkaInput(config) => Box::new(KafkaFtInputEndpoint::new(config)?),
        #[cfg(not(feature = "with-kafka"))]
        TransportConfig::KafkaInput(_) => return Ok(None),
        #[cfg(feature = "with-nats")]
       ...

This means that adding new connectors types requires recompiling this file and requires a platform update.
preprocessors however are implemented differently: there's a dynamic map from string to implementation which is looked up. preprocessor factories are registered dynamically in this map and looked up by name.

I think a similar structure could be used for connectors as well. Configuration parsing would be moved into the factory.

This step may be a prerequisite for eventually linking connectors dynamically.

Metadata

Metadata

Assignees

No one assigned

    Labels

    connectorsIssues related to the adapters/connectors crate

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions