Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
The ingestion sources are currently mingled in with all the other ingestion code in the transforms package. This makes it hard to distinguish what all the sources are and makes it trickier for people to see how they might extend feast to support new sources.
Describe the solution you'd like
I'd like to create a source package in the ingestion module, and create common factory type class that can create FeatureIO.Read instances and have a function for returning their type string. New source implementations need only implement this interface and use an @AutoService annotation to make it available.
The import job can then load all available source factories from the class path and find the one with the matching type string against the import spec.
This follows the same pattern we use for storage types and allows us a path for including new source types as external dependencies, or moving existing ones into their own module. And then making them available simply by adding their jars on the class path.
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
The ingestion sources are currently mingled in with all the other ingestion code in the
transformspackage. This makes it hard to distinguish what all the sources are and makes it trickier for people to see how they might extend feast to support new sources.Describe the solution you'd like
I'd like to create a
sourcepackage in the ingestion module, and create common factory type class that can create FeatureIO.Read instances and have a function for returning their type string. New source implementations need only implement this interface and use an@AutoServiceannotation to make it available.The import job can then load all available source factories from the class path and find the one with the matching type string against the import spec.
This follows the same pattern we use for storage types and allows us a path for including new source types as external dependencies, or moving existing ones into their own module. And then making them available simply by adding their jars on the class path.