Skip to content

What is the need for using unique Kafka Consumer Group for ingestion job? #646

Description

@bharathelangovan

We are trying to figure out why a new consumer group is created when an ingestion job is started. This means that if there is a gap between the termination and starting of a Dataflow job there will be a data within the Kafka topic that is not ingested
https://github.com/gojek/feast/blob/196f7112f0c4ba58f2417e0c9856dac07435e0be/ingestion/src/main/java/feast/ingestion/transform/ReadFromSource.java#L86
We would like to know if this was intentional and if so what are the ways to avoid data loss?

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/questionFurther information is requested

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions