We are trying to figure out why a new consumer group is created when an ingestion job is started. This means that if there is a gap between the termination and starting of a Dataflow job there will be a data within the Kafka topic that is not ingested
https://github.com/gojek/feast/blob/196f7112f0c4ba58f2417e0c9856dac07435e0be/ingestion/src/main/java/feast/ingestion/transform/ReadFromSource.java#L86
We would like to know if this was intentional and if so what are the ways to avoid data loss?
We are trying to figure out why a new consumer group is created when an ingestion job is started. This means that if there is a gap between the termination and starting of a Dataflow job there will be a data within the Kafka topic that is not ingested
https://github.com/gojek/feast/blob/196f7112f0c4ba58f2417e0c9856dac07435e0be/ingestion/src/main/java/feast/ingestion/transform/ReadFromSource.java#L86
We would like to know if this was intentional and if so what are the ways to avoid data loss?