Add configurable prefix to Consumer Group in IngestionJob's Kafka reader - #969
Conversation
|
/retest |
fe37fa6 to
cde8bbe
Compare
|
/retest |
1 similar comment
|
/retest |
| # Enabling JobManagement | ||
| enabled: true | ||
|
|
||
| # Prefix for JobId |
There was a problem hiding this comment.
This comment says absolutely nothing.
| source.getKafkaSourceConfig().getBootstrapServers(), | ||
| source.getKafkaSourceConfig().getTopic()), | ||
| dateSuffix); | ||
| if (!this.jobProperties.getJobIdPrefix().isEmpty()) { |
There was a problem hiding this comment.
What happens when jobProperties is null?
| feastProperties.getJobs().getController().getConsolidateJobsPerSource(); | ||
| if (shouldConsolidateJobs) { | ||
| return new ConsolidatedJobStrategy(jobRepository); | ||
| return new ConsolidatedJobStrategy(jobRepository, feastProperties.getJobs()); |
There was a problem hiding this comment.
Can we extract the jobProperties before passing it? It's a convention that I am trying to get us to follow, even if it takes more lines of code
There was a problem hiding this comment.
Extracted before passing jobProperties.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: pyalex, terryyylim The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/lgtm |
What this PR does / why we need it:
To allow multiple deployments with jobs running in parallel, we should separate their Kafka consumer groups which is based on the JobId. This would prevent overwriting each other's offsets.
Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?: