Skip to content

Use JobManager's backend as persistent storage and source of truth - #903

Merged
feast-ci-bot merged 10 commits into
feast-dev:masterfrom
pyalex:jc-dataflow-as-database
Aug 11, 2020
Merged

Use JobManager's backend as persistent storage and source of truth#903
feast-ci-bot merged 10 commits into
feast-dev:masterfrom
pyalex:jc-dataflow-as-database

Conversation

@pyalex

@pyalex pyalex commented Jul 29, 2020

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:

As part of the plan to decouple JobCoordinator & Core and eventually move Job Management to dedicated service this PR is the first step in this direction.

JobCoordinator doesn't rely on any persistent storage anymore. We store temporary state in memory (see InMemoryJobRepository) and fetch this state each time application is being started (see JobManager.listJobs).

For DataflowJobManager it means that we fetch list of jobs directly from Dataflow. We distinguish jobs that are being managed by current application by job's labels (see property feast.jobs.coordinator.jobSelector). Same set of labels is being applied to job on creation.
For DirectJobManager we just start with empty job list.

Refactoring:

  1. JobServiceTest was replaced with JobServiceIT

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

API:
1. listIngestionJobs now returns IngestionJob with list of FeatureSetReference instead of list of FeatureSet

Properties (core):
feast:
  jobs:
    coordinator:
      # moved here from feast.jobs
      consolidate-jobs-per-source: false
      
      # labels that being applied to new jobs and used to filter existing jobs
      jobSelector:
        application: feast

@pyalex

pyalex commented Jul 29, 2020

Copy link
Copy Markdown
Collaborator Author

/hold

Comment thread core/pom.xml Outdated
Comment thread core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java Outdated
Comment thread core/src/main/java/feast/core/config/FeastProperties.java Outdated
Comment thread core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java Outdated
@pyalex
pyalex force-pushed the jc-dataflow-as-database branch from a18e468 to 2cd87a1 Compare August 2, 2020 04:17
@pyalex

pyalex commented Aug 2, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end

fixes after rebase

fix python tests
@pyalex
pyalex force-pushed the jc-dataflow-as-database branch from 1a6c17e to 3532dcf Compare August 2, 2020 07:10
@pyalex

pyalex commented Aug 2, 2020

Copy link
Copy Markdown
Collaborator Author

/unhold

@pyalex

pyalex commented Aug 3, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-batch

@pyalex

pyalex commented Aug 3, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-redis-cluster

Comment thread core/src/main/java/feast/core/config/FeastProperties.java
Comment thread core/src/main/java/feast/core/job/InMemoryJobRepository.java
Comment thread core/src/main/java/feast/core/job/InMemoryJobRepository.java
Comment thread core/src/main/java/feast/core/job/JobManager.java
Comment thread core/src/main/java/feast/core/job/task/RestartJobTask.java Outdated
Comment thread core/src/main/java/feast/core/model/FeatureSetDeliveryStatus.java
Comment thread core/src/main/java/feast/core/service/JobService.java Outdated
Comment thread core/src/main/resources/application.yml
@pyalex

pyalex commented Aug 11, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-auth

@woop

woop commented Aug 11, 2020

Copy link
Copy Markdown
Member

/lgtm

@feast-ci-bot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pyalex, woop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@feast-ci-bot
feast-ci-bot merged commit 51f4fb6 into feast-dev:master Aug 11, 2020
@mrzzy mrzzy added the compat/breaking Breaking user-facing change label Sep 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants