Skip to content

Run JobCoontroller as separate application#951

Merged
feast-ci-bot merged 26 commits into
feast-dev:masterfrom
pyalex:move-jc
Aug 21, 2020
Merged

Run JobCoontroller as separate application#951
feast-ci-bot merged 26 commits into
feast-dev:masterfrom
pyalex:move-jc

Conversation

@pyalex

@pyalex pyalex commented Aug 19, 2020 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:

This is final PR in series of decoupling JobCoordinator (renamed to JobController) from Core.
Main features of PR:

  1. There's new module job-controller that should be run as standalone (spring) application (helm charts, docker compose, e2e tests updated)
  2. This module communicates with Core through GRPC API. Although job-controller use core in test scope.
  3. Ingestion related methods were moved out of CoreService API into JobController API (see CoreService.proto). Since it could be deployed with several instances of JobController - routing between them is now client's responsibility. Also, client code (in python SDK) calling those methods were moved out of Feast Client. Now it's in contrib (see feast.contrib.job_controller.client in python SDK).

Refactoring:

  1. Common code for integration tests moved to feast-common-test module

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

Dedicated spring application job-controller
feast.jobs properties moved from core to job-controller

[Deprecated] CoreService.ListIngestJobs, CoreService.RestartIngestJob, CoreService.StopIngestJob

[New API] JobControllerService.ListIngestJobs, JobControllerService.RestartIngestJob, JobControllerService.ListIngestJobs.StopIngestJob

[Python SDK] feast.client.Client.list_ingest_jobs -> feast.contrib.job_controller.client.Client.list_ingest_jobs
[Python SDK] feast.client.Client.restart_ingest_job -> feast.contrib.job_controller.client.Client.restart_ingest_job
[Python SDK] feast.client.Client.stop_ingest_job -> feast.contrib.job_controller.client.Client.stop_ingest_job

@pyalex
infra
e161eee 
fix simple client
@pyalex

pyalex commented Aug 19, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-batch-dataflow

@pyalex

pyalex commented Aug 20, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-redis-cluster

@pyalex

pyalex commented Aug 20, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end

@pyalex

pyalex commented Aug 20, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-redis-cluster

woop
woop reviewed Aug 20, 2020
View reviewed changes
Comment thread common-test/src/main/java/feast/common/it/SimpleJCClient.java Outdated
woop
woop reviewed Aug 20, 2020
View reviewed changes
Comment thread common/src/main/java/feast/common/models/FeatureSetReference.java
@pyalex

pyalex commented Aug 20, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-redis-cluster

@pyalex

pyalex commented Aug 20, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-auth
/test test-end-to-end

@pyalex pyalex requested a review from woop August 20, 2020 10:20
@pyalex

pyalex commented Aug 20, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-auth

woop
woop reviewed Aug 20, 2020
View reviewed changes
Comment thread job-controller/src/main/java/feast/jobcontroller/grpc/CoreServiceImpl.java Outdated
woop
woop reviewed Aug 20, 2020
View reviewed changes
Comment thread job-controller/src/main/java/feast/jobcontroller/service/JobControllerService.java Outdated
woop
woop reviewed Aug 20, 2020
View reviewed changes
Comment thread job-controller/src/test/java/feast/jobcontroller/service/JobControllerIT.java Outdated
@pyalex

pyalex commented Aug 21, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-batch

1 similar comment
@pyalex

pyalex commented Aug 21, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-batch

@woop

woop commented Aug 21, 2020

Copy link
Copy Markdown
Member

/lgtm

@pyalex

pyalex commented Aug 21, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-batch

3 similar comments
@pyalex

pyalex commented Aug 21, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-batch

@pyalex

pyalex commented Aug 21, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-batch

@pyalex

pyalex commented Aug 21, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-batch

@feast-ci-bot

feast-ci-bot commented Aug 21, 2020

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pyalex, woop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@feast-ci-bot feast-ci-bot merged commit 16bb358 into feast-dev:master Aug 21, 2020
@mrzzy mrzzy mentioned this pull request Sep 6, 2020
Closed
@mrzzy mrzzy added the compat/breaking Breaking user-facing change label Sep 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@woop woop woop approved these changes

@davidheryanto davidheryanto Awaiting requested review from davidheryanto

@khorshuheng khorshuheng Awaiting requested review from khorshuheng

@zhilingc zhilingc Awaiting requested review from zhilingc

Assignees

@woop woop

Labels

approved compat/breaking Breaking user-facing change kind/housekeeping lgtm size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@pyalex @woop @feast-ci-bot @mrzzy