Skip to content

Implement JobService API calls & connect it to SDK - #1129

Merged
woop merged 13 commits into
feast-dev:masterfrom
tsotnet:jobservice-impl
Nov 4, 2020
Merged

Implement JobService API calls & connect it to SDK#1129
woop merged 13 commits into
feast-dev:masterfrom
tsotnet:jobservice-impl

Conversation

@tsotnet

@tsotnet tsotnet commented Nov 4, 2020

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:

  • Implement start_offline_to_online_ingestion, start_historical_feature_retrieval_job, start_stream_to_online_ingestion SDK calls with JobService (including branching logic depending on whether job service is enabled or not)
  • Set correct environment for JobService (Helm Charts & Dockerfile)
  • Update JobService protos as necessary (according to Slack discussion)
  • Add e2e tests for JobService

Does this PR introduce a user-facing change?:

Users can now use Feast Job Service as a proxy from Python SDK to get historical features, start batch ingestion, or start streaming ingestion jobs. When running the corresponding SDK methods from the Python client, the actual logic will run on Job Service. Also, listing jobs and getting a job by job_id will go through Job Service.

Job Service can be enabled by setting FEAST_JOB_SERVICE_URL environment variable.

Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
…ne mode

* New API calls (start_offline_to_online_ingestion, start_stream_to_online_ingestion) now return Remote Jobs instead of job ids
* Implement list_jobs & get_job for standalone mode (looks like Spark is running in local mode and we can't get job statuses so we have to keep cache in memory)
* Wire up list_jobs & get_job on client side with job service
* Tested locally on Feast 101 notebook, everything works

Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
…mode

Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
@feast-ci-bot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tsotnet
To complete the pull request process, please assign zhilingc
You can assign the PR to them by writing /assign @zhilingc in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
@woop

woop commented Nov 4, 2020

Copy link
Copy Markdown
Member

/kind feature

@feast-ci-bot feast-ci-bot added kind/feature New feature or request and removed needs-kind labels Nov 4, 2020
@tsotnet

tsotnet commented Nov 4, 2020

Copy link
Copy Markdown
Collaborator Author

/test "complete / tests-docker-compose (pull_request)"

@woop
woop merged commit 4068b3e into feast-dev:master Nov 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/feature New feature or request size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants