Skip to content

Optional IngestionJob parameters passed by Spark Launcher - #1130

Merged
pyalex merged 22 commits into
feast-dev:masterfrom
pyalex:optional-job-options
Nov 4, 2020
Merged

Optional IngestionJob parameters passed by Spark Launcher#1130
pyalex merged 22 commits into
feast-dev:masterfrom
pyalex:optional-job-options

Conversation

@pyalex

@pyalex pyalex commented Nov 4, 2020 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Signed-off-by: Oleksii Moskalenko moskalenko.alexey@gmail.com

What this PR does / why we need it:

  1. SDK / JobService should be able to configure statsd, deadletterPath and stencilURL optional parameters of IngestionJobs.

  2. Non-default project passed to Job via SDK / JobService

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

@pyalex pyalex changed the title Optional IngestionJob parameters passed by SDK Optional IngestionJob parameters passed by Spark Launcher Nov 4, 2020
@pyalex

pyalex commented Nov 4, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-gcp

pyalex added 4 commits November 4, 2020 13:19
@pyalex
optional job parameters
6c7a404 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
default values
15b750b 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
default values
ba1481e 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
add job service to docker compose
5c2e162 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
pyalex force-pushed the optional-job-options branch from ff95b5a to 5c2e162 Compare November 4, 2020 05:38
@pyalex
remove jobcontroller from makefile
f93289b 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
woop
woop reviewed Nov 4, 2020
View reviewed changes
Comment thread sdk/python/feast/constants.py Outdated
@pyalex
pass project to job
6ef85a7 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
woop
woop reviewed Nov 4, 2020
View reviewed changes
Comment thread sdk/python/feast/constants.py Outdated
pyalex added 10 commits November 4, 2020 14:39
@pyalex
helm update: jobservice secrets
ecb1337 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
run feast cli directly
3f00e19 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
fix jobservice host name
a87e4ce 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
statsd enabled
e26a167 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
shared directory
e0d3e54 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
override staging path in local env
4ca5beb 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
override staging path in local env
3aa6e03 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
debug
b6d20b3 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
use root
ef24730 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
cleanup
2f694ee 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
pyalex added 2 commits November 4, 2020 17:01
@pyalex
ingestion & jobcontroller module deleted from pom
d985845 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
ingestion & jobcontroller module deleted from pom
a6a1894 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex

pyalex commented Nov 4, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-gcp

woop
woop reviewed Nov 4, 2020
View reviewed changes
Comment thread infra/docker-compose/docker-compose.yml
FEAST_SPARK_STAGING_LOCATION: file:///shared/staging
FEAST_HISTORICAL_FEATURE_OUTPUT_LOCATION: file:///shared/historical_feature_output
FEAST_HISTORICAL_FEATURE_OUTPUT_FORMAT: parquet
FEAST_REDIS_HOST: redis

@woop woop Nov 4, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a port?

@pyalex pyalex Nov 4, 2020

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FEAST_REDIS_PORT is a separate options, and its default value 6379 was not changed in this case. So I believe it was omitted to save 1 line)

woop
woop reviewed Nov 4, 2020
View reviewed changes
Comment thread infra/docker/jobservice/Dockerfile
chmod +x /usr/bin/grpc-health-probe

CMD ["feast", "server"]
CMD ["python", "-m", "feast.cli", "server"]

@woop woop Nov 4, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for this change?

@pyalex pyalex Nov 4, 2020

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

container was failing with

pkg_resources.DistributionNotFound: The 'Click==7.*' distribution was not found and is required by feast

It could be due to any reason, since it's conda.

woop
woop reviewed Nov 4, 2020
View reviewed changes
Comment thread sdk/python/feast/client.py Outdated
woop
woop reviewed Nov 4, 2020
View reviewed changes
Comment thread sdk/python/feast/job_service.py
@pyalex
--with-job-service
fe14419 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex

pyalex commented Nov 4, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end

@pyalex
make feast_jobservice fixture scope - session
a375d8e 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@feast-ci-bot

feast-ci-bot commented Nov 4, 2020

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pyalex, woop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@woop

woop commented Nov 4, 2020

Copy link
Copy Markdown
Member

/lgtm

@pyalex

pyalex commented Nov 4, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-gcp

@pyalex
local jar path
934e359 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@feast-ci-bot feast-ci-bot removed the lgtm label Nov 4, 2020
@feast-ci-bot

feast-ci-bot commented Nov 4, 2020

Copy link
Copy Markdown
Collaborator

New changes are detected. LGTM label has been removed.

@pyalex
stage only local files
e5903d0 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex

pyalex commented Nov 4, 2020

Copy link
Copy Markdown
Collaborator Author

/test test-end-to-end-aws

@feast-ci-bot

feast-ci-bot commented Nov 4, 2020 •
edited
Loading

Copy link
Copy Markdown
Collaborator

@pyalex: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
test-end-to-end-aws e5903d0 link /test test-end-to-end-aws

Full PR test history

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@pyalex
pyalex merged commit fdb4dea into feast-dev:master Nov 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@woop woop woop approved these changes
@davidheryanto davidheryanto Awaiting requested review from davidheryanto
@khorshuheng khorshuheng Awaiting requested review from khorshuheng
@zhilingc zhilingc Awaiting requested review from zhilingc

Assignees

@woop woop

Labels

approved kind/housekeeping size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pyalex @feast-ci-bot @woop