Skip to content

Feast SDK integration for historical feature retrieval using Spark - #1054

Merged
feast-ci-bot merged 3 commits into
feast-dev:masterfrom
khorshuheng:sdk-historical-retrieval
Oct 15, 2020
Merged

Feast SDK integration for historical feature retrieval using Spark#1054
feast-ci-bot merged 3 commits into
feast-dev:masterfrom
khorshuheng:sdk-historical-retrieval

Conversation

@khorshuheng

@khorshuheng khorshuheng commented Oct 15, 2020

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:
Add historical feature retrieval job back to the Feast client again.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

oavdeev
oavdeev reviewed Oct 15, 2020
View reviewed changes
Comment thread sdk/python/feast/pyspark/launcher.py Outdated
Comment thread sdk/python/feast/pyspark/launcher.py Outdated

@oavdeev oavdeev Oct 15, 2020

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not 100% follow what is this for? I'd expect this to be within the local launcher as it assumes that you have Spark env setup.

@khorshuheng khorshuheng Oct 15, 2020

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is useful for cases where, the user has their own SparkSession, in a jupyter notebook for example. In such case, it might be more convenient to return a Spark dataframe, rather than the actual output.

@khorshuheng
khorshuheng force-pushed the sdk-historical-retrieval branch from 245f49a to 2ba5ab9 Compare October 15, 2020 08:00
@khorshuheng
Feast SDK integration for historical feature retrieval using Spark
27deb7b 
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
@khorshuheng
khorshuheng force-pushed the sdk-historical-retrieval branch 3 times, most recently from 8df72d2 to 9c4f1cc Compare October 15, 2020 08:27
@khorshuheng
Downgrade pyspark dependencies, add more tests
a739121 
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
@khorshuheng
khorshuheng force-pushed the sdk-historical-retrieval branch from 9c4f1cc to a739121 Compare October 15, 2020 08:28
@khorshuheng
Don't expose the historical feature output config directly to the user
987234b 
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
@khorshuheng khorshuheng changed the title (WIP) Feast SDK integration for historical feature retrieval using Spark Feast SDK integration for historical feature retrieval using Spark Oct 15, 2020
@khorshuheng khorshuheng added the kind/feature New feature or request label Oct 15, 2020
@feast-ci-bot

feast-ci-bot commented Oct 15, 2020

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: khorshuheng, pyalex

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@pyalex

pyalex commented Oct 15, 2020

Copy link
Copy Markdown
Collaborator

/lgtm

@feast-ci-bot
feast-ci-bot merged commit e8b24bb into feast-dev:master Oct 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@oavdeev oavdeev oavdeev left review comments
@pyalex pyalex pyalex approved these changes
@davidheryanto davidheryanto Awaiting requested review from davidheryanto
@woop woop Awaiting requested review from woop
@zhilingc zhilingc Awaiting requested review from zhilingc

Assignees

@pyalex pyalex

Labels

approved kind/feature New feature or request lgtm size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@khorshuheng @feast-ci-bot @pyalex @oavdeev