Skip to content

feat: Implement spark offline store offline_write_batch method #3076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 6 commits into from
Aug 18, 2022
Merged

feat: Implement spark offline store offline_write_batch method #3076

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
60b5881
create integration spark data sets as files rather than a temp table
niklasvm Aug 12, 2022
b8d3ee9
add offline_write_batch method to spark offline store
niklasvm Aug 12, 2022
0a3a060
remove some comments
niklasvm Aug 12, 2022
767a664
fix linting issue
niklasvm Aug 12, 2022
967a299
fix more linting issues
niklasvm Aug 12, 2022
3704f4a
fix flake8 errors
niklasvm Aug 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
fix flake8 errors 
Signed-off-by: niklasvm <niklasvm@gmail.com>
  • Loading branch information
@niklasvm
niklasvm committed Aug 17, 2022
commit 3704f4abc7342d656ec853be1d50b2031b6b3e04
5 changes: 2 additions & 3 deletions sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
from feast.infra.offline_stores import offline_utils
from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import (
SavedDatasetSparkStorage,
SparkOptions,
SparkSource,
)
from feast.infra.offline_stores.offline_store import (
Expand Down Expand Up @@ -247,11 +246,11 @@ def offline_write_batch(
).save(feature_view.batch_source.path)
elif feature_view.batch_source.query:
raise NotImplementedError(
f"offline_write_batch not implemented for batch sources specified by query"
"offline_write_batch not implemented for batch sources specified by query"
)
else:
raise NotImplementedError(
f"offline_write_batch not implemented for batch sources specified by a table"
"offline_write_batch not implemented for batch sources specified by a table"
)

@staticmethod
Expand Down