Skip to content

feat: Implement spark materialization engine #3184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 19 commits into from
Sep 15, 2022
Merged

feat: Implement spark materialization engine #3184

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
26e0b3e
implement spark materialization engine
niklasvm Sep 5, 2022
807e7ca
remove redundant code
niklasvm Sep 5, 2022
f8e70ea
make function private
niklasvm Sep 5, 2022
b42352e
refactor serializing into a class
niklasvm Sep 5, 2022
f609cfb
switch to using `foreachPartition`
niklasvm Sep 5, 2022
79ea412
remove batch_size parameter
niklasvm Sep 5, 2022
937a0e3
add partitions parameter
niklasvm Sep 7, 2022
610614a
linting
niklasvm Sep 7, 2022
84cc858
rename spark to spark.offline and spark.engine
niklasvm Sep 7, 2022
8cc4928
fix to test
niklasvm Sep 7, 2022
6a9663d
forgot to stage
niklasvm Sep 7, 2022
715bb72
revert spark.offline to spark to ensure backward compatibility
niklasvm Sep 8, 2022
de1a85a
Merge branch 'master' into spark_materialization_engine
niklasvm Sep 8, 2022
32b3111
fix import
niklasvm Sep 8, 2022
0c13af9
remove code from testing a large data set
niklasvm Sep 8, 2022
542705f
linting
niklasvm Sep 8, 2022
262af10
test without repartition
niklasvm Sep 13, 2022
8e59da2
test alternate connection string
niklasvm Sep 13, 2022
f828d2b
use redis online creator
niklasvm Sep 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
rename spark to spark.offline and spark.engine 
Signed-off-by: niklasvm <niklasvm@gmail.com>
  • Loading branch information
@niklasvm
niklasvm committed Sep 7, 2022
commit 84cc858b02188575eb27bf372c87dab1450033eb
2 changes: 1 addition & 1 deletion sdk/python/feast/infra/materialization/contrib/spark/spark_materialization_engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
class SparkMaterializationEngineConfig(FeastConfigBaseModel):
Copy link
Copy Markdown
Collaborator

@adchia adchia Sep 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably makes sense to throw an error somewhere if the offline store is not the SparkOfflineStore?

"""Batch Materialization Engine config for spark engine"""

type: Literal["spark"] = "spark"
type: Literal["spark.engine"] = "spark.engine"
""" Type selector"""

partitions: int = 0
Expand Down
6 changes: 3 additions & 3 deletions sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@


class SparkOfflineStoreConfig(FeastConfigBaseModel):
type: StrictStr = "spark"
type: StrictStr = "spark.offline"
Comment thread
niklasvm marked this conversation as resolved.
Outdated
""" Offline store type selector"""

spark_conf: Optional[Dict[str, str]] = None
Expand All @@ -49,7 +49,7 @@ class SparkOfflineStoreConfig(FeastConfigBaseModel):

class SparkOfflineStore(OfflineStore):
@staticmethod
@log_exceptions_and_usage(offline_store="spark")
@log_exceptions_and_usage(offline_store="spark.offline")
def pull_latest_from_table_or_query(
config: RepoConfig,
data_source: DataSource,
Expand Down Expand Up @@ -247,7 +247,7 @@ def offline_write_batch(
)

@staticmethod
@log_exceptions_and_usage(offline_store="spark")
@log_exceptions_and_usage(offline_store="spark.offline")
def pull_all_from_table_or_query(
config: RepoConfig,
data_source: DataSource,
Expand Down
2 changes: 1 addition & 1 deletion sdk/python/tests/integration/materialization/contrib/spark/test_spark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ def test_spark_materialization_consistency():
# "path": "data/online_store.db"
},
offline_store_creator=SparkDataSourceCreator,
batch_engine={"type": "spark", "partitions": 100},
batch_engine={"type": "spark.engine", "partitions": 100},
)
spark_environment = construct_test_environment(
spark_config, None, entity_key_serialization_version=1
Expand Down