Skip to content

feat: Implement spark materialization engine #3184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 19 commits into from
Sep 15, 2022
Merged

feat: Implement spark materialization engine #3184

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
26e0b3e
implement spark materialization engine
niklasvm Sep 5, 2022
807e7ca
remove redundant code
niklasvm Sep 5, 2022
f8e70ea
make function private
niklasvm Sep 5, 2022
b42352e
refactor serializing into a class
niklasvm Sep 5, 2022
f609cfb
switch to using `foreachPartition`
niklasvm Sep 5, 2022
79ea412
remove batch_size parameter
niklasvm Sep 5, 2022
937a0e3
add partitions parameter
niklasvm Sep 7, 2022
610614a
linting
niklasvm Sep 7, 2022
84cc858
rename spark to spark.offline and spark.engine
niklasvm Sep 7, 2022
8cc4928
fix to test
niklasvm Sep 7, 2022
6a9663d
forgot to stage
niklasvm Sep 7, 2022
715bb72
revert spark.offline to spark to ensure backward compatibility
niklasvm Sep 8, 2022
de1a85a
Merge branch 'master' into spark_materialization_engine
niklasvm Sep 8, 2022
32b3111
fix import
niklasvm Sep 8, 2022
0c13af9
remove code from testing a large data set
niklasvm Sep 8, 2022
542705f
linting
niklasvm Sep 8, 2022
262af10
test without repartition
niklasvm Sep 13, 2022
8e59da2
test alternate connection string
niklasvm Sep 13, 2022
f828d2b
use redis online creator
niklasvm Sep 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add partitions parameter 
Signed-off-by: niklasvm <niklasvm@gmail.com>
  • Loading branch information
@niklasvm
niklasvm committed Sep 7, 2022
commit 937a0e380a3e950c70c0d252c9819e645afc5d04
8 changes: 8 additions & 0 deletions sdk/python/feast/infra/materialization/contrib/spark/spark_materialization_engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ class SparkMaterializationEngineConfig(FeastConfigBaseModel):
type: Literal["spark"] = "spark"
""" Type selector"""

partitions: int = None
"""Number of partitions to use when writing data to online store"""


@dataclass
class SparkMaterializationJob(MaterializationJob):
Expand Down Expand Up @@ -170,6 +173,11 @@ def _materialize_one(
)

spark_df = offline_job.to_spark_df()
if self.repo_config.batch_engine.partitions:
spark_df = spark_df.repartition(
self.repo_config.batch_engine.partitions
)

spark_df.foreachPartition(
lambda x: _process_by_partition(x, spark_serialized_artifacts)
)
Comment on lines +175 to +183
Copy link
Copy Markdown
Member

@achals achals Sep 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

Expand Down
2 changes: 1 addition & 1 deletion sdk/python/tests/integration/materialization/contrib/spark/test_spark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ def test_spark_materialization_consistency():
# "path": "data/online_store.db"
},
offline_store_creator=SparkDataSourceCreator,
batch_engine={"type": "spark"},
batch_engine={"type": "spark", "partitions": 100},
)
spark_environment = construct_test_environment(
spark_config, None, entity_key_serialization_version=1
Expand Down