Skip to content

Commit f3b522b

Browse files
sfc-gh-madkinssfc-gh-madkins
authored
feat: Add Snowflake materialization engine (#2948)
Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>
1 parent 83c5efb commit f3b522b

File tree

25 files changed

+1376
-102
lines changed

25 files changed

+1376
-102
lines changed

‎.github/workflows/pr_local_integration_tests.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,4 +75,4 @@ jobs:
7575
IS_TEST: "True"
7676
FEAST_LOCAL_ONLINE_CONTAINER: "True"
7777
FEAST_IS_LOCAL_TEST: "True"
78-
run: pytest -n 8 --cov=./ --cov-report=xml --color=yes --integration -k "not gcs_registry and not s3_registry and not test_lambda_materialization" sdk/python/tests
78+
run: pytest -n 8 --cov=./ --cov-report=xml --color=yes --integration -k "not gcs_registry and not s3_registry and not test_lambda_materialization and not test_snowflake_materialization" sdk/python/tests

‎docs/reference/batch-materialization/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,6 @@
22

33
Please see [Batch Materialization Engine](../../getting-started/architecture-and-components/batch-materialization-engine.md) for an explanation of batch materialization engines.
44

5+
{% page-ref page="snowflake.md" %}
6+
57
{% page-ref page="bytewax.md" %}

‎docs/reference/batch-materialization/snowflake.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Snowflake
2+
3+
## Description
4+
5+
The [Snowflake](https://trial.snowflake.com) batch materialization engine provides a highly scalable and parallel execution engine using a Snowflake Warehouse for batch materializations operations (`materialize` and `materialize-incremental`) when using a `SnowflakeSource`.
6+
7+
The engine requires no additional configuration other than for you to supply Snowflake's standard login and context details. The engine leverages custom (automatically deployed for you) Python UDFs to do the proper serialization of your offline store data to your online serving tables.
8+
9+
When using all three options together, `snowflake.offline`, `snowflake.engine`, and `snowflake.online`, you get the most unique experience of unlimited scale and performance + governance and data security.
10+
11+
## Example
12+
13+
{% code title="feature_store.yaml" %}
14+
```yaml
15+
...
16+
offline_store:
17+
type: snowflake.offline
18+
...
19+
batch_engine:
20+
type: snowflake.engine
21+
account: snowflake_deployment.us-east-1
22+
user: user_login
23+
password: user_password
24+
role: sysadmin
25+
warehouse: demo_wh
26+
database: FEAST
27+
```
28+
{% endcode %}

‎docs/reference/offline-stores/snowflake.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ Below is a matrix indicating which functionality is supported by `SnowflakeRetri
4646
| export to dataframe | yes |
4747
| export to arrow table | yes |
4848
| export to arrow batches | no |
49-
| export to SQL | no |
49+
| export to SQL | yes |
5050
| export to data lake (S3, GCS, etc.) | yes |
5151
| export to data warehouse | yes |
5252
| export as Spark dataframe | no |

‎docs/tutorials/tutorials-overview/driver-stats-on-snowflake.md‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ description: >-
66
# Drivers stats on Snowflake
77

88
In the steps below, we will set up a sample Feast project that leverages Snowflake
9-
as an offline store.
9+
as an offline store + materialization engine + online store.
1010

1111
Starting with data in a Snowflake table, we will register that table to the feature store and define features associated with the columns in that table. From there, we will generate historical training data based on those feature definitions and then materialize the latest feature values into the online store. Lastly, we will retrieve the materialized feature values.
1212

@@ -46,7 +46,7 @@ The following files will automatically be created in your project folder:
4646

4747
#### Inspect `feature_store.yaml`
4848

49-
Here you will see the information that you entered. This template will use Snowflake as an offline store and SQLite as the online store. The main thing to remember is by default, Snowflake objects have ALL CAPS names unless lower case was specified.
49+
Here you will see the information that you entered. This template will use Snowflake as the offline store, materialization engine, and the online store. The main thing to remember is by default, Snowflake objects have ALL CAPS names unless lower case was specified.
5050

5151
{% code title="feature_store.yaml" %}
5252
```yaml
@@ -61,6 +61,14 @@ offline_store:
6161
role: ROLE_NAME #case sensitive
6262
warehouse: WAREHOUSE_NAME #case sensitive
6363
database: DATABASE_NAME #case cap sensitive
64+
batch_engine:
65+
type: snowflake.engine
66+
account: SNOWFLAKE_DEPLOYMENT_URL #drop .snowflakecomputing.com
67+
user: USERNAME
68+
password: PASSWORD
69+
role: ROLE_NAME #case sensitive
70+
warehouse: WAREHOUSE_NAME #case sensitive
71+
database: DATABASE_NAME #case cap sensitive
6472
online_store:
6573
type: snowflake.online
6674
account: SNOWFLAKE_DEPLOYMENT_URL #drop .snowflakecomputing.com

‎sdk/python/docs/source/feast.infra.materialization.rst‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,14 @@ feast.infra.materialization.local\_engine module
2828
:undoc-members:
2929
:show-inheritance:
3030

31+
feast.infra.materialization.snowflake\_engine module
32+
----------------------------------------------------
33+
34+
.. automodule:: feast.infra.materialization.snowflake_engine
35+
:members:
36+
:undoc-members:
37+
:show-inheritance:
38+
3139
Module contents
3240
---------------
3341

‎sdk/python/docs/source/feast.infra.utils.rst‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -28,14 +28,6 @@ feast.infra.utils.hbase\_utils module
2828
:undoc-members:
2929
:show-inheritance:
3030

31-
feast.infra.utils.snowflake\_utils module
32-
-----------------------------------------
33-
34-
.. automodule:: feast.infra.utils.snowflake_utils
35-
:members:
36-
:undoc-members:
37-
:show-inheritance:
38-
3931
Module contents
4032
---------------
4133

0 commit comments

Comments
 (0)