Skip to content

feat: Add Snowflake online store (again) #2922

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 18 commits into from
Jul 29, 2022
Merged

feat: Add Snowflake online store (again) #2922

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
14fd591
feat: Add snowflake online store
sfc-gh-madkins Jul 2, 2022
40507ef
lint/format
sfc-gh-madkins Jul 2, 2022
6c76b5f
removing missing testing env variables
sfc-gh-madkins Jul 2, 2022
6933e34
test offline store first
sfc-gh-madkins Jul 2, 2022
98e3a5c
snowflake online test fixes
sfc-gh-madkins Jul 2, 2022
41ec30d
format
sfc-gh-madkins Jul 2, 2022
49286d0
fix snowflake testing (#2903)
sfc-gh-madkins Jul 2, 2022
fb534ec
change to transient table
sfc-gh-madkins Jul 11, 2022
1ddb4ee
remove extra lines
sfc-gh-madkins Jul 11, 2022
56feaea
docs + nits
sfc-gh-madkins Jul 27, 2022
c38fd29
docs + nits2
sfc-gh-madkins Jul 27, 2022
2309d4e
final docs + nits
sfc-gh-madkins Jul 28, 2022
7a03683
Update sdk/python/feast/infra/online_stores/snowflake.py
sfc-gh-madkins Jul 28, 2022
e1bda4f
Update sdk/python/feast/infra/online_stores/snowflake.py
sfc-gh-madkins Jul 28, 2022
cce1402
Update sdk/python/feast/infra/online_stores/snowflake.py
sfc-gh-madkins Jul 28, 2022
48ae284
Update sdk/python/feast/infra/online_stores/snowflake.py
sfc-gh-madkins Jul 28, 2022
53aa4f7
fix serialization
sfc-gh-madkins Jul 28, 2022
d7d0c04
fix serialization2
sfc-gh-madkins Jul 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
final docs + nits 
Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>
  • Loading branch information
@sfc-gh-madkins
sfc-gh-madkins committed Jul 28, 2022
commit 2309d4eeb5a856ad04fbca1b9161a4dac1be34e9
12 changes: 11 additions & 1 deletion docs/reference/online-stores/snowflake.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,20 @@

## Description

The [Snowflake](https://trial.snowflake.com) online store provides support for materializing feature values into an Snowflake Transient Table for serving online features.
The [Snowflake](https://trial.snowflake.com) online store provides support for materializing feature values into a Snowflake Transient Table for serving online features.

* Only the latest feature values are persisted

The data model for using a Snowflake Transient Table as an online store follows a tall format (one row per feature)):
* "entity_feature_key" (BINARY) -- unique key used when reading specific feature_view x entity combination
* "entity_key" (BINARY) -- repeated key currently unused for reading entity_combination
* "feature_name" (VARCHAR)
* "value" (BINARY)
* "event_ts" (TIMESTAMP)
* "created_ts" (TIMESTAMP)

(This model may be subject to change when Snowflake Hybrid Tables are released)

## Example

{% code title="feature_store.yaml" %}
Expand Down
27 changes: 11 additions & 16 deletions sdk/python/feast/templates/snowflake/bootstrap.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ def bootstrap():
from feast.driver_test_data import create_driver_hourly_stats_df

repo_path = pathlib.Path(__file__).parent.absolute()
config_file = repo_path / "feature_store.yaml"

project_name = str(repo_path)[str(repo_path).rfind("/") + 1 :]

Expand All @@ -23,7 +22,6 @@ def bootstrap():
driver_entities = [1001, 1002, 1003, 1004, 1005]
driver_df = create_driver_hourly_stats_df(driver_entities, start_date, end_date)

repo_path = pathlib.Path(__file__).parent.absolute()
data_path = repo_path / "data"
data_path.mkdir(exist_ok=True)
driver_stats_path = data_path / "driver_stats.parquet"
Expand All @@ -38,6 +36,17 @@ def bootstrap():
snowflake_warehouse = click.prompt("Snowflake Warehouse Name (Case Sensitive):")
snowflake_database = click.prompt("Snowflake Database Name (Case Sensitive):")

config_file = repo_path / "feature_store.yaml"
for i in range(2):
replace_str_in_file(
config_file, "SNOWFLAKE_DEPLOYMENT_URL", snowflake_deployment_url
)
replace_str_in_file(config_file, "SNOWFLAKE_USER", snowflake_user)
replace_str_in_file(config_file, "SNOWFLAKE_PASSWORD", snowflake_password)
replace_str_in_file(config_file, "SNOWFLAKE_ROLE", snowflake_role)
replace_str_in_file(config_file, "SNOWFLAKE_WAREHOUSE", snowflake_warehouse)
replace_str_in_file(config_file, "SNOWFLAKE_DATABASE", snowflake_database)

if click.confirm(
f'Should I upload example data to Snowflake (overwriting "{project_name}_feast_driver_hourly_stats" table)?',
default=True,
Expand Down Expand Up @@ -66,20 +75,6 @@ def bootstrap():
)
conn.close()

repo_path = pathlib.Path(__file__).parent.absolute()
config_file = repo_path / "feature_store.yaml"
driver_file = repo_path / "driver_repo.py"
replace_str_in_file(
config_file, "SNOWFLAKE_DEPLOYMENT_URL", snowflake_deployment_url
)
replace_str_in_file(config_file, "SNOWFLAKE_USER", snowflake_user)
replace_str_in_file(config_file, "SNOWFLAKE_PASSWORD", snowflake_password)
replace_str_in_file(config_file, "SNOWFLAKE_ROLE", snowflake_role)
replace_str_in_file(config_file, "SNOWFLAKE_WAREHOUSE", snowflake_warehouse)
replace_str_in_file(config_file, "SNOWFLAKE_DATABASE", snowflake_database)

replace_str_in_file(driver_file, "SNOWFLAKE_WAREHOUSE", snowflake_warehouse)


def replace_str_in_file(file_path, match_str, sub_str):
with open(file_path, "r") as f:
Expand Down
1 change: 0 additions & 1 deletion sdk/python/feast/templates/snowflake/driver_repo.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@
# The Snowflake table where features can be found
database=yaml.safe_load(open("feature_store.yaml"))["offline_store"]["database"],
table=f"{project_name}_feast_driver_hourly_stats",
warehouse="SNOWFLAKE_WAREHOUSE",
# The event timestamp is used for point-in-time joins and for ensuring only
# features within the TTL are returned
timestamp_field="event_timestamp",
Expand Down