Flesh out module 2

adchia · adchia · commit cad91c0b5360 · 2022-05-17T12:50:14.000-04:00
Signed-off-by: Danny Chiao &lt;danny@tecton.ai&gt;
diff --git a/module_0/feature_repo_aws/feature_services.py b/module_0/feature_repo_aws/feature_services.py
@@ -8,6 +8,6 @@
     owner="test3@gmail.com",
 )
 
-feature_service = FeatureService(
+feature_service_2 = FeatureService(
     name="model_v2", features=[driver_hourly_stats_view], owner="test3@gmail.com",
 )
diff --git a/module_1/feature_repo/feature_services.py b/module_1/feature_repo/feature_services.py
@@ -8,6 +8,6 @@
     owner="test3@gmail.com",
 )
 
-feature_service = FeatureService(
+feature_service_2 = FeatureService(
     name="model_v2", features=[driver_hourly_stats_view], owner="test3@gmail.com",
 )
diff --git a/module_2/README.md b/module_2/README.md
@@ -1,30 +1,77 @@
-# Module 2: On demand transformations
-TODO
+<h1>Module 2: On demand transformations</h1>
+
+In this module, we introduce the concept of on demand transforms. These are transformations that execute on-the-fly and accept as input other feature views or request data.
 
+TODO:
+- add architecture
 - Define request data
 - Define on demand transforms
   - Note that this can also transforms pushed features (e.g. stream features)
   - Note that this can combine multiple feature views and request data
 
+<h2>Table of Contents</h2>
 
-<h1>Module 2: On demand transformations</h1>
+- [Workshop](#workshop)
+  - [Step 1: Install Feast](#step-1-install-feast)
+  - [Step 2: Look at the data we have](#step-2-look-at-the-data-we-have)
+  - [Step 3: Apply features](#step-3-apply-features)
+  - [Step 3: Materialize batch features](#step-3-materialize-batch-features)
+  - [Step 4: Test retrieve features](#step-4-test-retrieve-features)
+- [Conclusion](#conclusion)
+  
+# Workshop
+## Step 1: Install Feast
 
-In this module, we introduce the concept of on demand transforms. These are transformations that execute on-the-fly and accept as input other feature views or request data.
+First, we install Feast as well as a Geohash module we want to use:
+```bash
+pip install feast
+pip install pygeohash
+```
 
-We  and focus on building features for online serving, and keeping them fresh with a combination of batch feature materialization and stream feature ingestion. We'll be roughly working towards the following:
+## Step 2: Look at the data we have
+We used `data/gen_lat_lon.py` to append randomly generated latitude and longitudes to the original driver stats dataset.
 
-- **Data sources**: Kafka + File source
-- **Online store**: Redis
-- **Use case**: Predicting churn for drivers in real time.
+```python
+import pandas as pd
+pd.read_parquet("data/driver_stats_lat_lon.parquet")
+```
 
-<img src="architecture.png" width=750>
+![](data.png)
 
-<h2>Table of Contents</h2>
+## Step 3: Apply features
+```console
+$ feast apply
 
-# Workshop
-## Step 1: Install Feast
+Created entity driver
+Created feature view driver_daily_features
+Created feature view driver_hourly_stats
+Created on demand feature view transformed_conv_rate
+Created on demand feature view avg_hourly_miles_driven
+Created on demand feature view location_features_from_push
+Created feature service model_v3
+Created feature service model_v2
+Created feature service model_v1
 
-First, we install Feast with Spark and Redis support:
-```bash
-pip install "feast[spark,redis]"
-```
+Created sqlite table feast_demo_odfv_driver_daily_features
+Created sqlite table feast_demo_odfv_driver_hourly_stats
+```
+
+## Step 3: Materialize batch features
+```console
+$ feast materialize-incremental $(date +%Y-%m-%d)
+
+Materializing 2 feature views to 2022-05-17 12:41:18-04:00 into the sqlite online store.
+
+driver_hourly_stats from 1748-08-01 16:41:20-04:56:02 to 2022-05-17 12:41:18-04:00:
+100%|████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 495.03it/s]
+driver_daily_features from 1748-08-01 16:41:20-04:56:02 to 2022-05-17 12:41:18-04:00:
+100%|███████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1274.48it/s]
+```
+
+## Step 4: Test retrieve features 
+Now we'll see how these transformations are executed offline at `get_historical_features` and online at `get_online_features` time. We'll also see how `OnDemandFeatureView` interacts with request data, regular feature views, and streaming / push features.
+
+Try out the Jupyter notebook in [client/module_2_client.ipynb](client/module_2_client.ipynb). This is in a separate directory that contains just a `feature_store.yaml`.
+
+# Conclusion
+TODO
diff --git a/module_2/client/test_fetch.py b/module_2/client/test_fetch.py
diff --git a/module_2/feature_repo/data_sources.py b/module_2/feature_repo/data_sources.py
@@ -8,7 +8,7 @@
 
 driver_stats = FileSource(
     name="driver_stats_source",
-    path="../data/driver_stats_lat_lon.parquet",  # Should be a remote path in reality for re-use
+    path="../data/driver_stats_lat_lon.parquet",
     timestamp_field="event_timestamp",
     created_timestamp_column="created",
     description="A table describing the stats of a driver based on hourly logs",
@@ -24,10 +24,14 @@
 # available at request time (e.g. part of the user initiated HTTP request)
 driver_request = RequestSource(
     name="driver_request",
+    schema=[Field(name="lat", dtype=Float32), Field(name="lon", dtype=Float32),],
+)
+
+
+val_to_add_request = RequestSource(
+    name="vals_to_add",
     schema=[
         Field(name="val_to_add", dtype=Int64),
         Field(name="val_to_add_2", dtype=Int64),
-        Field(name="lat", dtype=Float32),
-        Field(name="lon", dtype=Float32),
     ],
 )
diff --git a/module_2/feature_repo/feature_services.py b/module_2/feature_repo/feature_services.py
@@ -8,25 +8,15 @@
     owner="test3@gmail.com",
 )
 
-feature_service = FeatureService(
+feature_service_2 = FeatureService(
     name="model_v2",
-    features=[
-        driver_hourly_stats_view,
-        driver_daily_features_view,
-        transformed_conv_rate,
-    ],
+    features=[driver_hourly_stats_view[["conv_rate"]], transformed_conv_rate,],
     owner="test3@gmail.com",
 )
 
-feature_service = FeatureService(
+feature_service_3 = FeatureService(
     name="model_v3",
-    features=[
-        driver_hourly_stats_view,
-        driver_daily_features_view,
-        transformed_conv_rate,
-        avg_hourly_miles_driven,
-        location_features,
-    ],
+    features=[driver_daily_features_view, location_features_from_push,],
     owner="test3@gmail.com",
 )
 
diff --git a/module_2/feature_repo/features.py b/module_2/feature_repo/features.py
@@ -32,7 +32,11 @@
     name="driver_daily_features",
     entities=["driver"],
     ttl=timedelta(seconds=8640000000),
-    schema=[Field(name="daily_miles_driven", dtype=Float32),],
+    schema=[
+        Field(name="daily_miles_driven", dtype=Float32),
+        Field(name="lat", dtype=Float32),
+        Field(name="lon", dtype=Float32),
+    ],
     online=True,
     source=driver_stats_push_source,
     tags={"production": "True"},
@@ -43,7 +47,7 @@
 # Define an on demand feature view which can generate new features based on
 # existing feature views and RequestSource features
 @on_demand_feature_view(
-    sources=[driver_hourly_stats_view, driver_request],
+    sources=[driver_hourly_stats_view, val_to_add_request],
     schema=[
         Field(name="conv_rate_plus_val1", dtype=Float64),
         Field(name="conv_rate_plus_val2", dtype=Float64),
@@ -67,14 +71,17 @@ def avg_hourly_miles_driven(inputs: pd.DataFrame) -> pd.DataFrame:
 
 
 @on_demand_feature_view(
-    sources=[driver_daily_features_view, driver_request],
+    sources=[driver_daily_features_view],
     schema=[Field(name=f"geohash_{i}", dtype=String) for i in range(1, 7)],
 )
-def location_features(inputs: pd.DataFrame) -> pd.DataFrame:
+def location_features_from_push(inputs: pd.DataFrame) -> pd.DataFrame:
     import pygeohash as gh
 
     df = pd.DataFrame()
-    geohash = df.apply(lambda x: gh.encode(x.lat, x.lon), axis=1)
+    df["geohash"] = inputs.apply(lambda x: gh.encode(x.lat, x.lon), axis=1).astype(
+        "string"
+    )
+
     for i in range(1, 7):
-        df[f"geohash_{i}"] = geohash.str[:i]
+        df[f"geohash_{i}"] = df["geohash"].str[:i].astype("string")
     return df

Original file line number	Diff line number	Diff line change
`@@ -8,6 +8,6 @@`
`8`	`8`	`owner="test3@gmail.com",`
`9`	`9`	`)`
`10`	`10`
`11`		`-feature_service = FeatureService(`
	`11`	`+feature_service_2 = FeatureService(`
`12`	`12`	`name="model_v2", features=[driver_hourly_stats_view], owner="test3@gmail.com",`
`13`	`13`	`)`