Explain Spark feature

adchia · adchia · commit a5e75b4cf2b2 · 2022-05-18T15:28:10.000-04:00
Signed-off-by: Danny Chiao &lt;danny@tecton.ai&gt;
diff --git a/module_1/feature_repo/module_1.ipynb b/module_1/feature_repo/module_1.ipynb
@@ -440,7 +440,14 @@
     "- These features can then be further post-processed and combined with other features or request data in on demand transforms.\n",
     "- An example might be to push in the last 5 transactions, and in on demand transforms generate the average of those transactions.\n",
     "\n",
-    "Feast will help manage both batch and streaming sources for you. You can run `feast materialize-incremental` as well as ingest streaming features to the same online store."
+    "Feast will help manage both batch and streaming sources for you. You can run `feast materialize-incremental` as well as ingest streaming features to the same online store.\n",
+    "\n",
+    "**Below, what's happening:**\n",
+    "- We use Spark to compute a sliding window aggregate feature that computes `daily_miles_driven` using the `miles_driven` column in the event.\n",
+    "- Triggers every 30 seconds\n",
+    "    - In this case, because we’re just reading from the `driver_stats.parquet`, we could have multiple windows of data coming in. Thus, in this code, we filter for the latest (`driver_id`, `window`) feature\n",
+    "    - If you have a larger watermark and can get events across multiple windows, you’ll want to have the latest window too.\n",
+    "- Rename `end` to `event_timestamp`, otherwise Feast will throw a validation error since it doesn’t match the schema of the `FeatureView`"
    ]
   },
   {
