You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We then use Docker Compose to spin up a local Kafka cluster and automatically publish events to it.
23
23
- This leverages a script (in `kafka_demo/`) that creates a topic, reads from `feature_repo/data/driver_stats.parquet`, generates newer timestamps, and emits them to the topic.
24
+
- This also deploys a Feast push server + a Feast feature server. The Dockerfile mostly delegates to calling the `feast serve` CLI command:
# Needed to reach online store within Docker network.
33
+
RUN sed -i 's/localhost:6379/redis:6379/g' feature_store.yaml
34
+
ENV FEAST_USAGE=False
35
+
36
+
CMD ["feast", "serve", "-h", "0.0.0.0"]
37
+
```
24
38
25
39
Start up the Docker daemon and then use Docker Compose to spin up the services as described above:
26
40
```console
@@ -47,9 +61,44 @@ This will guide you through:
47
61
- Materializing feature view values from batch sources to the online store (e.g. Redis).
48
62
- Ingesting feature view values from streaming sources (e.g. window aggregate features from Spark + Kafka)
49
63
- Retrieve features at low latency from Redis through Feast.
64
+
- Working with a Feast push server + feature server to ingest and retrieve features through HTTP endpoints (instead of needing `feature_store.yaml` and `FeatureStore` instances)
65
+
66
+
### A note on Feast feature servers + push servers
67
+
The above notebook introduces a way to curl an HTTP endpoint to push or retrieve features from Redis.
68
+
69
+
The servers by default cache the registry (expiring and reloading every 10 minutes). If you want to customize that time period, you can do so in `feature_store.yaml`.
70
+
71
+
Let's look at the `feature_store.yaml` used in this module (which configures the registry differently than in the previous module):
72
+
73
+
```yaml
74
+
project: feast_demo_local
75
+
provider: local
76
+
registry:
77
+
path: data/local_registry.db
78
+
cache_ttl_seconds: 5
79
+
online_store:
80
+
type: redis
81
+
connection_string: localhost:6379
82
+
offline_store:
83
+
type: file
84
+
```
85
+
86
+
The `registry` config maps to constructor arguments for `RegistryConfig` Pydantic model([reference](https://rtd.feast.dev/en/master/index.html#feast.repo_config.RegistryConfig)).
87
+
- In the `feature_store.yaml` above, note that there is a `cache_ttl_seconds` of 5. This ensures that every five seconds, the feature server and push server will expire its registry cache. On the following request, it will refresh its registry by pulling from the registry path.
88
+
- Feast adds a convenience wrapper though so if you specify just `registry: [path]`, Feast will map that to `RegistryConfig(path=[your path])`.
89
+
90
+
## FAQ
91
+
92
+
### Can feature / push servers refresh their registry in response to an event? e.g. after a PR merges and `feast apply` is run?
93
+
Unfortunately, currently the servers don't support this. Feel free to contribute a PR though to enable this! The tricky part here is that Feast would need to keep track of these servers in the registry (or in some other way), which is not the way Feast is currently designed.
94
+
95
+
### How do I scale up materialization?
96
+
Materialization in Feast by default pulls the latest feature values for each unique entity locally and writes in batches to the online store.
97
+
98
+
- Feast users can materialize multiple feature views by using the CLI:
- **Caveat**: By default, Feast's registry store is a single protobuf written to a file. This means that there's the chance that metadata around materialization intervals gets lost if the registry has changed during materialization.
101
+
- The community is ideating on how to improve this. See [RFC-035: Scalable Materialization](https://docs.google.com/document/d/1tCZzClj3H8CfhJzccCytWK-bNDw_lkZk4e3fUbPYIP0/edit#)
102
+
- Users often also implement their own custom providers. The provider interface has a `materialize_single_feature_view` method, which users are free to implement differently (e.g. materializing with Spark or Dataflow jobs).
50
103
51
-
## Step 3: Setting up push servers + feature servers
52
-
TODO
53
-
- Have some feature servers be read-only (for `/get_online_features`)
54
-
- (pre-alpha): with `go_feature_retrieval=True` and a `LoggingConfig`, the feature server will also log served features to the offline store.
55
-
- Have some feature servers be write-only (for `/push`)
104
+
Feast is actively investigating ways to improve this
0 commit comments