Skip to content

Commit bde2067

Browse files
ntkatholentkathole
committed
feat: Addresses performance issues in the Redis online store
Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>
1 parent f630056 commit bde2067

6 files changed

Lines changed: 979 additions & 28 deletions

File tree

‎docs/how-to-guides/online-server-performance-tuning.md‎

Lines changed: 121 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,10 @@ When the server processes a `get_online_features()` call, it groups the requeste
3131

3232
**Guideline:** For features that share the same entity key and are frequently requested together, consolidate them into a **single feature view**. This reduces the number of store round-trips per request. Split feature views only when features have different entities, different materialization schedules, or different data source update frequencies.
3333

34+
{% hint style="info" %}
35+
**Redis exception:** The Redis online store overrides `get_online_features()` to batch all `HMGET` commands across every feature view into a **single pipeline execution**. Because all feature views for the same entity share one Redis hash key, the number of Redis round trips is always **1**, regardless of how many feature views the request touches. This means the "fewer feature views" guideline is less critical for Redis than for other stores — but consolidating feature views still reduces serialization and protobuf overhead at the application layer.
36+
{% endhint %}
37+
3438
### Feature services are free
3539

3640
A [Feature Service](../getting-started/concepts/feature-retrieval.md) is a named collection of feature references — it's a convenience grouping, not a separate storage or execution unit. Using a feature service adds only a registry lookup (cached) compared to listing features individually. There is no performance penalty for using feature services, and they are the recommended way to define stable, versioned feature sets for production models.
@@ -229,7 +233,7 @@ The online store is the single largest factor in `get_online_features()` latency
229233

230234
| Store | Typical p50 latency | Async read | Best for | Key trade-off |
231235
| ----- | ------------------- | ---------- | -------- | ------------- |
232-
| **Redis / Dragonfly** | < 1 ms | No (threadpool) | Ultra-low latency, high throughput | Requires in-memory capacity for your dataset |
236+
| **Redis / Dragonfly** | < 1 ms | No (threadpool) | Ultra-low latency, high throughput; all FV reads batched into 1 pipeline | Requires in-memory capacity for your dataset |
233237
| **DynamoDB** | 2–5 ms | Yes | Serverless, auto-scaling on AWS | Pay-per-request cost; batch API limits (100 items) |
234238
| **PostgreSQL** | 3–10 ms | No (threadpool) | Teams with existing Postgres infra | Connection pooling needed at scale |
235239
| **MongoDB** | 2–5 ms | Yes | Flexible schema, async-native | Requires index tuning for large datasets |
@@ -259,7 +263,7 @@ The feature server can read from the online store using either an **async** or *
259263
| **DynamoDB** | Yes | Yes | Uses `aiobotocore` for non-blocking I/O |
260264
| **MongoDB** | Yes | Yes | Uses `motor` (async MongoDB driver) |
261265
| **PostgreSQL** | Implemented | No | Has `online_read_async` but does not yet advertise via `async_supported`; uses sync/threadpool path |
262-
| **Redis** | Implemented | No | Has `online_read_async` but does not yet advertise via `async_supported`; uses sync/threadpool path |
266+
| **Redis** | Implemented | **Yes** | `online_read_async` and `online_write_batch_async` both implemented; uses sync/threadpool path for `get_online_features` (overridden with batched single pipeline) |
263267
| All others | No | No | Fall back to sync with `run_in_threadpool()` |
264268

265269
**When async matters most:**
@@ -329,12 +333,57 @@ online_store:
329333
connection_string: "redis-cluster.internal:6379,ssl=true"
330334
redis_type: redis_cluster
331335
key_ttl_seconds: 604800
336+
skip_dedup: false # set true for initial bulk loads to halve write round trips
332337
```
333338

334339
- Use `redis_cluster` for horizontal partitioning across shards.
335-
- Set `key_ttl_seconds` to auto-expire stale feature data, keeping memory usage bounded.
340+
- Set `key_ttl_seconds` to auto-expire stale feature data, keeping memory usage bounded. This is a **key-level** TTL — it expires the entire entity hash (all feature views for that entity) together.
336341
- Ensure the Redis instance is in the **same availability zone** as the feature server pods to minimize network round-trips.
337342

343+
#### Batched multi-feature-view reads
344+
345+
The Redis online store overrides `get_online_features()` to issue all `HMGET` commands — across every feature view in the request — in a **single pipeline execution**. This reduces Redis round trips from `N` (one per feature view) to `1` regardless of request size.
346+
347+
| Feature views | Round trips (other stores) | Round trips (Redis) |
348+
| :---: | :---: | :---: |
349+
| 1 | 1 | 1 |
350+
| 5 | 5 | **1** |
351+
| 10 | 10 | **1** |
352+
| 20 | 20 | **1** |
353+
354+
This means the latency cost of adding feature views to a Redis-backed request is primarily **serialization and protobuf overhead** at the application layer, not Redis network latency.
355+
356+
#### Per-feature-view TTL at retrieval
357+
358+
Set a `ttl` on individual `FeatureView` objects to enforce staleness checks at read time. Features whose stored event timestamp is older than the TTL are returned as `NOT_FOUND` rather than stale values:
359+
360+
```python
361+
from datetime import timedelta
362+
363+
driver_stats = FeatureView(
364+
name="driver_stats",
365+
entities=["driver"],
366+
ttl=timedelta(hours=6), # features older than 6 hours returned as NOT_FOUND
367+
schema=[...],
368+
source=driver_stats_source,
369+
)
370+
```
371+
372+
This is the correct way to enforce per-feature-view TTL semantics. The `key_ttl_seconds` config option applies a Redis `EXPIRE` to the whole entity hash (all feature views share one expiry), so it cannot enforce different TTLs per feature view.
373+
374+
#### Write throughput: `skip_dedup`
375+
376+
For initial bulk loads or append-only materialization pipelines:
377+
378+
```yaml
379+
online_store:
380+
type: redis
381+
connection_string: "localhost:6379"
382+
skip_dedup: true
383+
```
384+
385+
With `skip_dedup: true`, `online_write_batch()` skips the existing-timestamp read pipeline and writes all values directly in a single pass, halving write round trips. Use only when you can guarantee write ordering — under concurrent writers, an older record can overwrite a newer one.
386+
338387
### Cassandra / ScyllaDB tuning
339388

340389
Cassandra and ScyllaDB share the same Feast connector. The key read-path knobs are concurrency and data-center-aware routing:
@@ -421,7 +470,7 @@ Different online stores have different optimal batch sizes for `get_online_featu
421470
| Store | Default batch size | Max batch size | Recommendation |
422471
| ----- | ------------------ | -------------- | -------------- |
423472
| DynamoDB | 100 | 100 (API limit) | Keep at 100; tune `max_read_workers` for parallelism |
424-
| Redis | N/A (pipelined) | N/A | Redis pipelines all keys in one round-trip; no batch tuning needed |
473+
| Redis | N/A (pipelined) | N/A | All entity keys **and** all feature views are batched into a single pipeline; no batch tuning needed |
425474
| PostgreSQL | N/A (single query) | N/A | Single `SELECT ... WHERE key IN (...)` query; tune connection pool instead |
426475

427476
Profile your workload by measuring `feast_feature_server_online_store_read_duration_seconds` (see [Metrics setup](#metrics-setup-prometheus--opentelemetry)) across different entity counts to find the sweet spot.
@@ -980,12 +1029,79 @@ registry:
9801029

9811030
---
9821031

1032+
## Materialization write performance
1033+
1034+
Materialization (`feast materialize` / `feast materialize-incremental`) reads features from the offline store and writes them to the online store. For large feature views, two common bottlenecks arise: **memory exhaustion** during proto conversion and **write throughput** to the online store.
1035+
1036+
### Memory: `online_write_batch_size`
1037+
1038+
By default, Feast converts the entire Arrow table returned by the offline store into Python protobuf objects in a single pass before writing to the online store. For datasets with millions of rows this can consume tens of gigabytes of memory on the materialization worker.
1039+
1040+
Set `online_write_batch_size` in `feature_store.yaml` to break the write into manageable chunks:
1041+
1042+
```yaml
1043+
materialization:
1044+
online_write_batch_size: 10000 # rows per write batch
1045+
```
1046+
1047+
Each chunk is independently converted and written, keeping peak memory proportional to the batch size rather than the full dataset. This is supported by the **local, Spark, and Ray** compute engines.
1048+
1049+
| Dataset size | Without batching | With `online_write_batch_size: 10000` |
1050+
| --- | --- | --- |
1051+
| 1 M rows (100 bytes/row) | ~100 MB peak | ~1 MB peak |
1052+
| 10 M rows | ~1 GB peak | ~1 MB peak |
1053+
| 100 M rows | OOM / swap | ~1 MB peak |
1054+
1055+
**Choosing a value:**
1056+
1057+
- **Larger batches** (50 000+): fewer write calls to the online store, lower overhead per row — good when worker memory allows.
1058+
- **Smaller batches** (1 000–5 000): lower peak memory — necessary for memory-constrained workers or very wide feature views (many features per row).
1059+
- For **Redis**: pipeline overhead per batch is negligible; a batch size of 10 000–50 000 is a good starting point.
1060+
- For **DynamoDB**: each batch maps to one or more `BatchWriteItem` calls (max 25 items per call); a larger `online_write_batch_size` amortizes the per-call overhead but doesn't change the 25-item DynamoDB limit.
1061+
1062+
See the [feature-store-yaml reference](../reference/feature-repository/feature-store-yaml.md#online_write_batch_size) for the complete option documentation.
1063+
1064+
### Throughput: parallel feature view materialization
1065+
1066+
Each feature view in a `feast materialize` call is materialized sequentially by the local engine. To materialize multiple feature views in parallel, use a job orchestrator (Airflow, Kubernetes Jobs) and materialize one feature view per job:
1067+
1068+
```bash
1069+
# Airflow / cron: one task per feature view
1070+
feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S") \
1071+
--views driver_stats
1072+
```
1073+
1074+
Alternatively, use the **Spark** or **Ray** compute engines which distribute the work across a cluster.
1075+
1076+
### Redis: combine with `skip_dedup` for bulk reloads
1077+
1078+
When performing a full historical reload into Redis (not an incremental update), combine `online_write_batch_size` with `skip_dedup` for maximum throughput:
1079+
1080+
```yaml
1081+
materialization:
1082+
online_write_batch_size: 50000 # large chunks — memory is bounded
1083+
1084+
online_store:
1085+
type: redis
1086+
connection_string: "redis-cluster.internal:6379"
1087+
skip_dedup: true # skip per-row timestamp check — halves write round trips
1088+
```
1089+
1090+
`skip_dedup: true` eliminates the timestamp-read pipeline before each write (see [Redis tuning](#redis-tuning)), while `online_write_batch_size` prevents the write worker from converting the entire dataset into memory at once.
1091+
1092+
{% hint style="warning" %}
1093+
Reset `skip_dedup` to `false` (or remove it) after the bulk reload. Under normal incremental materialization, deduplication prevents older feature values from overwriting newer ones.
1094+
{% endhint %}
1095+
1096+
---
1097+
9831098
## Further reading
9841099

9851100
- [Scaling Feast](./scaling-feast.md) — Horizontal scaling, HPA, KEDA, and HA in detail
9861101
- [Python Feature Server](../reference/feature-servers/python-feature-server.md) — CLI flags, metrics reference, and API endpoints
9871102
- [OpenTelemetry Integration](../getting-started/components/open-telemetry.md) — Full OTEL setup with Prometheus Operator
9881103
- [DynamoDB Online Store](../reference/online-stores/dynamodb.md) — Store-specific configuration and performance tuning
9891104
- [PostgreSQL Online Store](../reference/online-stores/postgres.md) — Connection pooling and SSL configuration
990-
- [Redis Online Store](../reference/online-stores/redis.md) — Cluster mode, Sentinel, and TTL configuration
1105+
- [Redis Online Store](../reference/online-stores/redis.md) — Cluster mode, Sentinel, TTL configuration, and batched reads
9911106
- [On Demand Feature Views](../reference/beta-on-demand-feature-view.md) — Transformation modes and write-time transforms
1107+
- [feature_store.yaml reference](../reference/feature-repository/feature-store-yaml.md) — Full configuration reference including `materialization` options

‎docs/how-to-guides/running-feast-in-production.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,15 @@ Feast keeps the history of materialization in its registry so that the choice co
7575

7676
However, the amount of work can quickly outgrow the resources of a single machine. That happens because the materialization job needs to repackage all rows before writing them to an online store. That leads to high utilization of CPU and memory. In this case, you might want to use a job orchestrator to run multiple jobs in parallel using several workers. Kubernetes Jobs or Airflow are good choices for more comprehensive job orchestration.
7777

78+
For large datasets, you can also reduce peak memory on the materialization worker by setting `online_write_batch_size` in `feature_store.yaml`. This breaks the proto conversion and write into chunks instead of loading the entire dataset into memory at once:
79+
80+
```yaml
81+
materialization:
82+
online_write_batch_size: 10000 # rows per write batch; reduces peak memory proportionally
83+
```
84+
85+
See the [Materialization write performance](./online-server-performance-tuning.md#materialization-write-performance) guide for sizing recommendations and the full option reference in [feature_store.yaml](../reference/feature-repository/feature-store-yaml.md#online_write_batch_size).
86+
7887
If you are using Airflow as a scheduler, Feast can be invoked through a [PythonOperator](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html) after the [Python SDK](https://pypi.org/project/feast/) has been installed into a virtual environment and your feature repo has been synced:
7988
8089
```python

‎docs/reference/feature-repository/feature-store-yaml.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,5 +25,69 @@ The following top-level configuration options exist in the `feature_store.yaml`
2525
* **offline_store** — Configures the offline store.
2626
* **project** — Defines a namespace for the entire feature store. Can be used to isolate multiple deployments in a single installation of Feast. Should only contain letters, numbers, and underscores.
2727
* **engine** - Configures the batch materialization engine.
28+
* **materialization** - Configures materialization behavior (write batching, feature pull strategy). See below.
2829

2930
Please see the [RepoConfig](https://rtd.feast.dev/en/latest/#feast.repo_config.RepoConfig) API reference for the full list of configuration options.
31+
32+
---
33+
34+
## `materialization` configuration
35+
36+
The `materialization` block controls how Feast reads from the offline store and writes to the online store during `feast materialize` / `feast materialize-incremental` runs.
37+
38+
{% code title="feature_store.yaml" %}
39+
```yaml
40+
project: my_feature_repo
41+
registry: data/registry.db
42+
provider: local
43+
online_store:
44+
type: redis
45+
connection_string: "localhost:6379"
46+
materialization:
47+
online_write_batch_size: 10000 # write rows in chunks of 10 000
48+
pull_latest_features: false # pull full time range (default)
49+
```
50+
{% endcode %}
51+
52+
### `online_write_batch_size`
53+
54+
| Field | Type | Default | Supported engines |
55+
| --- | --- | --- | --- |
56+
| `online_write_batch_size` | `int` (positive) | `null` | local, spark, ray |
57+
58+
Controls how many rows are converted to protobuf and written to the online store per batch during materialization.
59+
60+
**Default behaviour (`null`):** All rows fetched from the offline store are converted to protobuf in a single in-memory operation before writing. This is fast but can exhaust memory for large datasets — every row must be held as a Python proto object simultaneously.
61+
62+
**With `online_write_batch_size` set:** The Arrow table returned by the offline store is split into chunks of at most `online_write_batch_size` rows. Each chunk is converted and written independently, keeping peak memory proportional to the batch size rather than the full dataset size.
63+
64+
```yaml
65+
# Recommended for datasets > a few million rows or memory-constrained workers
66+
materialization:
67+
online_write_batch_size: 10000
68+
```
69+
70+
**Choosing a value:**
71+
72+
| Dataset size | Worker memory | Recommended batch size |
73+
| --- | --- | --- |
74+
| < 1 M rows | Any | `null` (default — single batch is fine) |
75+
| 1–10 M rows | ≥ 4 GB | `50000` |
76+
| 10–100 M rows | ≥ 8 GB | `10000` |
77+
| > 100 M rows | Any | `5000`–`10000` |
78+
79+
A smaller batch size reduces peak memory at the cost of more `online_write_batch` calls to the online store. For Redis, each call is a pipelined batch, so the overhead is low. For stores with higher per-call latency (e.g. DynamoDB), prefer larger batch sizes.
80+
81+
{% hint style="info" %}
82+
`online_write_batch_size` is applied **per feature view** within a single materialization job. If you materialize five feature views in parallel, peak memory is `5 × batch_size × bytes_per_row`.
83+
{% endhint %}
84+
85+
### `pull_latest_features`
86+
87+
| Field | Type | Default |
88+
| --- | --- | --- |
89+
| `pull_latest_features` | `bool` | `false` |
90+
91+
When `false` (default), the offline store retrieves **all** feature values within the requested time range for each entity.
92+
93+
When `true`, only the **latest** value per entity is retrieved. This reduces I/O and memory for feature views where historical values are not needed (e.g., slowly changing dimensions). It is equivalent to running a `GROUP BY entity, MAX(event_timestamp)` on the offline data before writing.

0 commit comments

Comments
 (0)