Skip to content

Commit 219d04a

Browse files
felixwang9817adchia
felixwang9817
and
adchia
authored
docs: Functionality matrix for online stores (feast-dev#3007)
* Cleanup online store docs Signed-off-by: Felix Wang <wangfelix98@gmail.com> * Add overview page Signed-off-by: Felix Wang <wangfelix98@gmail.com> * Add Snowflake online store to RTD Signed-off-by: Felix Wang <wangfelix98@gmail.com> * Online store docs Signed-off-by: Felix Wang <wangfelix98@gmail.com> * add overiew to README in online store Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Danny Chiao <danny@tecton.ai> Co-authored-by: Danny Chiao <danny@tecton.ai>
1 parent 41cb476 commit 219d04a

File tree

13 files changed

+279
-14
lines changed

13 files changed

+279
-14
lines changed

‎docs/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@
8686
* [Trino (contrib)](reference/offline-stores/trino.md)
8787
* [Azure Synapse + Azure SQL (contrib)](reference/offline-stores/mssql.md)
8888
* [Online stores](reference/online-stores/README.md)
89+
* [Overview](reference/online-stores/overview.md)
8990
* [SQLite](reference/online-stores/sqlite.md)
9091
* [Snowflake](reference/online-stores/snowflake.md)
9192
* [Redis](reference/online-stores/redis.md)

‎docs/getting-started/architecture-and-components/online-store.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,18 @@
11
# Online store
22

3-
The Feast online store is used for low-latency online feature value lookups. Feature values are loaded into the online store from data sources in feature views using the `materialize` command.
3+
Feast uses online stores to serve features at low latency.
4+
Feature values are loaded from data sources into the online store through _materialization_, which can be triggered through the `materialize` command.
45

5-
The storage schema of features within the online store mirrors that of the data source used to populate the online store. One key difference between the online store and data sources is that only the latest feature values are stored per entity key. No historical values are stored.
6+
The storage schema of features within the online store mirrors that of the original data source.
7+
One key difference is that for each [entity key](../concepts/entity.md), only the latest feature values are stored.
8+
No historical values are stored.
69

7-
Example batch data source
10+
Here is an example batch data source:
811

912
![](../../.gitbook/assets/image%20%286%29.png)
1013

11-
Once the above data source is materialized into Feast \(using `feast materialize`\), the feature values will be stored as follows:
14+
Once the above data source is materialized into Feast (using `feast materialize`), the feature values will be stored as follows:
1215

1316
![](../../.gitbook/assets/image%20%285%29.png)
1417

15-
Features can also be written to the online store via [push sources](../../reference/data-sources/push.md)
18+
Features can also be written directly to the online store via [push sources](../../reference/data-sources/push.md) .

‎docs/reference/online-stores/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@
22

33
Please see [Online Store](../../getting-started/architecture-and-components/online-store.md) for an explanation of online stores.
44

5+
{% content-ref url="overview.md" %}
6+
[overview.md](overview.md)
7+
{% endcontent-ref %}
8+
59
{% content-ref url="sqlite.md" %}
610
[sqlite.md](sqlite.md)
711
{% endcontent-ref %}

‎docs/reference/online-stores/cassandra.md‎

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,34 @@ online_store:
5555
```
5656
{% endcode %}
5757

58+
The full set of configuration options is available in [CassandraOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.contrib.cassandra_online_store.cassandra_online_store.CassandraOnlineStoreConfig).
5859
For a full explanation of configuration options please look at file
5960
`sdk/python/feast/infra/online_stores/contrib/cassandra_online_store/README.md`.
6061

61-
Storage specifications can be found at `docs/specs/online_store_format.md`.
62+
Storage specifications can be found at `docs/specs/online_store_format.md`.
63+
64+
## Functionality Matrix
65+
66+
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
67+
Below is a matrix indicating which functionality is supported by the Cassandra online store.
68+
69+
| | Cassandra |
70+
| :-------------------------------------------------------- | :-- |
71+
| write feature values to the online store | yes |
72+
| read feature values from the online store | yes |
73+
| update infrastructure (e.g. tables) in the online store | yes |
74+
| teardown infrastructure (e.g. tables) in the online store | yes |
75+
| generate a plan of infrastructure changes | yes |
76+
| support for on-demand transforms | yes |
77+
| readable by Python SDK | yes |
78+
| readable by Java | no |
79+
| readable by Go | no |
80+
| support for entityless feature views | yes |
81+
| support for concurrent writing to the same key | no |
82+
| support for ttl (time to live) at retrieval | no |
83+
| support for deleting expired data | no |
84+
| collocated by feature view | yes |
85+
| collocated by feature service | no |
86+
| collocated by entity key | no |
87+
88+
To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).

‎docs/reference/online-stores/datastore.md‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,4 +18,30 @@ online_store:
1818
```
1919
{% endcode %}
2020
21-
Configuration options are available [here](https://rtd.feast.dev/en/latest/#feast.repo_config.DatastoreOnlineStoreConfig).
21+
The full set of configuration options is available in [DatastoreOnlineStoreConfig](https://rtd.feast.dev/en/latest/#feast.infra.online_stores.datastore.DatastoreOnlineStoreConfig).
22+
23+
## Functionality Matrix
24+
25+
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
26+
Below is a matrix indicating which functionality is supported by the Datastore online store.
27+
28+
| | Datastore |
29+
| :-------------------------------------------------------- | :-- |
30+
| write feature values to the online store | yes |
31+
| read feature values from the online store | yes |
32+
| update infrastructure (e.g. tables) in the online store | yes |
33+
| teardown infrastructure (e.g. tables) in the online store | yes |
34+
| generate a plan of infrastructure changes | no |
35+
| support for on-demand transforms | yes |
36+
| readable by Python SDK | yes |
37+
| readable by Java | no |
38+
| readable by Go | no |
39+
| support for entityless feature views | yes |
40+
| support for concurrent writing to the same key | no |
41+
| support for ttl (time to live) at retrieval | no |
42+
| support for deleting expired data | no |
43+
| collocated by feature view | yes |
44+
| collocated by feature service | no |
45+
| collocated by entity key | no |
46+
47+
To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).

‎docs/reference/online-stores/dynamodb.md‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ online_store:
1717
```
1818
{% endcode %}
1919
20-
Configuration options are available [here](https://github.com/feast-dev/feast/blob/17bfa6118d6658d2bff53d7de8e2ccef5681714d/sdk/python/feast/infra/online_stores/dynamodb.py#L36).
20+
The full set of configuration options is available in [DynamoDBOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.dynamodb.DynamoDBOnlineStoreConfig).
2121
2222
## Permissions
2323
@@ -53,3 +53,29 @@ The following inline policy can be used to grant Feast the necessary permissions
5353
```
5454

5555
Lastly, this IAM role needs to be associated with the desired Redshift cluster. Please follow the official AWS guide for the necessary steps [here](https://docs.aws.amazon.com/redshift/latest/dg/c-getting-started-using-spectrum-add-role.html).
56+
57+
## Functionality Matrix
58+
59+
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
60+
Below is a matrix indicating which functionality is supported by the DynamoDB online store.
61+
62+
| | DynamoDB |
63+
| :-------------------------------------------------------- | :-- |
64+
| write feature values to the online store | yes |
65+
| read feature values from the online store | yes |
66+
| update infrastructure (e.g. tables) in the online store | yes |
67+
| teardown infrastructure (e.g. tables) in the online store | yes |
68+
| generate a plan of infrastructure changes | no |
69+
| support for on-demand transforms | yes |
70+
| readable by Python SDK | yes |
71+
| readable by Java | no |
72+
| readable by Go | no |
73+
| support for entityless feature views | yes |
74+
| support for concurrent writing to the same key | no |
75+
| support for ttl (time to live) at retrieval | no |
76+
| support for deleting expired data | no |
77+
| collocated by feature view | yes |
78+
| collocated by feature service | no |
79+
| collocated by entity key | no |
80+
81+
To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).

‎docs/reference/online-stores/overview.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Overview
2+
3+
## Functionality
4+
5+
Here are the methods exposed by the `OnlineStore` interface, along with the core functionality supported by the method:
6+
* `online_write_batch`: write feature values to the online store
7+
* `online_read`: read feature values from the online store
8+
* `update`: update infrastructure (e.g. tables) in the online store
9+
* `teardown`: teardown infrastructure (e.g. tables) in the online store
10+
* `plan`: generate a plan of infrastructure changes based on feature repo changes
11+
12+
There is also additional functionality not properly captured by these interface methods:
13+
* support for on-demand transforms
14+
* readable by Python SDK
15+
* readable by Java
16+
* readable by Go
17+
* support for entityless feature views
18+
* support for concurrent writing to the same key
19+
* support for ttl (time to live) at retrieval
20+
* support for deleting expired data
21+
22+
Finally, there are multiple data models for storing the features in the online store. For example, features could be:
23+
* collocated by feature view
24+
* collocated by feature service
25+
* collocated by entity key
26+
27+
See this [issue](https://github.com/feast-dev/feast/issues/2254) for a discussion around the tradeoffs of each of these data models.
28+
29+
## Functionality Matrix
30+
31+
There are currently five core online store implementations: `SqliteOnlineStore`, `RedisOnlineStore`, `DynamoDBOnlineStore`, `SnowflakeOnlineStore`, and `DatastoreOnlineStore`.
32+
There are several additional implementations contributed by the Feast community (`PostgreSQLOnlineStore`, `HbaseOnlineStore`, and `CassandraOnlineStore`), which are not guaranteed to be stable or to match the functionality of the core implementations.
33+
Details for each specific online store, such as how to configure it in a `feature_store.yaml`, can be found [here](README.md).
34+
35+
Below is a matrix indicating which online stores support what functionality.
36+
37+
| | Sqlite | Redis | DynamoDB | Snowflake | Datastore | Postgres | Hbase | Cassandra |
38+
| :-------------------------------------------------------- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |
39+
| write feature values to the online store | yes | yes | yes | yes | yes | yes | yes | yes |
40+
| read feature values from the online store | yes | yes | yes | yes | yes | yes | yes | yes |
41+
| update infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes |
42+
| teardown infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes |
43+
| generate a plan of infrastructure changes | yes | no | no | no | no | no | no | yes |
44+
| support for on-demand transforms | yes | yes | yes | yes | yes | yes | yes | yes |
45+
| readable by Python SDK | yes | yes | yes | yes | yes | yes | yes | yes |
46+
| readable by Java | no | yes | no | no | no | no | no | no |
47+
| readable by Go | yes | yes | no | no | no | no | no | no |
48+
| support for entityless feature views | yes | yes | yes | yes | yes | yes | yes | yes |
49+
| support for concurrent writing to the same key | no | yes | no | no | no | no | no | no |
50+
| support for ttl (time to live) at retrieval | no | yes | no | no | no | no | no | no |
51+
| support for deleting expired data | no | yes | no | no | no | no | no | no |
52+
| collocated by feature view | yes | no | yes | yes | yes | yes | yes | yes |
53+
| collocated by feature service | no | no | no | no | no | no | no | no |
54+
| collocated by entity key | no | yes | no | no | no | no | no | no |

‎docs/reference/online-stores/postgres.md‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,4 +30,30 @@ online_store:
3030
```
3131
{% endcode %}
3232
33-
Configuration options are available [here](https://rtd.feast.dev/en/latest/feast.infra.utils.postgres.html#module-feast.infra.utils.postgres.postgres_config).
33+
The full set of configuration options is available in [PostgreSQLOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.contrib.postgres.PostgreSQLOnlineStoreConfig).
34+
35+
## Functionality Matrix
36+
37+
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
38+
Below is a matrix indicating which functionality is supported by the Postgres online store.
39+
40+
| | Postgres |
41+
| :-------------------------------------------------------- | :-- |
42+
| write feature values to the online store | yes |
43+
| read feature values from the online store | yes |
44+
| update infrastructure (e.g. tables) in the online store | yes |
45+
| teardown infrastructure (e.g. tables) in the online store | yes |
46+
| generate a plan of infrastructure changes | no |
47+
| support for on-demand transforms | yes |
48+
| readable by Python SDK | yes |
49+
| readable by Java | no |
50+
| readable by Go | no |
51+
| support for entityless feature views | yes |
52+
| support for concurrent writing to the same key | no |
53+
| support for ttl (time to live) at retrieval | no |
54+
| support for deleting expired data | no |
55+
| collocated by feature view | yes |
56+
| collocated by feature service | no |
57+
| collocated by entity key | no |
58+
59+
To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).

‎docs/reference/online-stores/redis.md‎

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,12 @@
44

55
The [Redis](https://redis.io) online store provides support for materializing feature values into Redis.
66

7-
* Both Redis and Redis Cluster are supported
7+
* Both Redis and Redis Cluster are supported.
88
* The data model used to store feature values in Redis is described in more detail [here](../../specs/online\_store\_format.md).
99

1010
## Examples
1111

12-
Connecting to a single Redis instance
12+
Connecting to a single Redis instance:
1313

1414
{% code title="feature_store.yaml" %}
1515
```yaml
@@ -22,7 +22,7 @@ online_store:
2222
```
2323
{% endcode %}
2424
25-
Connecting to a Redis Cluster with SSL enabled and password authentication
25+
Connecting to a Redis Cluster with SSL enabled and password authentication:
2626
2727
{% code title="feature_store.yaml" %}
2828
```yaml
@@ -36,4 +36,30 @@ online_store:
3636
```
3737
{% endcode %}
3838
39-
Configuration options are available [here](https://rtd.feast.dev/en/master/#feast.infra.online\_stores.redis.RedisOnlineStoreConfig).
39+
The full set of configuration options is available in [RedisOnlineStoreConfig](https://rtd.feast.dev/en/latest/#feast.infra.online_stores.redis.RedisOnlineStoreConfig).
40+
41+
## Functionality Matrix
42+
43+
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
44+
Below is a matrix indicating which functionality is supported by the Redis online store.
45+
46+
| | Redis |
47+
| :-------------------------------------------------------- | :-- |
48+
| write feature values to the online store | yes |
49+
| read feature values from the online store | yes |
50+
| update infrastructure (e.g. tables) in the online store | yes |
51+
| teardown infrastructure (e.g. tables) in the online store | yes |
52+
| generate a plan of infrastructure changes | no |
53+
| support for on-demand transforms | yes |
54+
| readable by Python SDK | yes |
55+
| readable by Java | yes |
56+
| readable by Go | yes |
57+
| support for entityless feature views | yes |
58+
| support for concurrent writing to the same key | yes |
59+
| support for ttl (time to live) at retrieval | yes |
60+
| support for deleting expired data | yes |
61+
| collocated by feature view | no |
62+
| collocated by feature service | no |
63+
| collocated by entity key | yes |
64+
65+
To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).

‎docs/reference/online-stores/snowflake.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,3 +33,31 @@ online_store:
3333
database: SNOWFLAKE_DATABASE
3434
```
3535
{% endcode %}
36+
37+
The full set of configuration options is available in [SnowflakeOnlineStoreConfig](https://rtd.feast.dev/en/latest/#feast.infra.online_stores.snowflake.SnowflakeOnlineStoreConfig).
38+
39+
## Functionality Matrix
40+
41+
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
42+
Below is a matrix indicating which functionality is supported by the Snowflake online store.
43+
44+
| | Snowflake |
45+
| :-------------------------------------------------------- | :-- |
46+
| write feature values to the online store | yes |
47+
| read feature values from the online store | yes |
48+
| update infrastructure (e.g. tables) in the online store | yes |
49+
| teardown infrastructure (e.g. tables) in the online store | yes |
50+
| generate a plan of infrastructure changes | no |
51+
| support for on-demand transforms | yes |
52+
| readable by Python SDK | yes |
53+
| readable by Java | no |
54+
| readable by Go | no |
55+
| support for entityless feature views | yes |
56+
| support for concurrent writing to the same key | no |
57+
| support for ttl (time to live) at retrieval | no |
58+
| support for deleting expired data | no |
59+
| collocated by feature view | yes |
60+
| collocated by feature service | no |
61+
| collocated by entity key | no |
62+
63+
To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).

0 commit comments

Comments
 (0)