Skip to content

Commit d8bd8cf

Browse files
adchiagitbook-bot
adchia
authored and
gitbook-bot
committed
GitBook: [#332] Updating roadmap and adding stream push API docs
1 parent 3ee88f4 commit d8bd8cf

4 files changed

Lines changed: 80 additions & 33 deletions

File tree

‎docs/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@
7272
* [feature\_store.yaml](reference/feature-repository/feature-store-yaml.md)
7373
* [.feastignore](reference/feature-repository/feast-ignore.md)
7474
* [\[Alpha\] On demand feature view](reference/alpha-on-demand-feature-view.md)
75+
* [\[Alpha\] Stream ingestion](reference/alpha-stream-ingestion.md)
7576
* [\[Alpha\] Local feature server](reference/feature-server.md)
7677
* [\[Alpha\] AWS Lambda feature server](reference/alpha-aws-lambda-feature-server.md)
7778
* [Feast CLI reference](reference/feast-cli-commands.md)

‎docs/how-to-guides/adding-or-reusing-tests.md‎

Lines changed: 30 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -17,35 +17,35 @@ $ tree
1717

1818
.
1919
├── e2e
20-
│   └── test_universal_e2e.py
20+
│ └── test_universal_e2e.py
2121
├── feature_repos
22-
│   ├── repo_configuration.py
23-
│   └── universal
24-
│   ├── data_source_creator.py
25-
│   ├── data_sources
26-
│   │   ├── bigquery.py
27-
│   │   ├── file.py
28-
│   │   └── redshift.py
29-
│   ├── entities.py
30-
│   └── feature_views.py
22+
│ ├── repo_configuration.py
23+
│ └── universal
24+
│ ├── data_source_creator.py
25+
│ ├── data_sources
26+
│ │ ├── bigquery.py
27+
│ │ ├── file.py
28+
│ │ └── redshift.py
29+
│ ├── entities.py
30+
│ └── feature_views.py
3131
├── offline_store
32-
│   ├── test_s3_custom_endpoint.py
33-
│   └── test_universal_historical_retrieval.py
32+
│ ├── test_s3_custom_endpoint.py
33+
│ └── test_universal_historical_retrieval.py
3434
├── online_store
35-
│   ├── test_e2e_local.py
36-
│   ├── test_feature_service_read.py
37-
│   ├── test_online_retrieval.py
38-
│   └── test_universal_online.py
35+
│ ├── test_e2e_local.py
36+
│ ├── test_feature_service_read.py
37+
│ ├── test_online_retrieval.py
38+
│ └── test_universal_online.py
3939
├── registration
40-
│   ├── test_cli.py
41-
│   ├── test_cli_apply_duplicated_featureview_names.py
42-
│   ├── test_cli_chdir.py
43-
│   ├── test_feature_service_apply.py
44-
│   ├── test_feature_store.py
45-
│   ├── test_inference.py
46-
│   ├── test_registry.py
47-
│   ├── test_universal_odfv_feature_inference.py
48-
│   └── test_universal_types.py
40+
│ ├── test_cli.py
41+
│ ├── test_cli_apply_duplicated_featureview_names.py
42+
│ ├── test_cli_chdir.py
43+
│ ├── test_feature_service_apply.py
44+
│ ├── test_feature_store.py
45+
│ ├── test_inference.py
46+
│ ├── test_registry.py
47+
│ ├── test_universal_odfv_feature_inference.py
48+
│ └── test_universal_types.py
4949
└── scaffolding
5050
├── test_init.py
5151
├── test_partial_apply.py
@@ -148,30 +148,30 @@ The key fixtures are the `environment` and `universal_data_sources` fixtures, wh
148148

149149
## Writing a new test or reusing existing tests
150150

151-
To add a new test to an existing test file:
151+
### To add a new test to an existing test file
152152

153153
* Use the same function signatures as an existing test (e.g. use `environment` as an argument) to include the relevant test fixtures.
154154
* If possible, expand an individual test instead of writing a new test, due to the cost of standing up offline / online stores.
155155

156-
To test a new offline / online store from a plugin repo:
156+
### To test a new offline / online store from a plugin repo
157157

158158
* Install Feast in editable mode with `pip install -e`.
159159
* The core tests for offline / online store behavior are parametrized by the `FULL_REPO_CONFIGS` variable defined in `feature_repos/repo_configuration.py`. To overwrite this variable without modifying the Feast repo, create your own file that contains a `FULL_REPO_CONFIGS` (which will require adding a new `IntegrationTestRepoConfig` or two) and set the environment variable `FULL_REPO_CONFIGS_MODULE` to point to that file. Then the core offline / online store tests can be run with `make test-python-universal`.
160160
* See the [custom offline store demo](https://github.com/feast-dev/feast-custom-offline-store-demo) and the [custom online store demo](https://github.com/feast-dev/feast-custom-online-store-demo) for examples.
161161

162-
To include a new offline / online store in the main Feast repo:
162+
### To include a new offline / online store in the main Feast repo
163163

164164
* Extend `data_source_creator.py` for your offline store.
165165
* In `repo_configuration.py` add a new`IntegrationTestRepoConfig` or two (depending on how many online stores you want to test).
166166
* Run the full test suite with `make test-python-integration.`
167167

168-
To include a new online store:
168+
### To include a new online store
169169

170170
* In `repo_configuration.py` add a new config that maps to a serialized version of configuration you need in `feature_store.yaml` to setup the online store.
171171
* In `repo_configuration.py`, add new`IntegrationTestRepoConfig` for offline stores you want to test.
172172
* Run the full test suite with `make test-python-integration`
173173

174-
To use custom data in a new test:
174+
### To use custom data in a new test
175175

176176
* Check `test_universal_types.py` for an example of how to do this.
177177

‎docs/reference/alpha-stream-ingestion.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# \[Alpha] Stream ingestion
2+
3+
**Warning**: This is an _experimental_ feature. It's intended for early testing and feedback, and could change without warnings in future releases.
4+
5+
{% hint style="info" %}
6+
To enable this feature, run **`feast alpha enable direct_ingest_to_online_store`**
7+
{% endhint %}
8+
9+
## Overview
10+
11+
Streaming data sources are important sources of feature values. A typical setup with streaming data looks like:
12+
13+
1. Raw events come in (stream 1)
14+
2. Streaming transformations applied (e.g. `last_N_purchased_categories`) (stream 2)
15+
3. Write stream 2 values to an offline store as a historical log for training
16+
4. Write stream 2 values to an online store for low latency feature serving
17+
5. Periodically materialize feature values from the offline store into the online store for improved correctness
18+
19+
Feast now allows users to push features previously registered in a feature view to the online store. This most commonly would be done from a stream processing job (e.g. a Beam or Spark Streaming job). Future versions of Feast will allow writing features directly to the offline store as well.
20+
21+
## Example
22+
23+
See [https://github.com/feast-dev/feast-demo](https://github.com/feast-dev/on-demand-feature-views-demo) for an example on how to use on demand feature views.
24+
25+
We register a feature view as normal, and during stream processing (e.g. Kafka consumers), now we push a dataframe matching the feature view schema:
26+
27+
```python
28+
event_df = pd.DataFrame.from_dict(
29+
{
30+
"driver_id": [1001],
31+
"event_timestamp": [
32+
datetime(2021, 5, 13, 10, 59, 42),
33+
],
34+
"created": [
35+
datetime(2021, 5, 13, 10, 59, 42),
36+
],
37+
"conv_rate": [1.0],
38+
"acc_rate": [1.0],
39+
"avg_daily_trips": [1000],
40+
}
41+
)
42+
store.write_to_online_store("driver_hourly_stats", event_df)
43+
```
44+
45+
Feast will coordinate between pushed stream data and regular materialization jobs to ensure only the latest feature values are written to the online store. This ensures correctness in served features for model inference.

‎docs/roadmap.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ The list below contains the functionality that contributors are planning to deve
1313
* [x] [Synapse source (community plugin)](https://github.com/Azure/feast-azure)
1414
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
1515
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
16-
* [ ] Kafka source (Planned for Q4 2021)
16+
* [x] Kafka source (with [push support into the online store](reference/alpha-stream-ingestion.md))
1717
* [ ] Snowflake source (Planned for Q4 2021)
1818
* [ ] HTTP source
1919
* **Offline Stores**
@@ -38,7 +38,8 @@ The list below contains the functionality that contributors are planning to deve
3838
* [ ] Cassandra
3939
* **Streaming**
4040
* [x] [Custom streaming ingestion job support](https://docs.feast.dev/how-to-guides/creating-a-custom-provider)
41-
* [ ] Streaming ingestion on AWS (Planned for Q4 2021)
41+
* [x] [Push based streaming data ingestion](reference/alpha-stream-ingestion.md)
42+
* [ ] Streaming ingestion on AWS
4243
* [ ] Streaming ingestion on GCP
4344
* **Feature Engineering**
4445
* [x] On-demand Transformations (Alpha release. See [RFC](https://docs.google.com/document/d/1lgfIw0Drc65LpaxbUu49RCeJgMew547meSJttnUqz7c/edit#))
@@ -53,9 +54,9 @@ The list below contains the functionality that contributors are planning to deve
5354
* [x] Python Client
5455
* [x] REST Feature Server (Python) (Alpha release. See [RFC](https://docs.google.com/document/d/1iXvFhAsJ5jgAhPOpTdB3j-Wj1S9x3Ev\_Wr6ZpnLzER4/edit))
5556
* [x] gRPC Feature Server (Java) (See [#1497](https://github.com/feast-dev/feast/issues/1497))
57+
* [x] Push API
5658
* [ ] Java Client
5759
* [ ] Go Client
58-
* [ ] Push API
5960
* [ ] Delete API
6061
* [ ] Feature Logging (for training)
6162
* **Data Quality Management**

0 commit comments

Comments
 (0)