Skip to content

Commit 9664222

Browse files
adchiaadchia
authored
chore: Fixing outdated documentation and roadmap. Also fixing FileOptions + PushSource (feast-dev#2543)
* chore: Fixing outdated documentation and roadmap. Making PushSource schema consistent with FV. Remove FileOptions warnings Signed-off-by: Danny Chiao <danny@tecton.ai> * fix Signed-off-by: Danny Chiao <danny@tecton.ai> * fix Signed-off-by: Danny Chiao <danny@tecton.ai> * fix Signed-off-by: Danny Chiao <danny@tecton.ai> * change request source example Signed-off-by: Danny Chiao <danny@tecton.ai>
1 parent a2ef375 commit 9664222

24 files changed

Lines changed: 139 additions & 165 deletions

‎Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ test: test-python test-java test-go
3232

3333
protos: compile-protos-go compile-protos-python compile-protos-docs
3434

35-
build: protos build-java build-docker build-html
35+
build: protos build-java build-docker
3636

3737
# Python SDK
3838

‎README.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -147,8 +147,8 @@ The list below contains the functionality that contributors are planning to deve
147147
* [x] [Synapse source (community plugin)](https://github.com/Azure/feast-azure)
148148
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
149149
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
150-
* [x] [Spark (community plugin)](https://github.com/Adyen/feast-spark-offline-store)
151-
* [x] Kafka source (with [push support into the online store](https://docs.feast.dev/reference/alpha-stream-ingestion))
150+
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/data-sources/spark)
151+
* [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push)
152152
* [ ] HTTP source
153153
* **Offline Stores**
154154
* [x] [Snowflake](https://docs.feast.dev/reference/offline-stores/snowflake)
@@ -158,7 +158,7 @@ The list below contains the functionality that contributors are planning to deve
158158
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
159159
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
160160
* [x] [Trino (community plugin)](https://github.com/Shopify/feast-trino)
161-
* [x] [Spark (community plugin)](https://github.com/Adyen/feast-spark-offline-store)
161+
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/offline-stores/spark)
162162
* [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
163163
* [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store)
164164
* **Online Stores**
@@ -173,12 +173,12 @@ The list below contains the functionality that contributors are planning to deve
173173
* [ ] Cassandra
174174
* **Streaming**
175175
* [x] [Custom streaming ingestion job support](https://docs.feast.dev/how-to-guides/creating-a-custom-provider)
176-
* [x] [Push based streaming data ingestion](reference/alpha-stream-ingestion.md)
176+
* [x] [Push based streaming data ingestion](https://docs.feast.dev/reference/data-sources/push.md)
177177
* [ ] Streaming ingestion on AWS
178178
* [ ] Streaming ingestion on GCP
179179
* **Feature Engineering**
180180
* [x] On-demand Transformations (Alpha release. See [RFC](https://docs.google.com/document/d/1lgfIw0Drc65LpaxbUu49RCeJgMew547meSJttnUqz7c/edit#))
181-
* [ ] Batch transformation (SQL. In progress. See [RFC](https://docs.google.com/document/d/1964OkzuBljifDvkV-0fakp2uaijnVzdwWNGdz7Vz50A/edit))
181+
* [ ] Batch transformation (In progress. See [RFC](https://docs.google.com/document/d/1964OkzuBljifDvkV-0fakp2uaijnVzdwWNGdz7Vz50A/edit))
182182
* [ ] Streaming transformation
183183
* **Deployments**
184184
* [x] AWS Lambda (Alpha release. See [RFC](https://docs.google.com/document/d/1eZWKWzfBif66LDN32IajpaG-j82LSHCCOzY6R7Ax7MI/edit))
@@ -204,7 +204,7 @@ The list below contains the functionality that contributors are planning to deve
204204
* [x] CLI for browsing feature registry
205205
* [x] Model-centric feature tracking (feature services)
206206
* [x] Amundsen integration (see [Feast extractor](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/feast_extractor.py))
207-
* [ ] Feast Web UI (in progress)
207+
* [x] Feast Web UI (alpha)
208208
* [ ] REST API for browsing feature registry
209209
* [ ] Feature versioning
210210

‎docs/getting-started/architecture-and-components/online-store.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,4 @@ Once the above data source is materialized into Feast \(using `feast materialize
1212

1313
![](../../.gitbook/assets/image%20%285%29.png)
1414

15+
Features can also be written to the online store via [push sources](https://docs.feast.dev/reference/data-sources/push)

‎docs/getting-started/concepts/feature-view.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,8 @@ driver_stats_fv = FeatureView(
1717
Field(name="trips_today", dtype=Int64),
1818
Field(name="rating", dtype=Float32),
1919
],
20-
batch_source=BigQuerySource(
21-
table_ref="feast-oss.demo_data.driver_activity"
20+
source=BigQuerySource(
21+
table="feast-oss.demo_data.driver_activity"
2222
)
2323
)
2424
```
@@ -28,7 +28,7 @@ driver_stats_fv = FeatureView(
2828
Feature views are used during
2929

3030
* The generation of training datasets by querying the data source of feature views in order to find historical feature values. A single training dataset may consist of features from multiple feature views.
31-
* Loading of feature values into an online store. Feature views determine the storage schema in the online store. Feature values can be loaded from batch sources or from [stream sources](../../reference/alpha-stream-ingestion.md).
31+
* Loading of feature values into an online store. Feature views determine the storage schema in the online store. Feature values can be loaded from batch sources or from [stream sources](../../reference/data-sources/push.md).
3232
* Retrieval of features from the online store. Feature views provide the schema definition to Feast in order to look up features from the online store.
3333

3434
{% hint style="info" %}
@@ -37,7 +37,7 @@ Feast does not generate feature values. It acts as the ingestion and serving sys
3737

3838
## Feature views without entities
3939

40-
If a feature view contains features that are not related to a specific entity, the feature view can be defined without entities.
40+
If a feature view contains features that are not related to a specific entity, the feature view can be defined without entities (only event timestamps are needed for this feature view).
4141

4242
{% tabs %}
4343
{% tab title="global_stats.py" %}
@@ -51,8 +51,8 @@ global_stats_fv = FeatureView(
5151
schema=[
5252
Field(name="total_trips_today_by_all_drivers", dtype=Int64),
5353
],
54-
batch_source=BigQuerySource(
55-
table_ref="feast-oss.demo_data.global_stats"
54+
source=BigQuerySource(
55+
table="feast-oss.demo_data.global_stats"
5656
)
5757
)
5858
```
@@ -87,8 +87,8 @@ location_stats_fv= FeatureView(
8787
schema=[
8888
Field(name="temperature", dtype=Int32)
8989
],
90-
batch_source=BigQuerySource(
91-
table_ref="feast-oss.demo_data.location_stats"
90+
source=BigQuerySource(
91+
table="feast-oss.demo_data.location_stats"
9292
),
9393
)
9494
```

‎docs/getting-started/concepts/point-in-time-joins.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ The above table can be registered with Feast through the following feature view:
99
```python
1010
from feast import FeatureView, Field, FileSource
1111
from feast.types import Float32, Int64
12+
from datetime import timedelta
1213

1314
driver_stats_fv = FeatureView(
1415
name="driver_hourly_stats",
@@ -18,7 +19,7 @@ driver_stats_fv = FeatureView(
1819
Field(name="earnings_today", dtype=Float32),
1920
],
2021
ttl=timedelta(hours=2),
21-
batch_source=FileSource(
22+
source=FileSource(
2223
path="driver_hourly_stats.parquet"
2324
)
2425
)

‎docs/getting-started/quickstart.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -106,14 +106,14 @@ driver = Entity(name="driver", value_type=ValueType.INT64, join_key="driver_id",
106106
driver_hourly_stats_view = FeatureView(
107107
name="driver_hourly_stats",
108108
entities=["driver"], # reference entity by name
109-
ttl=Duration(seconds=86400 * 1),
109+
ttl=timedelta(seconds=86400 * 1),
110110
schema=[
111111
Field(name="conv_rate", dtype=Float32),
112112
Field(name="acc_rate", dtype=Float32),
113113
Field(name="avg_daily_trips", dtype=Int64),
114114
],
115115
online=True,
116-
batch_source=driver_hourly_stats,
116+
source=driver_hourly_stats,
117117
tags={},
118118
)
119119
```
@@ -176,14 +176,14 @@ driver = Entity(name="driver", value_type=ValueType.INT64, join_key="driver_id",
176176
driver_hourly_stats_view = FeatureView(
177177
name="driver_hourly_stats",
178178
entities=["driver"], # reference entity by name
179-
ttl=Duration(seconds=86400 * 1),
179+
ttl=timedelta(seconds=86400 * 1),
180180
schema=[
181181
Field(name="conv_rate", dtype=Float32),
182182
Field(name="acc_rate", dtype=Float32),
183183
Field(name="avg_daily_trips", dtype=Int64),
184184
],
185185
online=True,
186-
batch_source=driver_hourly_stats,
186+
source=driver_hourly_stats,
187187
tags={},
188188
)
189189
```

‎docs/getting-started/third-party-integrations.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,8 @@ Don't see your offline store or online store of choice here? Check out our guide
2020
* [x] [Synapse source (community plugin)](https://github.com/Azure/feast-azure)
2121
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
2222
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
23-
* [x] [Spark (community plugin)](https://github.com/Adyen/feast-spark-offline-store)
24-
* [x] Kafka source (with [push support into the online store](https://docs.feast.dev/reference/alpha-stream-ingestion))
23+
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/data-sources/spark)
24+
* [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push)
2525
* [ ] HTTP source
2626

2727
### Offline Stores
@@ -33,7 +33,7 @@ Don't see your offline store or online store of choice here? Check out our guide
3333
* [x] [Hive (community plugin)](https://github.com/baineng/feast-hive)
3434
* [x] [Postgres (community plugin)](https://github.com/nossrannug/feast-postgres)
3535
* [x] [Trino (community plugin)](https://github.com/Shopify/feast-trino)
36-
* [x] [Spark (community plugin)](https://github.com/Adyen/feast-spark-offline-store)
36+
* [x] [Spark (community plugin)](https://docs.feast.dev/reference/offline-stores/spark)
3737
* [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
3838
* [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store)
3939

‎docs/how-to-guides/adding-a-new-offline-store.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -252,7 +252,7 @@ pdriver_hourly_stats = CustomFileDataSource(
252252
253253
254254
driver_hourly_stats_view = FeatureView(
255-
batch_source=driver_hourly_stats,
255+
source=driver_hourly_stats,
256256
...
257257
)
258258
```

‎docs/how-to-guides/running-feast-in-production.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -287,7 +287,7 @@ store = FeatureStore(...)
287287

288288
def feast_writer(spark_df):
289289
pandas_df = spark_df.to_pandas()
290-
store.write_to_online_store("driver_hourly_stats", pandas_df)
290+
store.push("driver_hourly_stats", pandas_df)
291291

292292
streamingDF.writeStream.foreachBatch(feast_writer).start()
293293
```

‎docs/reference/alpha-on-demand-feature-view.md‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -25,20 +25,21 @@ See [https://github.com/feast-dev/on-demand-feature-views-demo](https://github.c
2525

2626
### **Registering transformations**
2727

28-
We register `RequestDataSource` inputs and the transform in `on_demand_feature_view`:
28+
We register `RequestSource` inputs and the transform in `on_demand_feature_view`:
2929

3030
```python
3131
from feast import Field, RequestSource
32-
from feast.types import Float64
32+
from feast.types import Float64, Int64
33+
import pandas as pd
3334

3435
# Define a request data source which encodes features / information only
3536
# available at request time (e.g. part of the user initiated HTTP request)
36-
input_request = RequestDataSource(
37+
input_request = RequestSource(
3738
name="vals_to_add",
38-
schema={
39-
"val_to_add": ValueType.INT64,
40-
"val_to_add_2": ValueType.INT64
41-
}
39+
schema=[
40+
Field(name='val_to_add', dtype=Int64),
41+
Field(name='val_to_add_2', dtype=Int64)
42+
]
4243
)
4344

4445
# Use the input data and feature view features to create new features

0 commit comments

Comments
 (0)