Skip to content

Commit 4129833

Browse files
adchiaadchia
committed
Finish module 0
Signed-off-by: Danny Chiao <danny@tecton.ai>
1 parent 63fb0a9 commit 4129833

File tree

1 file changed

+44
-35
lines changed

1 file changed

+44
-35
lines changed

‎module_0/README.md‎

Lines changed: 44 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,13 @@ We focus on a specific example (that does not include online features + models):
88

99
# 1. Mapping to Feast concepts
1010
To support this, you'll need:
11-
| Concept | Requirements |
12-
| :-------------- | :---------------------------------------------------------------------------------------------------- |
13-
| Data sources | `FileSource` (with S3 paths and endpoint overrides) and `FeatureView`s registered with `feast apply` |
14-
| Feature views | Feature views tied to data sources that are shared by data scientists, registered with `feast apply` |
15-
| Provider | In `feature_store.yaml`, specifying the `aws` provider to ensure your registry can be stored in S3 |
16-
| Registry | In `feature_store.yaml`, specifying a path (within an existing S3 bucket) the registry is written to. |
17-
| Transformations | Feast supports last mile transformations with `OnDemandFeatureView` that can be re-used |
11+
| Concept | Requirements |
12+
| :-------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
13+
| Data sources | `FileSource` (with S3 paths and endpoint overrides) and `FeatureView`s registered with `feast apply` |
14+
| Feature views | Feature views tied to data sources that are shared by data scientists, registered with `feast apply` |
15+
| Provider | In `feature_store.yaml`, specifying the `aws` provider to ensure your registry can be stored in S3 |
16+
| Registry | In `feature_store.yaml`, specifying a path (within an existing S3 bucket) the registry is written to. Users + model servers will pull from this to get the latest registered features + metadata |
17+
| Transformations | Feast supports last mile transformations with `OnDemandFeatureView`s that can be re-used |
1818

1919
# 2. User flows
2020
There are three user groups here worth considering. The ML platform team, the data scientists, and the ML engineers scheduling models in batch. We visit the first two of these in
@@ -37,30 +37,32 @@ flags:
3737
on_demand_transforms: true
3838
```
3939
40-
Some quick recap of what's happening here:
41-
- `project` gives infrastructure isolation.
42-
- Commonly, to start, users will start with one large project for multiple teams.
43-
- All Feast objects like `FeatureView`s have associated projects. Users can only request features from a single project.
44-
- Online stores (if used) namespace by project names
45-
- `provider` defines registry locations & sets defaults for offline / online stores options
46-
- For example, the `gcp` provider will enable registries in GCS and sets BigQuery + Datastore as the default offline / online stores.
47-
- Provider defaults can be easily overriden in `feature_store.yaml`
48-
- `registry` defines the specific path for the registry
49-
- The Feast registry is the source of truth on registered Feast objects.
50-
- Users + model servers will pull from this to get the latest registered features + metadata.
51-
- **Note**: technically, multiple projects can use the same registry, though Feast was not designed with this in mind. Discovery of adjacent features is possible in this flow, but not retrieval.
52-
- `online_store` here is set to null.
40+
A quick explanation of what's happening here:
41+
42+
| Key | What it does | Example |
43+
| :-------------- | :----------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------- |
44+
| `project` | Gives infrastructure isolation via namespacing (e.g. online stores + Feast objects). | any unique name (e.g. `feast_demo_aws`) |
45+
| `provider` | Defines registry locations & sets defaults for offline / online stores options | `gcp` enables registries in GCS and sets BigQuery + Datastore as the default offline / online stores. |
46+
| `registry` | Defines the specific path for the registry (local, gcs, s3, etc) | `s3://[YOUR BUCKET]/registry.pb` |
47+
| `online_store` | Specifies configuration for the online store (or null if online store isn't needed) | `redis`, `dynamodb`, `datastore`, `postgres`, `hbase` (each have their own extra config flags) |
48+
| `offline_store` | Specifies configuration for the offline store, which executes point in time joins | `bigquery`, `snowflake.offline`, `redshift`, `spark`, `trino` (each have their own extra config flags) |
49+
| `flags` | (legacy) Soon to be deprecated way to enable experimental functionality. | |
50+
51+
Some further notes and gotchas:
52+
- Generally, custom offline + online stores and providers are supported and can plug in.
53+
- e.g. see [adding a new offline store](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store), [adding a new online store](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store)
54+
- **Project**
55+
- users can only request features from a single project
56+
- **Provider**
57+
- defaults can be easily overriden in `feature_store.yaml`.
58+
- For example, one can use the `aws` provider and specify Snowflake as the offline store.
59+
- **Offline Store**
60+
- we recommend users use data warehouses or Spark as their offline store for performant training dataset generation.
61+
- Here, we use file sources for instructional purposes. This will directly read from files (local or remote) and use Dask to execute point-in-time joins.
62+
- A project can only support one type of offline store (cannot mix Snowflake + file for example)
63+
- **Online Store**
5364
- If you don't need to power real time models with fresh features, this is not needed.
54-
- If you are batch scoring, for example, then the online store is optional.
55-
- `offline_store` defines the offline compute that executes point in time joins
56-
- Can only be one type (cannot mix Snowflake + file for example)
57-
- Here, for instruction purposes, we use `file` sources.
58-
- This will directly read from files (local or remote) and use Dask to execute point-in-time joins.
59-
- We **do not** recommend this for production usage.
60-
- We recommend users to use data warehouses as their offline store for performant training dataset generation.
61-
- There is also a contrib plugin (`SparkOfflineStore`) which supports retrieving features with Spark.
62-
- `flags` are the legacy way of controlling alpha features
63-
- We're likely to deprecate this system soon, but today it still gates `OnDemandFeatureView` which is still under development.
65+
- If you are precomputing predictions in batch ("batch scoring"), then the online store is optional. You should be using the offline store and running `feature_store.get_historical_features`
6466

6567
With the `feature_store.yaml` setup, you can now run `feast apply` to create & populate the registry.
6668

@@ -119,11 +121,18 @@ Additionally, users will often want to have a dev/staging environment that's sep
119121
```
120122

121123
## 2b. Data scientists
122-
TODO
123-
124-
Two ways of working
125-
- Use the `client/` folder approach of not authoring features and primarily re-using features already used in production.
126-
- Have a local copy of the feature repository (e.g. `git clone`). Then the data scientist can iterate on features locally, apply features to their own dev project with a local registry, and then submit PRs to include features that should be used in production (including A/B experiments, or model training iterations)
124+
Data scientists will be using or authoring features in Feast.
125+
126+
There are two ways they can use Feast:
127+
- Use Feast primarily as a way of pulling production ready features.
128+
- See the `client/` folder for an example of how users can pull features by only having a `feature_store.yaml`
129+
- This is **not recommended** since data scientists cannot register feature services to indicate they depend on certain features in production.
130+
- **[Recommended]** Have a local copy of the feature repository (e.g. `git clone`) and author / iterate / re-use features.
131+
- Data scientist can:
132+
1. iterate on features locally
133+
2. apply features to their own dev project with a local registry & experiment
134+
3. build feature services in preparation for production
135+
4. submit PRs to include features that should be used in production (including A/B experiments, or model training iterations)
127136

128137
Data scientists can also investigate other models and their dependent features / data sources / on demand transformations through the repository or through the Web UI (by running `feast ui`)
129138

0 commit comments

Comments
 (0)