Feature Tables vs Feature Views #1583

Creating this issue to discuss some concepts in Feast, Feature Sets, Feature Tables and Feature Views.

Feature Sets

Prior to Feast 0.8, Feast had a concept of Feature Sets (not to be confused with the new Feature Set RFC). Feature sets were logical groups of features that occured together. These groups of features share an entity (which can be composite) and in the offline case they also share timestamps. For example, a feature set could be used to store a log of events, or it could be used to store the results of an aggregation. The idea is that different processes (stream or batch ETLs) would output data into their own tables, and Feast would join these different tables during retrieval. Therefore feature sets avoid a sparse table problem.

Importantly, Feature Sets did NOT have a source. Users were always asked to push data to the feature store. For batch ingestion, the users did the following

client.ingest("my_feature_set", my_pandas_dataframe)

For stream ingestion, teams would push to a specific topic for a feature set.

The feature store would provide both offline and online storage of user data, and allowed users to imperatively load features into the feature store. Feature sets made the feature store into the source of truth for feature data. Users would ingest from both their notebooks as well as their batch or streaming ETL pipelines.

Feature Tables

In Feast 0.8, we replaced Feature Set with Feature Table. The main reason was scoping. Many teams already have data being stored in specific locations like data warehouses and lakes. This allows Feast to materialize (load) data from outside the feature store into the feature store for storage and serving, and means that Feast doesn't have to become the source of truth for feature data (it lives externally). Feast would not create or manage the offline store in this case, unlike in Feast 0.7 and before.

The idea was not that Feast would never provide an offline store. The primary reason we did not start with managing an offline store was because the source-centric approach scoped down the project and allows us to address most use cases.

Because we had ingest() for feature sets in Feast 0.7, we had to provide backward compatibility for teams that wanted to ingest data from ETL pipelines. In order to do so, we still provided the ingest() functionality. However, this pushed directly to the source location, not into the offline store. The point of this ingest() was only to provide a migration path to the new Feast (0.8, 0.9), not to be a long term API to exist alongside sources. In fact, pushing directly to a source is an anti-pattern since its often the case that teams do not have write access to sources.

Feature Views

Feature Views were introduced in Feast 0.10. Feature views can be thought of as

• A removal of the ingest() functionality in feature tables
• A rename to be clearer about the functionality provided (federation over an external source).

Feature views in Feast 0.10 function the same as with feature tables in 0.9, but we do not allow direct ingestion to a feature view's source. The feature view can be "materialized", which pulls from the source and loads the data into the feature store. Right now we only materialize into the online store since we are able to query the batch source directly in order to build training datasets.

Note: Feast is not only concerned with loading data into an online store. Feature views only dictate that the source of data lives externally to the feature store, but there is a case for materialization into both an online and offline store in theory. The use case for the offline store is

• When we want to materialize data that is too expensive to compute in an ad hoc way for every training run. By storing it in the offline store, we'd minimize queries to the source.
• When the upstream source is unreliable and teams want to ensure that they retain copies of data inside the feature store. This is especially important to ensure online/offline consistency in the future.

Feature Tables (potential reintroduction)

Now that feature views have a clear purpose, we are considering introducing feature tables to address the previously removed ingestion functionality. The use case is the same as feature sets in Feast 0.7.

Users have data in their ETL pipelines or Jupyter notebooks, and they need a structured location to store that data for consumption in models. Feature tables would allow them to load and store their data in the feature store, thereby becoming the source of truth for this feature data. This solves the following problems.

• Provides a structured storage location for feature data where teams don't already have an offline store
• Reduces the amount of work that users need to do in order to get data into the feature store. In the case of feature views, they need to create a table in a database, upload their data, register a feature view to point to the data, and then materialize the data. What is unnatural in this case is that the user has to manage their own offline store, instead of the feature store providing it to them.

Pseudo code

# load dataframe
df = pd.read_csv("my_data.csv")

# create feature table
ft = FeatureTable.infer_from_df(df)

# register feature table
fs.apply(ft)

# ingest/load data
ft.ingest(df)

Alternatives

An alternative proposed by @animeshsingh is to only use feature views and to ask users to always bring their own sources. Users would be responsible for uploading their data to a source location. The benefit of this approach is that we introduce less concepts to Feast and keep our APIs simpler.