GitBook: [#313] Instructions for custom data sources

adchia · gitbook-bot · commit ec4c02bb149b · 2021-10-20T21:23:37.000Z
diff --git a/docs/how-to-guides/adding-a-new-offline-store.md b/docs/how-to-guides/adding-a-new-offline-store.md
@@ -2,7 +2,7 @@
 
 ## Overview
 
-Feast makes adding support for a new offline store (database) easy. Developers can simply implement the [OfflineStore](../../sdk/python/feast/infra/offline_stores/offline_store.py#L41) interface to add support for a new store (other than the existing stores like Parquet files, Redshift, and Bigquery). 
+Feast makes adding support for a new offline store (database) easy. Developers can simply implement the [OfflineStore](../../sdk/python/feast/infra/offline\_stores/offline\_store.py#L41) interface to add support for a new store (other than the existing stores like Parquet files, Redshift, and Bigquery).&#x20;
 
 In this guide, we will show you how to extend the existing File offline store and use in a feature repo. While we will be implementing a specific store, this guide should be representative for adding support for any new offline store.
 
@@ -13,15 +13,16 @@ The process for using a custom offline store consists of 4 steps:
 1. Defining an `OfflineStore` class.
 2. Defining an `OfflineStoreConfig` class.
 3. Defining a `RetrievalJob` class for this offline store.
-4. Referencing the `OfflineStore` in a feature repo's `feature_store.yaml` file.
+4. Defining a `DataSource` class for the offline store
+5. Referencing the `OfflineStore` in a feature repo's `feature_store.yaml` file.
 
 ## 1. Defining an OfflineStore class
 
 {% hint style="info" %}
- OfflineStore class names must end with the OfflineStore suffix!
+&#x20;OfflineStore class names must end with the OfflineStore suffix!
 {% endhint %}
 
-The OfflineStore class contains a couple of methods to read features from the offline store. Unlike the OnlineStore class, Feast does not manage any infrastructure for the offline store. 
+The OfflineStore class contains a couple of methods to read features from the offline store. Unlike the OnlineStore class, Feast does not manage any infrastructure for the offline store.&#x20;
 
 There are two methods that deal with reading data from the offline stores`get_historical_features`and `pull_latest_from_table_or_query`.
 
@@ -72,11 +73,11 @@ There are two methods that deal with reading data from the offline stores`get_hi
 
 Additional configuration may be needed to allow the OfflineStore to talk to the backing store. For example, Redshift needs configuration information like the connection information for the Redshift instance, credentials for connecting to the database, etc.
 
-To facilitate configuration, all OfflineStore implementations are **required** to also define a corresponding OfflineStoreConfig class in the same file. This OfflineStoreConfig class should inherit from the `FeastConfigBaseModel` class, which is defined [here](../../sdk/python/feast/repo_config.py#L44). 
+To facilitate configuration, all OfflineStore implementations are **required** to also define a corresponding OfflineStoreConfig class in the same file. This OfflineStoreConfig class should inherit from the `FeastConfigBaseModel` class, which is defined [here](../../sdk/python/feast/repo\_config.py#L44).&#x20;
 
 The `FeastConfigBaseModel` is a [pydantic](https://pydantic-docs.helpmanual.io) class, which parses yaml configuration into python objects. Pydantic also allows the model classes to define validators for the config classes, to make sure that the config classes are correctly defined.
 
-This config class **must** container a `type` field, which contains the fully qualified class name of its corresponding OfflineStore class. 
+This config class **must** container a `type` field, which contains the fully qualified class name of its corresponding OfflineStore class.&#x20;
 
 Additionally, the name of the config class must be the same as the OfflineStore class, with the `Config` suffix.
 
@@ -102,7 +103,7 @@ This configuration can be specified in the `feature_store.yaml` as follows:
 ```
 {% endcode %}
 
-This configuration information is available to the methods of the OfflineStore, via the`config: RepoConfig` parameter which is passed into the methods of the OfflineStore interface, specifically at the `config.offline_store` field of the `config` parameter. 
+This configuration information is available to the methods of the OfflineStore, via the`config: RepoConfig` parameter which is passed into the methods of the OfflineStore interface, specifically at the `config.offline_store` field of the `config` parameter.&#x20;
 
 {% code title="feast_custom_offline_store/file.py" %}
 ```python
@@ -153,9 +154,69 @@ class CustomFileRetrievalJob(RetrievalJob):
 ```
 {% endcode %}
 
-## 4. Using the custom offline store 
+## 4. Defining a DataSource class for the offline store
 
-After implementing these classes, the custom offline store can be used by referencing it in a feature repo's `feature_store.yaml` file, specifically in the `offline_store` field. The value specified should be the fully qualified class name of the OfflineStore. 
+Before this offline store can be used as the batch source for a feature view in a feature repo, a subclass of the `DataSource` [base class](https://rtd.feast.dev/en/master/index.html?highlight=DataSource#feast.data\_source.DataSource) needs to be defined. This class is responsible for holding information needed by specific feature views to support reading historical values from the offline store. For example, a feature view using Redshift as the offline store may need to know which table contains historical feature values.
+
+The data source class should implement two methods - `from_proto`, and `to_proto`.&#x20;
+
+For custom offline stores that are not being implemented in the main feature repo, the `custom_options` field should be used to store any configuration needed by the data source. In this case, the implementer is responsible for serializing this configuration into bytes in the `to_proto` method and reading the value back from bytes in the `from_proto` method.
+
+{% code title="feast_custom_offline_store/file.py" %}
+```python
+class CustomFileDataSource(FileSource):
+    """Custom data source class for local files"""
+    def __init__(
+        self,
+        event_timestamp_column: Optional[str] = "",
+        path: Optional[str] = None,
+        field_mapping: Optional[Dict[str, str]] = None,
+        created_timestamp_column: Optional[str] = "",
+        date_partition_column: Optional[str] = "",
+    ):
+        super(CustomFileDataSource, self).__init__(
+            event_timestamp_column,
+            created_timestamp_column,
+            field_mapping,
+            date_partition_column,
+        )
+        self._path = path
+
+
+    @staticmethod
+    def from_proto(data_source: DataSourceProto):
+        custom_source_options = str(
+            data_source.custom_options.configuration, encoding="utf8"
+        )
+        path = json.loads(custom_source_options)["path"]
+        return CustomFileDataSource(
+            field_mapping=dict(data_source.field_mapping),
+            path=path,
+            event_timestamp_column=data_source.event_timestamp_column,
+            created_timestamp_column=data_source.created_timestamp_column,
+            date_partition_column=data_source.date_partition_column,
+        )
+
+    def to_proto(self) -> DataSourceProto:
+        config_json = json.dumps({"path": self.path})
+        data_source_proto = DataSourceProto(
+            type=DataSourceProto.CUSTOM_SOURCE,
+            custom_options=DataSourceProto.CustomSourceOptions(
+                configuration=bytes(config_json, encoding="utf8")
+            ),
+        )
+
+        data_source_proto.event_timestamp_column = self.event_timestamp_column
+        data_source_proto.created_timestamp_column = self.created_timestamp_column
+        data_source_proto.date_partition_column = self.date_partition_column
+
+        return data_source_proto
+```
+{% endcode %}
+
+## 5. Using the custom offline store&#x20;
+
+After implementing these classes, the custom offline store can be used by referencing it in a feature repo's `feature_store.yaml` file, specifically in the `offline_store` field. The value specified should be the fully qualified class name of the OfflineStore.&#x20;
 
 As long as your OfflineStore class is available in your Python environment, it will be imported by Feast dynamically at runtime.
 
@@ -181,3 +242,22 @@ provider: local
 offline_store: feast_custom_offline_store.file.CustomFileOfflineStore
 ```
 {% endcode %}
+
+Finally, the custom data source class can be use in the feature repo to define a data source, and refer to in a feature view definition.
+
+{% code title="feature_repo/repo.py" %}
+```python
+pdriver_hourly_stats = CustomFileDataSource(
+    path="feature_repo/data/driver_stats.parquet",
+    event_timestamp_column="event_timestamp",
+    created_timestamp_column="created",
+)
+
+
+driver_hourly_stats_view = FeatureView(
+    batch_source=driver_hourly_stats,
+    ...
+)
+
+```
+{% endcode %}
