Skip to content

chore: Update offline and online store docs #3048

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 4 commits into from
Aug 12, 2022
Merged

chore: Update offline and online store docs #3048

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c77984c
Fix
kevjumba Aug 9, 2022
7ed3676
Fix
kevjumba Aug 9, 2022
d5ee116
Fix
kevjumba Aug 12, 2022
bb86857
Fix
kevjumba Aug 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Fix 
Signed-off-by: Kevin Zhang <kzhang@tecton.ai>
  • Loading branch information
@kevjumba
kevjumba committed Aug 12, 2022
commit c77984c292bc73448aa37f7864a344db63818111
26 changes: 25 additions & 1 deletion docs/how-to-guides/customizing-feast/adding-a-new-offline-store.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -406,7 +406,31 @@ Even if you have created the `OfflineStore` class in a separate repo, you can st
```

If the integration tests fail, this indicates that there is a mistake in the implementation of this offline store!
5. Remember to add your datasource to `repo_config.py` similar to how we added `spark`, `trino`, etc, to the dictionary `OFFLINE_STORE_CLASS_FOR_TYPE` and add the necessary configuration to `repo_configuration.py`. Namely, `AVAILABLE_OFFLINE_STORES` should load your repo configuration module.

5. Remember to add your datasource to `repo_config.py` similar to how we added `spark`, `trino`, etc, to the dictionary `OFFLINE_STORE_CLASS_FOR_TYPE`. This will allow Feast to load your class from the `feature_store.yaml`.

6. Finally, add a Makefile target to the Makefile to run your datastore specific tests by setting the `FULL_REPO_CONFIGS_MODULE` and `PYTEST_PLUGINS` environment variables. You can remove certain tests that are not relevant or still do not work for your datastore using the `-k` option.

{% code title="Makefile" %}
```Makefile
test-python-universal-postgres:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.postgres_repo_configuration \
PYTEST_PLUGINS=sdk.python.feast.infra.offline_stores.contrib.postgres_offline_store.tests \
FEAST_USAGE=False \
IS_TEST=True \
python -m pytest -x --integration \
-k "not test_historical_retrieval_fails_on_validation and \
not test_historical_retrieval_with_validation and \
not test_historical_features_persisting and \
not test_historical_retrieval_fails_on_validation and \
not test_universal_cli and \
not test_go_feature_server and \
not test_feature_logging and \
not test_universal_types" \
sdk/python/tests
```
{% endcode %}

### 7. Dependencies

Expand Down
30 changes: 23 additions & 7 deletions docs/how-to-guides/customizing-feast/adding-support-for-a-new-online-store.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,7 @@ online_store: feast_custom_online_store.mysql.MySQLOnlineStore

## 4. Testing the OnlineStore class

### Integrating with the integration test suite and unit test suite.
### 4.1 Integrating with the integration test suite and unit test suite.

Even if you have created the `OnlineStore` class in a separate repo, you can still test your implementation against the Feast test suite, as long as you have Feast as a submodule in your repo.

Expand Down Expand Up @@ -352,7 +352,7 @@ If you are planning to start the online store up locally(e.g spin up a local Red
}
```

If you are planning instead to use a Dockerized container to run your tests against your online store, you can define a `OnlineStoreCreator` and replace the `None` object above with your `OnlineStoreCreator` class.
If you are planning instead to use a Dockerized container to run your tests against your online store, you can define a `OnlineStoreCreator` and replace the `None` object above with your `OnlineStoreCreator` class. You should make this class available to feast through the `PYTEST_PLUGINS` environment variable.

If you create a containerized docker image for testing, developers who are trying to test with your online store will not have to spin up their own instance of the online store for testing. An example of an `OnlineStoreCreator` is shown below:

Expand All @@ -372,12 +372,28 @@ class RedisOnlineStoreCreator(OnlineStoreCreator):
```
{% endcode %}

3\. You should swap out the `FULL_REPO_CONFIGS` environment variable and run the integration tests against your online store. In the example repo, the file that overwrites `FULL_REPO_CONFIGS` is `feast_custom_online_store/feast_tests.py`, so you would run:

```bash
export FULL_REPO_CONFIGS_MODULE='feast_custom_online_store.feast_tests'
make test-python-universal
3\. Add a Makefile target to the Makefile to run your datastore specific tests by setting the `FULL_REPO_CONFIGS_MODULE` and `PYTEST_PLUGINS` environment variables. You can remove certain tests that are not relevant or still do not work for your datastore using the `-k` option.

{% code title="Makefile" %}
```Makefile
test-python-universal-postgres:
Copy link
Copy Markdown
Collaborator

@adchia adchia Aug 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this is actually not a good example since the pytest plugin uses the offline store. Maybe sync and copy the cassandra one?

Copy link
Copy Markdown
Collaborator Author

@kevjumba kevjumba Aug 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.postgres_repo_configuration \
PYTEST_PLUGINS=sdk.python.feast.infra.offline_stores.contrib.postgres_offline_store.tests \
FEAST_USAGE=False \
IS_TEST=True \
python -m pytest -x --integration \
-k "not test_historical_retrieval_fails_on_validation and \
not test_historical_retrieval_with_validation and \
not test_historical_features_persisting and \
not test_historical_retrieval_fails_on_validation and \
not test_universal_cli and \
not test_go_feature_server and \
not test_feature_logging and \
not test_universal_types" \
sdk/python/tests
```
{% endcode %}

* If there are some tests that fail, this indicates that there is a mistake in the implementation of this online store!

Expand Down