Skip to content

feat: Add interfaces for batch materialization engine #2901

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 11 commits into from
Jul 6, 2022
Merged

feat: Add interfaces for batch materialization engine #2901

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
6cc7e75
feat: Add scaffolding for batch materialization engine
achals Jul 1, 2022
1656436
fix tests
achals Jul 5, 2022
c16734b
fix tests
achals Jul 5, 2022
ceedbab
a little better
achals Jul 5, 2022
37fa081
a little better
achals Jul 5, 2022
795e65a
docs
achals Jul 5, 2022
5833556
more api updates'
achals Jul 5, 2022
5d1af33
fix typos
achals Jul 6, 2022
ff680a9
make engine importable
achals Jul 6, 2022
11b6da0
style stuff
achals Jul 6, 2022
2e9e7a8
style stuff
achals Jul 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
make engine importable 
Signed-off-by: Achal Shah <achals@gmail.com>
  • Loading branch information
@achals
achals committed Jul 6, 2022
commit ff680a927bd4c781ac0965d67989ae7f4449efb8
38 changes: 31 additions & 7 deletions sdk/python/feast/infra/passthrough_provider.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import pyarrow as pa
from tqdm import tqdm

from feast import importer
from feast.batch_feature_view import BatchFeatureView
from feast.entity import Entity
from feast.feature_logging import FeatureServiceLoggingSource
Expand Down Expand Up @@ -72,13 +73,36 @@ def batch_engine(self) -> BatchMaterializationEngine:
if self._batch_engine:
return self._batch_engine
else:
from feast.infra.materialization import LocalMaterializationEngine

_batch_engine = LocalMaterializationEngine(
repo_config=self.repo_config,
offline_store=self.offline_store,
online_store=self.online_store,
)
engine_config = self.repo_config.batch_engine_config
config_is_dict = False
if isinstance(engine_config, str):
engine_config_type = engine_config
elif isinstance(engine_config, Dict):
if "type" not in engine_config:
raise ValueError("engine_config needs to have a `type` specified.")
engine_config_type = engine_config["type"]
config_is_dict = True
else:
raise RuntimeError(f"Invalid config type specified for batch_engine: {type(engine_config)}")

if engine_config_type in BATCH_ENGINE_CLASS_FOR_TYPE:
engine_config_type = BATCH_ENGINE_CLASS_FOR_TYPE[engine_config_type]
engine_module, engine_class = engine_config_type.rsplit('.', 1)
engine_class = importer.import_class(engine_module, engine_class)

if config_is_dict:
_batch_engine = engine_class(
repo_config=self.repo_config,
offline_store=self.offline_store,
online_store=self.online_store,
**engine_config
)
else:
_batch_engine = engine_class(
repo_config=self.repo_config,
offline_store=self.offline_store,
online_store=self.online_store,
)
self._batch_engine = _batch_engine
return _batch_engine

Expand Down