Skip to content

Commit bf557bc

Browse files
Tsotne Tabidzewoop
Tsotne Tabidze
and
woop
authored
Implement Redshift historical retrieval (feast-dev#1720)
* Implement Redshift historical retrieval Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> * Fix imports Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> * Fixed get_historical_features where entity_df is a SQL query Fixed get_historical_features where entity_df is a SQL query, while keeping the utility functions common between Redshift and BigQuery. `infer_event_timestamp_from_entity_df` and `assert_expected_columns_in_entity_df` are now based on the entity schema rather than the dataframe. I also completely removed the min/max timestamp inference, since those could not be merged (needed to query BigQuery and Redshift). Instead, I moved the logic inside the SQL templates, reducing the code complexity. Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> * Address most of the comments Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> * Update sdk/python/feast/infra/offline_stores/redshift.py Co-authored-by: Willem Pienaar <6728866+woop@users.noreply.github.com> Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> * Merge common_utils and helpers into utils.py Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> * Add test_historical_retrieval test for Redshift, fix some bugs Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> * Use features instead of feature_refs Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> * Rename utils to offline_utils and add created_timestamp_column Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Co-authored-by: Willem Pienaar <6728866+woop@users.noreply.github.com>
1 parent 9c5a961 commit bf557bc

File tree

15 files changed

+949
-383
lines changed

15 files changed

+949
-383
lines changed

‎sdk/python/feast/data_source.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -376,6 +376,10 @@ def get_table_column_names_and_types(
376376
"""
377377
raise NotImplementedError
378378

379+
def get_table_query_string(self) -> str:
380+
"""Returns a string that can directly be used to reference this table in SQL"""
381+
raise NotImplementedError
382+
379383

380384
class KafkaSource(DataSource):
381385
def validate(self, config: RepoConfig):

‎sdk/python/feast/driver_test_data.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,9 @@
55
import pandas as pd
66
from pytz import FixedOffset, timezone, utc
77

8-
from feast.infra.provider import DEFAULT_ENTITY_DF_EVENT_TIMESTAMP_COL
8+
from feast.infra.offline_stores.offline_utils import (
9+
DEFAULT_ENTITY_DF_EVENT_TIMESTAMP_COL,
10+
)
911

1012

1113
class EventTimestampType(Enum):

‎sdk/python/feast/errors.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,3 +185,18 @@ def __init__(self):
185185
class RedshiftQueryError(Exception):
186186
def __init__(self, details):
187187
super().__init__(f"Redshift SQL Query failed to finish. Details: {details}")
188+
189+
190+
class EntityTimestampInferenceException(Exception):
191+
def __init__(self, expected_column_name: str):
192+
super().__init__(
193+
f"Please provide an entity_df with a column named {expected_column_name} representing the time of events."
194+
)
195+
196+
197+
class InvalidEntityType(Exception):
198+
def __init__(self, entity_type: type):
199+
super().__init__(
200+
f"The entity dataframe you have provided must be a Pandas DataFrame or a SQL query, "
201+
f"but we found: {entity_type} "
202+
)

‎sdk/python/feast/infra/aws.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
from feast import FeatureTable
88
from feast.entity import Entity
99
from feast.feature_view import FeatureView
10-
from feast.infra.offline_stores.helpers import get_offline_store_from_config
10+
from feast.infra.offline_stores.offline_utils import get_offline_store_from_config
1111
from feast.infra.online_stores.helpers import get_online_store_from_config
1212
from feast.infra.provider import (
1313
Provider,

‎sdk/python/feast/infra/gcp.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
from feast import FeatureTable
88
from feast.entity import Entity
99
from feast.feature_view import FeatureView
10-
from feast.infra.offline_stores.helpers import get_offline_store_from_config
10+
from feast.infra.offline_stores.offline_utils import get_offline_store_from_config
1111
from feast.infra.online_stores.helpers import get_online_store_from_config
1212
from feast.infra.provider import (
1313
Provider,

‎sdk/python/feast/infra/local.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
from feast import FeatureTable
99
from feast.entity import Entity
1010
from feast.feature_view import FeatureView
11-
from feast.infra.offline_stores.helpers import get_offline_store_from_config
11+
from feast.infra.offline_stores.offline_utils import get_offline_store_from_config
1212
from feast.infra.online_stores.helpers import get_online_store_from_config
1313
from feast.infra.provider import (
1414
Provider,

0 commit comments

Comments
 (0)