Expected Behavior
- if Feast has data with same keys, then run
get_historical_features' get latest data by created time'
- but it doesn't work expected way
Current Behavior
- if duplicated data on same keys are over 3 copies (diffrent 'created time') , feast cannot give right result
Steps to reproduce
data = pd.read_parquet("data/driver_stats.parquet")
new_data = data.copy()
new_data['avg_daily_trips'] = new_data['avg_daily_trips'] + 1000
new_data['event_timestamp'] = new_data['event_timestamp']
new_data['created'] = pd.Timestamp.now()
merged_df = pd.concat([data, new_data]).reset_index(drop=True)
merged_df.to_parquet("./data/driver_stats.parquet")
store = FeatureStore(".")
entity_df = pd.DataFrame.from_dict(
{
# entity's join key -> entity values
"driver_id": [1001, 1002, 1003],
# "event_timestamp" (reserved key) -> timestamps
"event_timestamp": [
datetime(2022, 11, 20, 23, 00, 42),
datetime(2022, 11, 20, 23, 00, 42),
datetime(2022, 11, 20, 23, 00, 42),
]
}
)
training_df = store.get_historical_features(
entity_df=entity_df,
features=[
"driver_hourly_stats:conv_rate",
"driver_hourly_stats:acc_rate",
"driver_hourly_stats:avg_daily_trips"
],
).to_df()
training_df # it is right, result has (plus+1000) datas
new_data = data.copy()
new_data['avg_daily_trips'] = new_data['avg_daily_trips'] + 10000
new_data['event_timestamp'] = new_data['event_timestamp']
new_data['created'] = pd.Timestamp.now()
# this "merged_df" which is merged by pd.concat is above one.
merged_df = pd.concat([merged_df, new_data]).reset_index(drop=True)
merged_df.to_parquet("./data/driver_stats.parquet")
store = FeatureStore(".")
entity_df = pd.DataFrame.from_dict(
{
# entity's join key -> entity values
"driver_id": [1001, 1002, 1003],
# "event_timestamp" (reserved key) -> timestamps
"event_timestamp": [
datetime(2022, 11, 20, 23, 00, 42),
datetime(2022, 11, 20, 23, 00, 42),
datetime(2022, 11, 20, 23, 00, 42),
]
}
)
training_df = store.get_historical_features(
entity_df=entity_df,
features=[
"driver_hourly_stats:conv_rate",
"driver_hourly_stats:acc_rate",
"driver_hourly_stats:avg_daily_trips"
],
).to_df()
training_df
above is wrong, in my pc result is
Specifications
- Version: Feast SDK Version: "feast 0.26.0"
- Platform: M1 mac
- Subsystem: local
Possible Solution
- This problem seems to have arisen because of pandas sorting & drop_duplicates. but i'm not sure.
- if it is not a but, let me know how to fix that.
Expected Behavior
get_historical_features' get latest data bycreated time'Current Behavior
Steps to reproduce
above is wrong, in my pc result is
Specifications
Possible Solution