Skip to content

Historical field mappings #2251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
michelle-rascati-sp wants to merge 17 commits into from
Closed

Historical field mappings #2251

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
2a7935b
historical_field_mappings fix CONTRIBUTING docs for creating venv
michelle-rascati-sp Jan 26, 2022
8b95281
historical_field_mappings fix mypy-protobuf conflict
michelle-rascati-sp Jan 26, 2022
6a8d126
historical_field_mappings adding tests for get_historical_features to…
michelle-rascati-sp Jan 27, 2022
99bf1ed
historical_field_mappings bigquery tests passing
michelle-rascati-sp Jan 27, 2022
a895594
historical_field_mappings redshift tests pass
michelle-rascati-sp Jan 27, 2022
12753f3
historical_field_mappings formatting
michelle-rascati-sp Jan 27, 2022
d15a210
historical_field_mappings make required so no .get() from None
michelle-rascati-sp Jan 27, 2022
ed011a3
historical_field_mappings type the registry so linter is happy
michelle-rascati-sp Jan 27, 2022
af8fc85
historical_field_mappings making pyling happy
michelle-rascati-sp Jan 27, 2022
e804fe9
historical_field_mappings formatting
michelle-rascati-sp Jan 27, 2022
b28aa79
historical_field_mappings Merge branch 'master' into historical_field…
michelle-rascati-sp Jan 27, 2022
0c1e79f
historical_field_mappings Revert "historical_field_mappings making py…
michelle-rascati-sp Jan 27, 2022
60e3692
historical_field_mappings redo FieldStatusValue
michelle-rascati-sp Jan 27, 2022
33dd5b9
historical_field_mappings already fixed upstream
michelle-rascati-sp Jan 27, 2022
c3c6747
historical_field_mappings formatting
michelle-rascati-sp Jan 27, 2022
2d43b74
historical_field_mappings remove unused import
michelle-rascati-sp Jan 27, 2022
cbcbed7
historical_field_mappings Revert "historical_field_mappings redo Fiel…
michelle-rascati-sp Jan 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
historical_field_mappings bigquery tests passing 
Signed-off-by: Michelle Rascati <michelle.rascati@sailpoint.com>
  • Loading branch information
@michelle-rascati-sp
michelle-rascati-sp committed Jan 27, 2022
commit 99bf1ed2cf143b0450965b2ec2df1321a2c12b0e
4 changes: 2 additions & 2 deletions sdk/python/feast/infra/offline_stores/bigquery.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -531,7 +531,7 @@ def _get_bigquery_client(project: Optional[str] = None, location: Optional[str]
{{ featureview.created_timestamp_column ~ ' as created_timestamp,' if featureview.created_timestamp_column else '' }}
{{ featureview.entity_selections | join(', ')}}{% if featureview.entity_selections %},{% else %}{% endif %}
{% for feature in featureview.features %}
{{ feature }} as {% if full_feature_names %}{{ featureview.name }}__{{feature}}{% else %}{{ feature }}{% endif %}{% if loop.last %}{% else %}, {% endif %}
{{ feature }} as {% if full_feature_names %}{{ featureview.name }}__{{featureview.field_mapping.get(feature, feature)}}{% else %}{{ featureview.field_mapping.get(feature, feature) }}{% endif %}{% if loop.last %}{% else %}, {% endif %}
{% endfor %}
FROM {{ featureview.table_subquery }}
WHERE {{ featureview.event_timestamp_column }} <= '{{ featureview.max_event_timestamp }}'
Expand Down Expand Up @@ -632,7 +632,7 @@ def _get_bigquery_client(project: Optional[str] = None, location: Optional[str]
SELECT
{{featureview.name}}__entity_row_unique_id
{% for feature in featureview.features %}
,{% if full_feature_names %}{{ featureview.name }}__{{feature}}{% else %}{{ feature }}{% endif %}
,{% if full_feature_names %}{{ featureview.name }}__{{featureview.field_mapping.get(feature, feature)}}{% else %}{{ featureview.field_mapping.get(feature, feature) }}{% endif %}
{% endfor %}
FROM {{ featureview.name }}__cleaned
) USING ({{featureview.name}}__entity_row_unique_id)
Expand Down
5 changes: 4 additions & 1 deletion sdk/python/feast/infra/offline_stores/offline_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ class FeatureViewQueryContext:
ttl: int
entities: List[str]
features: List[str] # feature reference format
field_mapping: Optional[Dict[str, str]]
event_timestamp_column: str
created_timestamp_column: Optional[str]
table_subquery: str
Expand Down Expand Up @@ -145,6 +146,7 @@ def get_feature_view_query_context(
ttl=ttl_seconds,
entities=join_keys,
features=[reverse_field_mapping.get(feature, feature) for feature in features],
field_mapping=feature_view.input.field_mapping,
event_timestamp_column=reverse_field_mapping.get(
event_timestamp_column, event_timestamp_column
),
Expand Down Expand Up @@ -175,7 +177,8 @@ def build_point_in_time_query(
final_output_feature_names = list(entity_df_columns)
final_output_feature_names.extend(
[
(f"{fv.name}__{feature}" if full_feature_names else feature)
(f"{fv.name}__{fv.field_mapping.get(feature, feature)}" if full_feature_names
else fv.field_mapping.get(feature, feature))
for fv in feature_view_query_contexts
for feature in fv.features
]
Expand Down
1 change: 1 addition & 0 deletions sdk/python/tests/integration/offline_store/test_universal_historical_retrieval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -399,6 +399,7 @@ def test_historical_features(environment, universal_data_sources, full_feature_n
"order:order_is_success",
"global_stats:num_rides",
"global_stats:avg_ride_length",
"field_mapping:feature_name"
],
full_feature_names=full_feature_names,
)
Expand Down