Skip to content

fix: Added Redshift and Spark typecheck to data_source event_timestamp_col inference #2389

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 8 commits into from
Mar 17, 2022
Merged

fix: Added Redshift and Spark typecheck to data_source event_timestamp_col inference #2389

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
68f6e5e
added Redshift typecheck to data_source event_timestamp_col inference
shaibruhis Mar 9, 2022
7f1f15f
address comments
shaibruhis Mar 9, 2022
f6e5907
moved non file data sources into their own test
shaibruhis Mar 9, 2022
ca788ce
addressed comments
shaibruhis Mar 9, 2022
d6703b9
fixed texts
shaibruhis Mar 10, 2022
ef17d06
remove previously defined event_timestamp_column from data_source to …
shaibruhis Mar 10, 2022
87981e8
made a deepcopy of data_sources to not affect other tests
shaibruhis Mar 12, 2022
302e295
linter
shaibruhis Mar 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address comments 
Signed-off-by: Shai Bruhis <shaibruhis@gmail.com>
  • Loading branch information
@shaibruhis
shaibruhis committed Mar 12, 2022
commit 7f1f15f629f0d41f62c4de7bdf5eb68ba1e8e8df
1 change: 1 addition & 0 deletions sdk/python/feast/inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ def update_data_sources_with_inferred_event_timestamp_col(
or isinstance(data_source, BigQuerySource)
or isinstance(data_source, RedshiftSource)
Comment thread
adchia marked this conversation as resolved.
Outdated
or isinstance(data_source, SnowflakeSource)
or "SparkSource" == data_source.__class__.__name__
)

# loop through table columns to find singular match
Expand Down
14 changes: 13 additions & 1 deletion sdk/python/tests/integration/registration/test_inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,9 @@ def test_infer_datasource_names_dwh():


@pytest.mark.integration
def test_update_data_sources_with_inferred_event_timestamp_col(simple_dataset_1):
def test_update_data_sources_with_inferred_event_timestamp_col(
simple_dataset_1, universal_data_sources
):
df_with_two_viable_timestamp_cols = simple_dataset_1.copy(deep=True)
df_with_two_viable_timestamp_cols["ts_2"] = simple_dataset_1["ts_1"]

Expand All @@ -137,6 +139,16 @@ def test_update_data_sources_with_inferred_event_timestamp_col(simple_dataset_1)
[file_source], RepoConfig(provider="local", project="test")
)

(_, _, data_sources) = universal_data_sources
Comment thread
adchia marked this conversation as resolved.
Outdated
update_data_sources_with_inferred_event_timestamp_col(
Comment thread
adchia marked this conversation as resolved.
Outdated
data_sources, RepoConfig(provider="local", project="test")
)
actual_event_timestamp_cols = [
source.event_timestamp_column for source in data_sources
]

assert actual_event_timestamp_cols == ["event_timestamp" * len(data_sources)]


def test_on_demand_features_type_inference():
# Create Feature Views
Expand Down