Skip to content

fix: Added Redshift and Spark typecheck to data_source event_timestamp_col inference #2389

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 8 commits into from
Mar 17, 2022
Merged

fix: Added Redshift and Spark typecheck to data_source event_timestamp_col inference #2389

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
68f6e5e
added Redshift typecheck to data_source event_timestamp_col inference
shaibruhis Mar 9, 2022
7f1f15f
address comments
shaibruhis Mar 9, 2022
f6e5907
moved non file data sources into their own test
shaibruhis Mar 9, 2022
ca788ce
addressed comments
shaibruhis Mar 9, 2022
d6703b9
fixed texts
shaibruhis Mar 10, 2022
ef17d06
remove previously defined event_timestamp_column from data_source to …
shaibruhis Mar 10, 2022
87981e8
made a deepcopy of data_sources to not affect other tests
shaibruhis Mar 12, 2022
302e295
linter
shaibruhis Mar 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove previously defined event_timestamp_column from data_source to … 
…allow for inference

Signed-off-by: Shai Bruhis <shaibruhis@gmail.com>
  • Loading branch information
@shaibruhis
shaibruhis committed Mar 12, 2022
commit ef17d06cedd8e7761c8b9eb0aa510067bcbc58a2
3 changes: 3 additions & 0 deletions sdk/python/tests/integration/feature_repos/repo_configuration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,9 @@ class UniversalDataSources:
global_ds: DataSource
field_mapping: DataSource

def values(self):
return dataclasses.asdict(self).values()


def construct_universal_data_sources(
datasets: UniversalDatasets, data_source_creator: DataSourceCreator
Expand Down
10 changes: 7 additions & 3 deletions sdk/python/tests/integration/registration/test_inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,15 +143,19 @@ def test_update_file_data_source_with_inferred_event_timestamp_col(simple_datase
def test_update_data_sources_with_inferred_event_timestamp_col(universal_data_sources):
(_, _, data_sources) = universal_data_sources
Comment thread
adchia marked this conversation as resolved.
Outdated

# remove defined event_timestamp_column to allow for inference
for data_source in data_sources.values():
data_source.event_timestamp_column = None

update_data_sources_with_inferred_event_timestamp_col(
Comment thread
adchia marked this conversation as resolved.
Outdated
universal_data_sources.asdict().values(),
data_sources.values(),
RepoConfig(provider="local", project="test"),
)
actual_event_timestamp_cols = [
source.event_timestamp_column for source in data_sources
source.event_timestamp_column for source in data_sources.values()
]

assert actual_event_timestamp_cols == ["event_timestamp" * len(data_sources)]
assert actual_event_timestamp_cols == ["event_timestamp"] * len(data_sources.values())


def test_on_demand_features_type_inference():
Expand Down