Skip to content

Schema Inferencing should happen at apply time #1646

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 8 commits into from
Jun 18, 2021
Merged

Schema Inferencing should happen at apply time #1646

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ed8d461
wip1
mavysavydav Jun 12, 2021
91ef76b
just need to do clean up
mavysavydav Jun 12, 2021
0dab89c
linted
mavysavydav Jun 12, 2021
b24e571
improve test coverage
mavysavydav Jun 13, 2021
bb3a08a
changed placement of inference methods in repo_operation apply_total
mavysavydav Jun 15, 2021
3053599
updated inference method name + changed to void return since it updat…
mavysavydav Jun 15, 2021
d558b0b
fixed integration test and added comments
mavysavydav Jun 15, 2021
9a854bf
Made DataSource event_timestamp_column optional
mavysavydav Jun 18, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixed integration test and added comments 
Signed-off-by: David Y Liu <davidyliuliu@gmail.com>
  • Loading branch information
@mavysavydav
mavysavydav committed Jun 17, 2021
commit d558b0b6b657a15c516a3589a8cbefb9aa88fffe
8 changes: 4 additions & 4 deletions sdk/python/feast/data_source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -523,7 +523,7 @@ def to_proto(self) -> DataSourceProto:
class FileSource(DataSource):
def __init__(
self,
event_timestamp_column: Optional[str] = None,
event_timestamp_column: Optional[str] = "",
file_url: Optional[str] = None,
path: Optional[str] = None,
file_format: FileFormat = None,
Expand Down Expand Up @@ -563,7 +563,7 @@ def __init__(
self._file_options = FileOptions(file_format=file_format, file_url=file_url)

super().__init__(
event_timestamp_column or "",
event_timestamp_column or "", # for satisfying type checker
created_timestamp_column,
field_mapping,
date_partition_column,
Expand Down Expand Up @@ -627,7 +627,7 @@ def get_table_column_names_and_types(self) -> Iterable[Tuple[str, str]]:
class BigQuerySource(DataSource):
def __init__(
self,
event_timestamp_column: Optional[str] = None,
event_timestamp_column: Optional[str] = "",
table_ref: Optional[str] = None,
created_timestamp_column: Optional[str] = "",
field_mapping: Optional[Dict[str, str]] = None,
Expand All @@ -637,7 +637,7 @@ def __init__(
self._bigquery_options = BigQueryOptions(table_ref=table_ref, query=query)

super().__init__(
event_timestamp_column or "",
event_timestamp_column or "", # for satisfying type checker
created_timestamp_column,
field_mapping,
date_partition_column,
Expand Down
15 changes: 7 additions & 8 deletions sdk/python/tests/test_inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,14 @@ def test_infer_event_timestamp_column_for_data_source(simple_dataset_1):
df_with_two_viable_timestamp_cols["ts_2"] = simple_dataset_1["ts_1"]

with prep_file_source(df=simple_dataset_1) as file_source:
actual_processed_data_sources = update_data_sources_with_inferred_event_timestamp_col(
[
file_source,
simple_bq_source_using_table_ref_arg(simple_dataset_1),
simple_bq_source_using_query_arg(simple_dataset_1),
]
)
data_sources = [
file_source,
simple_bq_source_using_table_ref_arg(simple_dataset_1),
simple_bq_source_using_query_arg(simple_dataset_1),
]
update_data_sources_with_inferred_event_timestamp_col(data_sources)
actual_event_timestamp_cols = [
source.event_timestamp_column for source in actual_processed_data_sources
source.event_timestamp_column for source in data_sources
]

assert actual_event_timestamp_cols == ["ts_1", "ts_1", "ts_1"]
Expand Down