Skip to content

fix: Request data api update #2488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 14 commits into from
Apr 5, 2022
Merged

fix: Request data api update #2488

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
7cea469
Update request data source
kevjumba Apr 4, 2022
7b7395c
Fix
kevjumba Apr 4, 2022
945e0f8
Fix
kevjumba Apr 4, 2022
19158f8
Fix
kevjumba Apr 4, 2022
b284d5f
Data source update
kevjumba Apr 5, 2022
c90a224
Data source
kevjumba Apr 5, 2022
a362e9f
Temp fix
kevjumba Apr 5, 2022
f3c9d7a
Fix
kevjumba Apr 5, 2022
b442599
Update internal var names
kevjumba Apr 5, 2022
af8e98f
Add test
kevjumba Apr 5, 2022
b28932d
Fix
kevjumba Apr 5, 2022
9ecf9c8
Fix
kevjumba Apr 5, 2022
61230c8
Fix
kevjumba Apr 5, 2022
340b98a
Fix
kevjumba Apr 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/getting-started/concepts/feature-view.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,9 +130,9 @@ Feature names must be unique within a [feature view](feature-view.md#feature-vie
On demand feature views allows users to use existing features and request time data (features only available at request time) to transform and create new features. Users define python transformation logic which is executed in both historical retrieval and online retrieval paths:

```python
# Define a request data source which encodes features / information only
# Define a request data source which encodes features / information only
# available at request time (e.g. part of the user initiated HTTP request)
input_request = RequestDataSource(
input_request = RequestSource(
name="vals_to_add",
schema={
"val_to_add": ValueType.INT64,
Expand Down
20 changes: 14 additions & 6 deletions sdk/python/feast/data_source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.


import enum
import warnings
from abc import ABC, abstractmethod
Expand Down Expand Up @@ -144,7 +143,7 @@ def to_proto(self) -> DataSourceProto.KinesisOptions:
DataSourceProto.SourceType.BATCH_SNOWFLAKE: "feast.infra.offline_stores.snowflake_source.SnowflakeSource",
DataSourceProto.SourceType.STREAM_KAFKA: "feast.data_source.KafkaSource",
DataSourceProto.SourceType.STREAM_KINESIS: "feast.data_source.KinesisSource",
DataSourceProto.SourceType.REQUEST_SOURCE: "feast.data_source.RequestDataSource",
DataSourceProto.SourceType.REQUEST_SOURCE: "feast.data_source.RequestSource",
DataSourceProto.SourceType.PUSH_SOURCE: "feast.data_source.PushSource",
}

Expand Down Expand Up @@ -422,9 +421,9 @@ def get_table_query_string(self) -> str:
raise NotImplementedError


class RequestDataSource(DataSource):
class RequestSource(DataSource):
Comment thread
kevjumba marked this conversation as resolved.
Outdated
"""
RequestDataSource that can be used to provide input features for on demand transforms
RequestSource that can be used to provide input features for on demand transforms

Args:
name: Name of the request data source
Expand All @@ -446,7 +445,7 @@ def __init__(
tags: Optional[Dict[str, str]] = None,
owner: Optional[str] = "",
):
"""Creates a RequestDataSource object."""
"""Creates a RequestSource object."""
super().__init__(name=name, description=description, tags=tags, owner=owner)
self.schema = schema

Expand All @@ -464,7 +463,7 @@ def from_proto(data_source: DataSourceProto):
schema = {}
for key, val in schema_pb.items():
schema[key] = ValueType(val)
return RequestDataSource(
return RequestSource(
name=data_source.name,
schema=schema,
description=data_source.description,
Expand Down Expand Up @@ -496,6 +495,15 @@ def source_datatype_to_feast_value_type() -> Callable[[str], ValueType]:
raise NotImplementedError


class RequestDataSource(RequestSource):
Copy link
Copy Markdown
Member

@achals achals Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also keep a test for RequestDataSource to make sure that it continues to work, and prints the deprecation warning? I think there's examples of this in the unit tests already.

Copy link
Copy Markdown
Collaborator Author

@kevjumba kevjumba Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup Added.

def __init__(self, *args, **kwargs):
warnings.warn(
"The 'RequestDataSource' class is deprecated and was renamed to RequestSource. Please use RequestSource instead. This class name will be removed in Feast 0.23.",
DeprecationWarning,
)
RequestSource.__init__(*args, **kwargs)


class KinesisSource(DataSource):
def validate(self, config: RepoConfig):
pass
Expand Down
4 changes: 2 additions & 2 deletions sdk/python/feast/inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
RedshiftSource,
SnowflakeSource,
)
from feast.data_source import DataSource, RequestDataSource
from feast.data_source import DataSource, RequestSource
from feast.errors import RegistryInferenceFailure
from feast.feature_view import FeatureView
from feast.repo_config import RepoConfig
Expand Down Expand Up @@ -78,7 +78,7 @@ def update_data_sources_with_inferred_event_timestamp_col(
ERROR_MSG_PREFIX = "Unable to infer DataSource event_timestamp_column"

for data_source in data_sources:
if isinstance(data_source, RequestDataSource):
if isinstance(data_source, RequestSource):
continue
if (
data_source.event_timestamp_column is None
Expand Down
18 changes: 9 additions & 9 deletions sdk/python/feast/on_demand_feature_view.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import pandas as pd

from feast.base_feature_view import BaseFeatureView
from feast.data_source import RequestDataSource
from feast.data_source import RequestSource
from feast.errors import RegistryInferenceFailure, SpecifiedFeaturesNotPresentError
from feast.feature import Feature
from feast.feature_view import FeatureView
Expand Down Expand Up @@ -46,7 +46,7 @@ class OnDemandFeatureView(BaseFeatureView):
source_feature_view_projections: A map from input source names to actual input
sources with type FeatureViewProjection.
source_request_data_sources: A map from input source names to the actual input
sources with type RequestDataSource.
sources with type RequestSource.
udf: The user defined transformation function, which must take pandas dataframes
as inputs.
description: A human-readable description.
Expand All @@ -59,7 +59,7 @@ class OnDemandFeatureView(BaseFeatureView):
name: str
features: List[Feature]
source_feature_view_projections: Dict[str, FeatureViewProjection]
source_request_data_sources: Dict[str, RequestDataSource]
source_request_data_sources: Dict[str, RequestSource]
udf: MethodType
description: str
tags: Dict[str, str]
Expand All @@ -71,11 +71,11 @@ def __init__(
name: str,
features: List[Feature],
sources: Optional[
Dict[str, Union[FeatureView, FeatureViewProjection, RequestDataSource]]
Dict[str, Union[FeatureView, FeatureViewProjection, RequestSource]]
] = None,
udf: Optional[MethodType] = None,
inputs: Optional[
Dict[str, Union[FeatureView, FeatureViewProjection, RequestDataSource]]
Dict[str, Union[FeatureView, FeatureViewProjection, RequestSource]]
] = None,
description: str = "",
tags: Optional[Dict[str, str]] = None,
Expand Down Expand Up @@ -124,9 +124,9 @@ def __init__(

assert sources is not None
self.source_feature_view_projections: Dict[str, FeatureViewProjection] = {}
self.source_request_data_sources: Dict[str, RequestDataSource] = {}
self.source_request_data_sources: Dict[str, RequestSource] = {}
for source_name, odfv_source in sources.items():
if isinstance(odfv_source, RequestDataSource):
if isinstance(odfv_source, RequestSource):
self.source_request_data_sources[source_name] = odfv_source
elif isinstance(odfv_source, FeatureViewProjection):
self.source_feature_view_projections[source_name] = odfv_source
Expand Down Expand Up @@ -241,7 +241,7 @@ def from_proto(cls, on_demand_feature_view_proto: OnDemandFeatureViewProto):
on_demand_source.feature_view_projection
)
else:
sources[source_name] = RequestDataSource.from_proto(
sources[source_name] = RequestSource.from_proto(
on_demand_source.request_data_source
)
on_demand_feature_view_obj = cls(
Expand Down Expand Up @@ -385,7 +385,7 @@ def get_requested_odfvs(feature_refs, project, registry):


def on_demand_feature_view(
features: List[Feature], sources: Dict[str, Union[FeatureView, RequestDataSource]]
features: List[Feature], sources: Dict[str, Union[FeatureView, RequestSource]]
):
"""
Declare an on-demand feature view
Expand Down
8 changes: 4 additions & 4 deletions sdk/python/feast/request_feature_view.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
from typing import Dict, List, Optional, Type

from feast.base_feature_view import BaseFeatureView
from feast.data_source import RequestDataSource
from feast.data_source import RequestSource
from feast.feature import Feature
from feast.feature_view_projection import FeatureViewProjection
from feast.protos.feast.core.RequestFeatureView_pb2 import (
Expand All @@ -30,7 +30,7 @@ class RequestFeatureView(BaseFeatureView):
"""

name: str
request_data_source: RequestDataSource
request_data_source: RequestSource
Copy link
Copy Markdown
Member

@achals achals Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change field name?

Copy link
Copy Markdown
Collaborator Author

@kevjumba kevjumba Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed

features: List[Feature]
description: str
tags: Dict[str, str]
Expand All @@ -40,7 +40,7 @@ class RequestFeatureView(BaseFeatureView):
def __init__(
self,
name: str,
request_data_source: RequestDataSource,
request_data_source: RequestSource,
Copy link
Copy Markdown
Member

@achals achals Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change field name? also, change to use kwargs? Or is that a different PR?

Copy link
Copy Markdown
Collaborator Author

@kevjumba kevjumba Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a different PR, I am not deprecating feature view params here.

description: str = "",
tags: Optional[Dict[str, str]] = None,
owner: str = "",
Expand Down Expand Up @@ -110,7 +110,7 @@ def from_proto(cls, request_feature_view_proto: RequestFeatureViewProto):

request_feature_view_obj = cls(
name=request_feature_view_proto.spec.name,
request_data_source=RequestDataSource.from_proto(
request_data_source=RequestSource.from_proto(
request_feature_view_proto.spec.request_data_source
),
description=request_feature_view_proto.spec.description,
Expand Down
12 changes: 5 additions & 7 deletions sdk/python/tests/integration/feature_repos/universal/feature_views.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
import pandas as pd

from feast import Feature, FeatureView, OnDemandFeatureView, PushSource, ValueType
from feast.data_source import DataSource, RequestDataSource
from feast.data_source import DataSource, RequestSource


def driver_feature_view(
Expand Down Expand Up @@ -51,7 +51,7 @@ def conv_rate_plus_100(features_df: pd.DataFrame) -> pd.DataFrame:


def conv_rate_plus_100_feature_view(
sources: Dict[str, Union[RequestDataSource, FeatureView]],
sources: Dict[str, Union[RequestSource, FeatureView]],
infer_features: bool = False,
features: Optional[List[Feature]] = None,
) -> OnDemandFeatureView:
Expand Down Expand Up @@ -86,7 +86,7 @@ def similarity(features_df: pd.DataFrame) -> pd.DataFrame:


def similarity_feature_view(
sources: Dict[str, Union[RequestDataSource, FeatureView]],
sources: Dict[str, Union[RequestSource, FeatureView]],
infer_features: bool = False,
features: Optional[List[Feature]] = None,
) -> OnDemandFeatureView:
Expand All @@ -103,13 +103,11 @@ def similarity_feature_view(


def create_conv_rate_request_data_source():
return RequestDataSource(
name="conv_rate_input", schema={"val_to_add": ValueType.INT32}
)
return RequestSource(name="conv_rate_input", schema={"val_to_add": ValueType.INT32})


def create_similarity_request_data_source():
return RequestDataSource(
return RequestSource(
Comment on lines 105 to +110
Copy link
Copy Markdown
Member

@achals achals Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

create_*_source?

Copy link
Copy Markdown
Collaborator Author

@kevjumba kevjumba Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

name="similarity_input",
schema={
"vector_double": ValueType.DOUBLE_LIST,
Expand Down
6 changes: 3 additions & 3 deletions sdk/python/tests/integration/registration/test_inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
SnowflakeSource,
ValueType,
)
from feast.data_source import RequestDataSource
from feast.data_source import RequestSource
from feast.errors import (
DataSourceNoNameException,
RegistryInferenceFailure,
Expand Down Expand Up @@ -164,7 +164,7 @@ def test_update_data_sources_with_inferred_event_timestamp_col(universal_data_so

def test_on_demand_features_type_inference():
# Create Feature Views
date_request = RequestDataSource(
date_request = RequestSource(
name="date_request", schema={"some_date": ValueType.UNIX_TIMESTAMP}
)

Expand Down Expand Up @@ -217,7 +217,7 @@ def test_view_with_missing_feature(features_df: pd.DataFrame) -> pd.DataFrame:

def test_datasource_inference():
# Create Feature Views
date_request = RequestDataSource(
date_request = RequestSource(
name="date_request", schema={"some_date": ValueType.UNIX_TIMESTAMP}
)

Expand Down
4 changes: 2 additions & 2 deletions sdk/python/tests/integration/registration/test_registry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
from feast.entity import Entity
from feast.feature import Feature
from feast.feature_view import FeatureView
from feast.on_demand_feature_view import RequestDataSource, on_demand_feature_view
from feast.on_demand_feature_view import RequestSource, on_demand_feature_view
from feast.protos.feast.types import Value_pb2 as ValueProto
from feast.registry import Registry
from feast.repo_config import RegistryConfig
Expand Down Expand Up @@ -244,7 +244,7 @@ def test_modify_feature_views_success(test_registry):
created_timestamp_column="timestamp",
)

request_source = RequestDataSource(
request_source = RequestSource(
name="request_source", schema={"my_input_1": ValueType.INT32}
)

Expand Down
6 changes: 3 additions & 3 deletions ui/feature_repo/features.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
FileSource,
ValueType,
)
from feast.data_source import RequestDataSource
from feast.data_source import RequestSource
from feast.request_feature_view import RequestFeatureView
from feast.on_demand_feature_view import on_demand_feature_view
import pandas as pd
Expand Down Expand Up @@ -126,12 +126,12 @@

# Define a request data source which encodes features / information only
# available at request time (e.g. part of the user initiated HTTP request)
input_request = RequestDataSource(
input_request = RequestSource(
name="transaction", schema={"transaction_amt": ValueType.INT64},
)

# Define an on demand feature view which can generate new features based on
# existing feature views and RequestDataSource features
# existing feature views and RequestSource features
@on_demand_feature_view(
inputs={"credit_history": credit_history, "transaction": input_request,},
features=[
Expand Down