Skip to content

Commit b42a8de

Browse files
ntkatholentkathole
committed
feat: Fix Map/Dict support and implement schema validation
Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>
1 parent 12d2d9b commit b42a8de

36 files changed

Lines changed: 2279 additions & 170 deletions

‎.pre-commit-config.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ repos:
1414
stages: [commit]
1515
language: system
1616
types: [python]
17+
exclude: '_pb2\.py$'
1718
entry: bash -c 'uv run ruff check --fix "$@" && uv run ruff format "$@"' --
1819
pass_filenames: true
1920

@@ -24,6 +25,7 @@ repos:
2425
stages: [commit]
2526
language: system
2627
types: [python]
28+
exclude: '_pb2\.py$'
2729
entry: bash -c 'uv run ruff check "$@" && uv run ruff format --check "$@"' --
2830
pass_filenames: true
2931

‎docs/getting-started/concepts/feast-types.md‎

Lines changed: 36 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,44 @@ To make this possible, Feast itself has a type system for all the types it is ab
55

66
Feast's type system is built on top of [protobuf](https://github.com/protocolbuffers/protobuf). The messages that make up the type system can be found [here](https://github.com/feast-dev/feast/blob/master/protos/feast/types/Value.proto), and the corresponding python classes that wrap them can be found [here](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/types.py).
77

8-
Feast supports primitive data types (numerical values, strings, bytes, booleans and timestamps). The only complex data type Feast supports is Arrays, and arrays cannot contain other arrays.
8+
Feast supports the following categories of data types:
9+
10+
- **Primitive types**: numerical values (`Int32`, `Int64`, `Float32`, `Float64`), `String`, `Bytes`, `Bool`, and `UnixTimestamp`.
11+
- **Array types**: ordered lists of any primitive type, e.g. `Array(Int64)`, `Array(String)`.
12+
- **Set types**: unordered collections of unique values for any primitive type, e.g. `Set(String)`, `Set(Int64)`.
13+
- **Map types**: dictionary-like structures with string keys and values that can be any supported Feast type (including nested maps), e.g. `Map`, `Array(Map)`.
14+
- **JSON type**: opaque JSON data stored as a string at the proto level but semantically distinct from `String` — backends use native JSON types (`jsonb`, `VARIANT`, etc.), e.g. `Json`, `Array(Json)`.
15+
- **Struct type**: schema-aware structured type with named, typed fields. Unlike `Map` (which is schema-free), a `Struct` declares its field names and their types, enabling schema validation, e.g. `Struct({"name": String, "age": Int32})`.
16+
17+
For a complete reference with examples, see [Type System](../../reference/type-system.md).
918

1019
Each feature or schema field in Feast is associated with a data type, which is stored in Feast's [registry](registry.md). These types are also used to ensure that Feast operates on values correctly (e.g. making sure that timestamp columns used for [point-in-time correct joins](point-in-time-joins.md) actually have the timestamp type).
1120

12-
As a result, each system that feast interacts with needs a way to translate data types from the native platform, into a feast type. E.g., Snowflake SQL types are converted to Feast types [here](https://rtd.feast.dev/en/master/feast.html#feast.type_map.snowflake_python_type_to_feast_value_type). The onus is therefore on authors of offline or online store connectors to make sure that this type mapping happens correctly.
21+
As a result, each system that Feast interacts with needs a way to translate data types from the native platform into a Feast type. E.g., Snowflake SQL types are converted to Feast types [here](https://rtd.feast.dev/en/master/feast.html#feast.type_map.snowflake_python_type_to_feast_value_type). The onus is therefore on authors of offline or online store connectors to make sure that this type mapping happens correctly.
22+
23+
### Backend Type Mapping for Complex Types
24+
25+
Map, JSON, and Struct types are supported across all major Feast backends:
26+
27+
| Backend | Native Type | Feast Type |
28+
|---------|-------------|------------|
29+
| PostgreSQL | `jsonb` | `Map`, `Json`, `Struct` |
30+
| PostgreSQL | `jsonb[]` | `Array(Map)` |
31+
| Snowflake | `VARIANT`, `OBJECT` | `Map` |
32+
| Snowflake | `JSON` | `Json` |
33+
| Redshift | `SUPER` | `Map` |
34+
| Redshift | `json` | `Json` |
35+
| BigQuery | `JSON` | `Json` |
36+
| BigQuery | `STRUCT`, `RECORD` | `Struct` |
37+
| Spark | `map<string,string>` | `Map` |
38+
| Spark | `array<map<string,string>>` | `Array(Map)` |
39+
| Spark | `struct<...>` | `Struct` |
40+
| Spark | `array<struct<...>>` | `Array(Struct(...))` |
41+
| MSSQL | `nvarchar(max)` | `Map`, `Json`, `Struct` |
42+
| DynamoDB | Proto bytes | `Map`, `Json`, `Struct` |
43+
| Redis | Proto bytes | `Map`, `Json`, `Struct` |
44+
| Milvus | `VARCHAR` (serialized) | `Map`, `Json`, `Struct` |
45+
46+
**Note**: When the backend native type is ambiguous (e.g., `jsonb` could be `Map`, `Json`, or `Struct`), the **schema-declared Feast type takes precedence**. The backend-to-Feast type mappings above are only used for schema inference when no explicit type is provided.
1347

1448
**Note**: Feast currently does *not* support a null type in its type system.

‎docs/getting-started/concepts/feature-view.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ Feature views consist of:
2424
* (optional, but recommended) a schema specifying one or more [features](feature-view.md#field) (without this, Feast will infer the schema by reading from the data source)
2525
* (optional, but recommended) metadata (for example, description, or other free-form metadata via `tags`)
2626
* (optional) a TTL, which limits how far back Feast will look when generating historical datasets
27+
* (optional) `enable_validation=True`, which enables schema validation during materialization (see [Schema Validation](#schema-validation) below)
2728

2829
Feature views allow Feast to model your existing feature data in a consistent way in both an offline (training) and online (serving) environment. Feature views generally contain features that are properties of a specific object, in which case that object is defined as an entity and included in the feature view.
2930

@@ -159,6 +160,43 @@ Feature names must be unique within a [feature view](feature-view.md#feature-vie
159160

160161
Each field can have additional metadata associated with it, specified as key-value [tags](https://rtd.feast.dev/en/master/feast.html#feast.field.Field).
161162

163+
## Schema Validation
164+
165+
Feature views support an optional `enable_validation` parameter that enables schema validation during materialization and historical feature retrieval. When enabled, Feast verifies that:
166+
167+
- All declared feature columns are present in the input data.
168+
- Column data types match the expected Feast types (mismatches are logged as warnings).
169+
170+
This is useful for catching data quality issues early in the pipeline. To enable it:
171+
172+
```python
173+
from feast import FeatureView, Field
174+
from feast.types import Int32, Int64, Float32, Json, Map, String, Struct
175+
176+
validated_fv = FeatureView(
177+
name="validated_features",
178+
entities=[driver],
179+
schema=[
180+
Field(name="trips_today", dtype=Int64),
181+
Field(name="rating", dtype=Float32),
182+
Field(name="preferences", dtype=Map),
183+
Field(name="config", dtype=Json), # opaque JSON data
184+
Field(name="address", dtype=Struct({"street": String, "city": String, "zip": Int32})), # typed struct
185+
],
186+
source=my_source,
187+
enable_validation=True, # enables schema checks
188+
)
189+
```
190+
191+
**JSON vs Map vs Struct**: These three complex types serve different purposes:
192+
- **`Map`**: Schema-free dictionary with string keys. Use when the keys and values are dynamic.
193+
- **`Json`**: Opaque JSON data stored as a string. Backends use native JSON types (`jsonb`, `VARIANT`). Use for configuration blobs or API responses where you don't need field-level typing.
194+
- **`Struct`**: Schema-aware structured type with named, typed fields. Persisted through the registry via Field tags. Use when you know the exact structure and want type safety.
195+
196+
Validation is supported in all compute engines (Local, Spark, and Ray). When a required column is missing, a `ValueError` is raised. Type mismatches are logged as warnings but do not block execution, allowing for safe gradual adoption.
197+
198+
The `enable_validation` parameter is also available on `BatchFeatureView` and `StreamFeatureView`, as well as their respective decorators (`@batch_feature_view` and `@stream_feature_view`).
199+
162200
## \[Alpha] On demand feature views
163201

164202
On demand feature views allows data scientists to use existing features and request time data (features only available at request time) to transform and create new features. Users define python transformation logic which is executed in both the historical retrieval and online retrieval paths.

‎docs/how-to-guides/dbt-integration.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -289,6 +289,12 @@ Feast automatically maps dbt/warehouse column types to Feast types:
289289
| `TIMESTAMP`, `DATETIME` | `UnixTimestamp` |
290290
| `BYTES`, `BINARY` | `Bytes` |
291291
| `ARRAY<type>` | `Array(type)` |
292+
| `JSON`, `JSONB` | `Map` (or `Json` if declared in schema) |
293+
| `VARIANT`, `OBJECT` | `Map` |
294+
| `SUPER` | `Map` |
295+
| `MAP<string,string>` | `Map` |
296+
| `STRUCT`, `RECORD` | `Struct` (BigQuery) |
297+
| `struct<...>` | `Struct` (Spark) |
292298

293299
Snowflake `NUMBER(precision, scale)` types are handled specially:
294300
- Scale > 0: `Float64`

‎docs/specs/offline_store_format.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,12 @@ Here's how Feast types map to Pandas types for Feast APIs that take in or return
4949
| DOUBLE\_LIST | `list[float]`|
5050
| FLOAT\_LIST | `list[float]`|
5151
| BOOL\_LIST | `list[bool]`|
52+
| MAP | `dict` (`Dict[str, Any]`)|
53+
| MAP\_LIST | `list[dict]` (`List[Dict[str, Any]]`)|
54+
| JSON | `object` (parsed Python dict/list/str)|
55+
| JSON\_LIST | `list[object]`|
56+
| STRUCT | `dict` (`Dict[str, Any]`)|
57+
| STRUCT\_LIST | `list[dict]` (`List[Dict[str, Any]]`)|
5258

5359
Note that this mapping is non-injective, that is more than one Pandas type may corresponds to one Feast type (but not vice versa). In these cases, when converting Feast values to Pandas, the **first** Pandas type in the table above is used.
5460

@@ -78,6 +84,12 @@ Here's how Feast types map to BigQuery types when using BigQuery for offline sto
7884
| DOUBLE\_LIST | `ARRAY<FLOAT64>`|
7985
| FLOAT\_LIST | `ARRAY<FLOAT64>`|
8086
| BOOL\_LIST | `ARRAY<BOOL>`|
87+
| MAP | `JSON` / `STRUCT` |
88+
| MAP\_LIST | `ARRAY<JSON>` / `ARRAY<STRUCT>` |
89+
| JSON | `JSON` |
90+
| JSON\_LIST | `ARRAY<JSON>` |
91+
| STRUCT | `STRUCT` / `RECORD` |
92+
| STRUCT\_LIST | `ARRAY<STRUCT>` |
8193

8294
Values that are not specified by the table above will cause an error on conversion.
8395

@@ -94,3 +106,23 @@ https://docs.snowflake.com/en/user-guide/python-connector-pandas.html#snowflake-
94106
| INT32 | `INT8 / UINT8 / INT16 / UINT16 / INT32 / UINT32` |
95107
| INT64 | `INT64 / UINT64` |
96108
| DOUBLE | `FLOAT64` |
109+
| MAP | `VARIANT` / `OBJECT` |
110+
| JSON | `JSON` / `VARIANT` |
111+
112+
#### Redshift Types
113+
Here's how Feast types map to Redshift types when using Redshift for offline storage:
114+
115+
| Feast Type | Redshift Type |
116+
|-------------|--|
117+
| Event Timestamp | `TIMESTAMP` / `TIMESTAMPTZ` |
118+
| BYTES | `VARBYTE` |
119+
| STRING | `VARCHAR` |
120+
| INT32 | `INT4` / `SMALLINT` |
121+
| INT64 | `INT8` / `BIGINT` |
122+
| DOUBLE | `FLOAT8` / `DOUBLE PRECISION` |
123+
| FLOAT | `FLOAT4` / `REAL` |
124+
| BOOL | `BOOL` |
125+
| MAP | `SUPER` |
126+
| JSON | `json` / `SUPER` |
127+
128+
Note: Redshift's `SUPER` type stores semi-structured JSON data. During materialization, Feast automatically handles `SUPER` columns that are exported as JSON strings by parsing them back into Python dictionaries before converting to `MAP` proto values.

‎protos/feast/core/FeatureView.proto‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ message FeatureView {
3636
FeatureViewMeta meta = 2;
3737
}
3838

39-
// Next available id: 17
39+
// Next available id: 18
4040
// TODO(adchia): refactor common fields from this and ODFV into separate metadata proto
4141
message FeatureViewSpec {
4242
// Name of the feature view. Must be unique. Not updated.
@@ -89,6 +89,9 @@ message FeatureViewSpec {
8989

9090
// The transformation mode (e.g., "python", "pandas", "spark", "sql", "ray")
9191
string mode = 16;
92+
93+
// Whether schema validation is enabled during materialization
94+
bool enable_validation = 17;
9295
}
9396

9497
message FeatureViewMeta {

‎protos/feast/core/StreamFeatureView.proto‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ message StreamFeatureView {
3737
FeatureViewMeta meta = 2;
3838
}
3939

40-
// Next available id: 20
40+
// Next available id: 21
4141
message StreamFeatureViewSpec {
4242
// Name of the feature view. Must be unique. Not updated.
4343
string name = 1;
@@ -99,5 +99,8 @@ message StreamFeatureViewSpec {
9999
// Hop size for tiling (e.g., 5 minutes). Determines the granularity of pre-aggregated tiles.
100100
// If not specified, defaults to 5 minutes. Only used when enable_tiling is true.
101101
google.protobuf.Duration tiling_hop_size = 19;
102+
103+
// Whether schema validation is enabled during materialization
104+
bool enable_validation = 20;
102105
}
103106

‎protos/feast/types/Value.proto‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,10 @@ message ValueType {
5353
FLOAT_SET = 27;
5454
BOOL_SET = 28;
5555
UNIX_TIMESTAMP_SET = 29;
56+
JSON = 32;
57+
JSON_LIST = 33;
58+
STRUCT = 34;
59+
STRUCT_LIST = 35;
5660
}
5761
}
5862

@@ -88,6 +92,10 @@ message Value {
8892
FloatSet float_set_val = 27;
8993
BoolSet bool_set_val = 28;
9094
Int64Set unix_timestamp_set_val = 29;
95+
string json_val = 32;
96+
StringList json_list_val = 33;
97+
Map struct_val = 34;
98+
MapList struct_list_val = 35;
9199
}
92100
}
93101

‎sdk/python/feast/batch_feature_view.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ def __init__(
9797
feature_transformation: Optional[Transformation] = None,
9898
batch_engine: Optional[Dict[str, Any]] = None,
9999
aggregations: Optional[List[Aggregation]] = None,
100+
enable_validation: bool = False,
100101
):
101102
if not flags_helper.is_test():
102103
warnings.warn(
@@ -136,6 +137,7 @@ def __init__(
136137
source=source, # type: ignore[arg-type]
137138
sink_source=sink_source,
138139
mode=mode,
140+
enable_validation=enable_validation,
139141
)
140142

141143
def get_feature_transformation(self) -> Optional[Transformation]:
@@ -169,6 +171,7 @@ def batch_feature_view(
169171
description: str = "",
170172
owner: str = "",
171173
schema: Optional[List[Field]] = None,
174+
enable_validation: bool = False,
172175
):
173176
"""
174177
Creates a BatchFeatureView object with the given user-defined function (UDF) as the transformation.
@@ -199,6 +202,7 @@ def decorator(user_function):
199202
schema=schema,
200203
udf=user_function,
201204
udf_string=udf_string,
205+
enable_validation=enable_validation,
202206
)
203207
functools.update_wrapper(wrapper=batch_feature_view_obj, wrapped=user_function)
204208
return batch_feature_view_obj

‎sdk/python/feast/feature_view.py‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,7 @@ class FeatureView(BaseFeatureView):
109109
owner: str
110110
materialization_intervals: List[Tuple[datetime, datetime]]
111111
mode: Optional[Union["TransformationMode", str]]
112+
enable_validation: bool
112113

113114
def __init__(
114115
self,
@@ -125,6 +126,7 @@ def __init__(
125126
tags: Optional[Dict[str, str]] = None,
126127
owner: str = "",
127128
mode: Optional[Union["TransformationMode", str]] = None,
129+
enable_validation: bool = False,
128130
):
129131
"""
130132
Creates a FeatureView object.
@@ -150,11 +152,14 @@ def __init__(
150152
primary maintainer.
151153
mode (optional): The transformation mode for feature transformations. Only meaningful
152154
when transformations are applied. Choose from TransformationMode enum values.
155+
enable_validation (optional): If True, enables schema validation during materialization
156+
to check that data conforms to the declared feature types. Default is False.
153157
154158
Raises:
155159
ValueError: A field mapping conflicts with an Entity or a Feature.
156160
"""
157161
self.name = name
162+
self.enable_validation = enable_validation
158163
self.entities = [e.name for e in entities] if entities else [DUMMY_ENTITY_NAME]
159164
self.ttl = ttl
160165
schema = schema or []
@@ -281,6 +286,7 @@ def __copy__(self):
281286
online=self.online,
282287
offline=self.offline,
283288
sink_source=self.batch_source if self.source_views else None,
289+
enable_validation=self.enable_validation,
284290
)
285291

286292
# This is deliberately set outside of the FV initialization as we do not have the Entity objects.
@@ -309,6 +315,7 @@ def __eq__(self, other):
309315
or sorted(self.entity_columns) != sorted(other.entity_columns)
310316
or self.source_views != other.source_views
311317
or self.materialization_intervals != other.materialization_intervals
318+
or self.enable_validation != other.enable_validation
312319
):
313320
return False
314321

@@ -447,6 +454,7 @@ def to_proto_spec(
447454
source_views=source_view_protos,
448455
feature_transformation=feature_transformation_proto,
449456
mode=mode_to_string(self.mode),
457+
enable_validation=self.enable_validation,
450458
)
451459

452460
def to_proto_meta(self):
@@ -616,6 +624,9 @@ def _from_proto_internal(
616624
f"Entities: {feature_view.entities} vs Entity Columns: {feature_view.entity_columns}"
617625
)
618626

627+
# Restore enable_validation from proto field.
628+
feature_view.enable_validation = feature_view_proto.spec.enable_validation
629+
619630
# FeatureViewProjections are not saved in the FeatureView proto.
620631
# Create the default projection.
621632
feature_view.projection = FeatureViewProjection.from_feature_view_definition(

0 commit comments

Comments
 (0)