Skip to content

feat: Support nested collection types (Array/Set of Array/Set) (#5947) #6132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
franciscojavierarceo merged 10 commits into from
Apr 2, 2026
Merged

feat: Support nested collection types (Array/Set of Array/Set) (#5947) #6132

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
5425a92
feat: Support nested collection types (Array/Set of Array/Set) (#5947)
soooojinlee Mar 17, 2026
734aee6
fix: Fix remote online read for nested collection types and add docs
soooojinlee Mar 18, 2026
e57ddd2
fix: Fix JSON deserialization, schema inference, and silent fallback …
soooojinlee Mar 18, 2026
36fdf4e
fix: Fix mypy type error in nested collection proto construction
soooojinlee Mar 19, 2026
3c81d5e
fix: Fix equality comparison for nested types and JSON deserializatio…
soooojinlee Mar 26, 2026
4dabd1b
feat: Remove depth limit for nested collection types and improve test…
soooojinlee Mar 26, 2026
fd0d745
refactor: Replace combinatorial nested collection enums with recursiv…
soooojinlee Mar 28, 2026
89379fc
fix: Preserve inner element types in PyArrow schema inference and opt…
soooojinlee Apr 1, 2026
a4dde0f
fix: Add np.ndarray support in nested collection proto conversion and…
soooojinlee Apr 1, 2026
646b6bc
Merge branch 'master' into feat/support-nested-collection-types
franciscojavierarceo Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: Add np.ndarray support in nested collection proto conversion and… 
… clarify placeholder pyarrow type

- Add np.ndarray to isinstance check in _convert_nested_collection_to_proto
  to fix KeyError for 3+ level nesting during materialization (PyArrow produces
  np.ndarray, not Python list)
- Add comment clarifying VALUE_LIST/VALUE_SET placeholder in feast_value_type_to_pa

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: soojin <soojin@dable.io>
  • Loading branch information
@soooojinlee @claude @ntkathole
2 people authored and ntkathole committed Apr 2, 2026
commit a4dde0f81f79681ad8d41fe5a068b12ea7deb724
5 changes: 4 additions & 1 deletion sdk/python/feast/type_map.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1088,7 +1088,8 @@ def _convert_nested_collection_to_proto(
# Empty inner collection: store as empty ProtoValue
inner_values.append(ProtoValue())
elif any(
isinstance(item, (list, set, tuple)) for item in inner_list
isinstance(item, (list, set, tuple, np.ndarray))
for item in inner_list
):
# Deeper nesting (3+ levels): recurse using VALUE_LIST
inner_proto = _convert_nested_collection_to_proto(
Expand Down Expand Up @@ -1765,6 +1766,8 @@ def feast_value_type_to_pa(
ValueType.JSON_LIST: pyarrow.list_(pyarrow.large_string()),
ValueType.STRUCT: pyarrow.struct([]),
ValueType.STRUCT_LIST: pyarrow.list_(pyarrow.struct([])),
# Placeholder: inner type is unknown from ValueType alone.
# Callers needing accurate inner types should use from_feast_to_pyarrow_type() with a FeastType.
ValueType.VALUE_LIST: pyarrow.list_(pyarrow.list_(pyarrow.string())),
ValueType.VALUE_SET: pyarrow.list_(pyarrow.list_(pyarrow.string())),
Comment on lines +1771 to +1772
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot Mar 30, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 feast_value_type_to_pa returns hardcoded list(list(string)) for all VALUE_LIST/VALUE_SET regardless of actual inner type

The feast_value_type_to_pa mapping at sdk/python/feast/type_map.py:1671-1672 hardcodes pyarrow.list_(pyarrow.list_(pyarrow.string())) for both VALUE_LIST and VALUE_SET. This means any caller that uses feast_value_type_to_pa to build a PyArrow schema for nested collections (e.g., Array(Array(Int32))) would get list<list<string>> instead of list<list<int32>>. While get_pyarrow_schema_from_batch_source in offline_utils.py:287-288 has special handling to avoid this, other potential callers would silently get the wrong type. The correct type-aware conversion exists in from_feast_to_pyarrow_type (sdk/python/feast/types.py:375-376), but that function takes a FeastType, not a ValueType. This inconsistency could cause silent data corruption if any code path relies on feast_value_type_to_pa for nested types without the special-case override.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

ValueType.NULL: pyarrow.null(),
Expand Down
Loading