Skip to content

Commit b428d7f

Browse files
feat: allow query + path in SparkSource for offline materialization
SparkSource previously required exactly one of table/query/path. This relaxes the constraint to allow query + path together: - query: used for reading raw data during materialization - path: used for offline write-back (offline=True) and as pre-computed read source in get_historical_features Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent dda43c2 commit b428d7f

1 file changed

Lines changed: 12 additions & 4 deletions

File tree

  • sdk/python/feast/infra/offline_stores/contrib/spark_offline_store

sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark_source.py

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -287,11 +287,19 @@ def __init__(
287287
date_partition_column_format: Optional[str] = "%Y-%m-%d",
288288
table_format: Optional[TableFormat] = None,
289289
):
290-
# Check that only one of the ways to load a spark dataframe can be used. We have
291-
# to treat empty string and null the same due to proto (de)serialization.
292-
if sum([(not (not arg)) for arg in [table, query, path]]) != 1:
290+
# query + path is allowed: query for reads during materialization,
291+
# path for offline write-back (offline=True) and get_historical_features.
292+
# table must be standalone (cannot combine with query or path).
293+
has_table = bool(table)
294+
has_query = bool(query)
295+
has_path = bool(path)
296+
if has_table and (has_query or has_path):
293297
raise ValueError(
294-
"Exactly one of params(table, query, path) must be specified."
298+
"'table' cannot be combined with 'query' or 'path'."
299+
)
300+
if not (has_table or has_query or has_path):
301+
raise ValueError(
302+
"At least one of params(table, query, path) must be specified."
295303
)
296304
if path:
297305
# If table_format is specified, file_format is optional (table format determines the reader)

0 commit comments

Comments
 (0)