BigQuery offline store: materializing a BatchFeatureView with a pandas/python UDF fails with Column <feature> not found in table #7047

Expected Behavior

feast materialize / materialize-incremental of a BatchFeatureView with a pandas (or python) UDF reads the source columns the UDF needs from BigQuery, runs the UDF, and writes the view's features to the online store, as it does with the file (Dask) offline store.

Current Behavior

Materialization fails:

  File "feast/infra/compute_engines/local/compute.py", line 84, in _materialize_one
  File "feast/infra/compute_engines/dag/plan.py", line 51, in execute
  File "feast/infra/compute_engines/local/nodes.py", line 395, in execute
  File "feast/utils.py", line 560, in _convert_arrow_to_proto
  File "feast/utils.py", line 582, in _convert_arrow_fv_to_proto
  File "feast/utils.py", line 514, in _columns_to_proto_values
    raise ValueError(f"Column {column} not found in table")
ValueError: Column views not found in table

The DataFrame passed to the UDF contains only the join key and the timestamp columns (e.g. ['listing_id', 'event_timestamp', 'etl_inserted_timestamp']), and none of the source's feature columns.

Cause

For views whose transformation mode is ray, pandas, python or flink, FeatureBuilder.get_column_info signals "read all source columns" by passing an empty feature list (sdk/python/feast/infra/compute_engines/feature_builder.py, L161–172):

# Signal to read all columns by passing empty list for feature_cols.
feature_cols = []

That list is forwarded as feature_name_columns to pull_latest_from_table_or_query / pull_all_from_table_or_query (compute_engines/utils.py, L64 / L76). BigQueryOfflineStore doesn't handle the empty case. It builds the select list from join keys, feature columns and timestamps (offline_stores/bigquery.py, L171 and L242), so it selects only the join keys and timestamps.

The signal was added in #6287 (863315e2, "Support pandas mode in feature builder and fix dask column extraction") and #6310 (a310eaf3, Spark). Those commits taught dask.py (if feature_name_columns:, L411 / L527) and Spark to treat an empty list as "all columns", but not BigQuery. A quick grep suggests postgres, ray, spark and dask handle the empty list, while bigquery, snowflake, redshift, athena, trino, mssql, clickhouse, duckdb, ibis and others don't. I've only verified BigQuery.

Steps to reproduce

Feast 0.67.0 (also current master 7903e2ed), BigQuery offline store, Redis online store (any online store will do), local compute engine (default), feast[gcp,redis] on Python 3.12.

# feature_repo/example.py
import pandas as pd
from feast import BatchFeatureView, BigQuerySource, Entity, Field
from feast.types import Int64, String
from datetime import timedelta

listing = Entity(name="listing", join_keys=["listing_id"])

source = BigQuerySource(
    name="listing_metrics",
    query="SELECT listing_id, views, TIMESTAMP(date) AS event_timestamp FROM `my-project.my_dataset.listing_metrics`",
    timestamp_field="event_timestamp",
)

def passthrough(df: pd.DataFrame) -> pd.DataFrame:
    return df

listing_engagement = BatchFeatureView(
    name="listing_engagement",
    mode="pandas",
    entities=[listing],
    ttl=timedelta(days=365),
    schema=[Field(name="listing_id", dtype=String), Field(name="views", dtype=Int64)],
    source=source,
    udf=passthrough,
    online=True,
)

feast apply
feast materialize 2026-01-01T00:00:00 $(date -u +%Y-%m-%dT%H:%M:%S)
# -> ValueError: Column views not found in table

The same repo with the file offline store materializes fine.

Specifications

• Version: 0.67.0 (bug still present on master 7903e2ed)
• Platform: Linux arm64 (Kubernetes), Python 3.12
• Subsystem: BigQuery offline store / local compute engine (BatchFeatureView materialization)

Possible Solution

Mirror the Dask fix in BigQueryOfflineStore.pull_all_from_table_or_query and pull_latest_from_table_or_query. When feature_name_columns is empty, select every source column:

if not feature_name_columns:
    # FeatureBuilder signals "read all source columns" with an empty list (UDF BatchFeatureViews).
    field_string = "*"          # pull_all: SELECT * FROM {from_expression} WHERE ...

For pull_latest_from_table_or_query, the inner query becomes SELECT *, ROW_NUMBER() OVER(...) and the outer one selects * EXCEPT (_feast_row). Alternatively, fill the list from data_source.get_table_column_names_and_types(config) minus the join keys and timestamps. That's what we do as a workaround (wrapping both methods), and it keeps the generated SQL unchanged.

Happy to open a PR with tests for BigQuery (and the other SQL stores, if maintainers prefer a shared helper).