Skip to content

RemoteOnlineStore.online_write_batch hardcodes timestamp column names (created/event_timestamp), breaking remote write_to_online_store for feature views with a custom created_timestamp_column #6595

Description

@BigyaPradhan

Expected Behavior

Remote write_to_online_store should round-trip regardless of the feature view's
timestamp column names. Writing works for a local provider (the full dataframe is
converted in-process using the source's configured column names) and should behave the
same when the online store is remote (online_store: { type: remote } / the feature
server /write-to-online-store endpoint).

Current Behavior

Writing to the online store through the remote path fails on the server with:

KeyError: 'Field "created_at" does not exist in schema'

whenever the feature view's batch source sets created_timestamp_column (or
timestamp_field) to anything other than the literals "created" /
"event_timestamp".

Root cause — RemoteOnlineStore.online_write_batch serializes the timestamps into
hardcoded column names when building the request body:

# sdk/python/feast/infra/online_stores/remote.py  (online_write_batch)
columnar_data["event_timestamp"].append(_to_naive_utc(event_ts).isoformat())
columnar_data["created"].append(
    _to_naive_utc(created_ts).isoformat() if created_ts else None
)
req_body = {"feature_view_name": table.name, "df": columnar_data, ...}
post_remote_online_write(config=config, req_body=req_body)

The server rebuilds the dataframe verbatim and runs the standard conversion:

# sdk/python/feast/feature_server.py
df = pd.DataFrame(request.df)     # columns: entities, features, event_timestamp, created
store.write_to_online_store(..., df=df)
# -> _prep_rows_to_write_for_ingestion -> _convert_arrow_to_proto

but _convert_arrow_fv_to_proto (in utils.py) resolves the created-timestamp column
by the feature-view-configured name, not the hardcoded "created":

if feature_view.batch_source.created_timestamp_column:
    created_timestamps = [
        ... table.column(
            feature_view.batch_source.created_timestamp_column   # e.g. "created_at"
        ) ...
    ]

So the client emits the column as created and the server looks it up as created_at
KeyError. The timestamp value is not lost — it is on the wire under created;
it is simply labeled with a name the server does not look up. The client write and
the server-side conversion disagree on the timestamp column names.

Server-side traceback:

File ".../feast/feature_server.py", in write_to_online_store
File ".../feast/feature_store.py", in write_to_online_store
File ".../feast/infra/passthrough_provider.py", in _prep_rows_to_write_for_ingestion
File ".../feast/utils.py", in _convert_arrow_fv_to_proto
KeyError: 'Field "created_at" does not exist in schema'

Steps to reproduce

  1. Define a feature view whose batch source uses non-default timestamp column names,
    e.g. PostgreSQLSource(timestamp_field="event_timestamp", created_timestamp_column="created_at", ...).
  2. Point a client at a remote online store: online_store: { type: remote, path: <feature-server-url> } (or call the feature server /write-to-online-store).
  3. store.write_to_online_store(fv_name, df=df) where df includes the created_at
    column. The client-side conversion succeeds; the server returns HTTP 500 with the
    KeyError above. The same write against a local provider succeeds.

Specifications

  • Version: 0.64.0 (client and feature server)
  • Platform: Linux, Python 3.13
  • Subsystem: remote online store (feast/infra/online_stores/remote.py) /
    Python feature server /write-to-online-store (feast/feature_server.py) /
    _convert_arrow_fv_to_proto (feast/utils.py)

Possible Solution

In RemoteOnlineStore.online_write_batch, label the timestamp columns using the
feature view's configured fields instead of hardcoded literals:

event_col   = table.batch_source.timestamp_field or "event_timestamp"
created_col = table.batch_source.created_timestamp_column or "created"
columnar_data[event_col].append(...)
if created_col:
    columnar_data[created_col].append(...)

Alternatively, have the server map the well-known event_timestamp / created
request columns onto the FV's configured names before conversion. Fixing the client to
emit the configured names is the smaller change and keeps the request self-describing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions