Skip to content

feat: Opt-in created_timestamp <= entity_timestamp cutoff in point-in-time joins (as-of-known-time retrieval) #6615

Description

@addenergyx

Is your feature request related to a problem? Please describe.

Picking this back up from a Slack thread a while ago, where the dedup-only handling of created_timestamp_column was confirmed.

get_historical_features enforces point-in-time correctness on the event timestamp only. When a feature view has a created_timestamp_column, it is used purely as a dedup tiebreaker (MAX(created_timestamp) per event timestamp), never as a filter. There is no created_timestamp <= entity_timestamp predicate anywhere in the join templates (verified on current master, BigQuery template as reference; the other offline stores follow the same pattern).

As a result, a training row at entity timestamp T can be served a value that did not exist yet at T: its event_timestamp <= T but its created_timestamp > T (backfills, late corrections, delayed batch writes). This causes:

  • Feature leakage: training rows see values that were written after the event.
  • Non-reproducible training sets: re-running after a backfill silently changes past feature vectors.
  • Offline/online skew: the online model at T served the older value, offline retrieval returns the newer one.

Concrete example: two rows for the same entity and event timestamp, value 4000 created before the entity timestamp and 6000 created after it. get_historical_features returns 6000.

Describe the solution you'd like

An opt-in, per-query flag, default off:

store.get_historical_features(
    entity_df=...,
    features=...,
    filter_by_created_timestamp=True,  # only use feature versions known as of each entity row's timestamp
)

When enabled and the feature view has a created_timestamp_column, add a single predicate to the point-in-time join:

AND subquery.created_timestamp <= entity_dataframe.entity_timestamp

Default False produces identical SQL and results to today. The flag can be threaded through the same optional-kwargs mechanism as the recently added start_date/end_date, so offline stores that have not implemented it are unaffected unless a user explicitly opts in.

Describe alternatives you've considered

  • A per-FeatureView or config-level setting. Per-query is preferred since the same view may want both leak-free training and "latest known" analysis.
  • reproducible historical feature queries #2980 proposed a global max_created_time fixed at query time (closed by the stale bot). This proposal is the per-row variant: each entity row is cut off at its own timestamp, which is what you need to emulate what the online store would have served at each event.
  • We currently run a patched BigQuery join template with this predicate in production. It works but has to be re-synced on every Feast upgrade.

Additional context

  • Rows with a NULL created_timestamp would be excluded by the predicate; suggest documenting the column must be non-null when the mode is enabled.
  • Orthogonal to ttl: TTL bounds by event time, this bounds by known time.
  • I have a PR ready implementing this across the offline stores (SQL templates, ibis-based stores, dask) with tests. Happy to iterate on naming and API surface.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions