Skip to content

fix: Keep null values of the latest row in deduplicate_keep_latest - #7050

Open
ShryukGrandhi wants to merge 1 commit into
feast-dev:masterfrom
ShryukGrandhi:fix/bug-94
Open

ShryukGrandhi wants to merge 1 commit into
feast-dev:masterfrom
ShryukGrandhi:fix/bug-94

Conversation

@ShryukGrandhi

Copy link
Copy Markdown

What this PR does / why we need it:

deduplicate_keep_latest is supposed to keep the latest row per entity. It used groupby(entity_keys).first(), which takes the first non-null value per column. When the latest row had a null feature (e.g. tiled std/var over a window with fewer than two events), that column was filled from an older row, so stale values got written to the online store by the Spark and Ray compute engines.

This change sorts by timestamp and then uses drop_duplicates(subset=entity_keys, keep="first"), so the latest row is kept unchanged, nulls included.

Which issue(s) this PR fixes:

Fixes #7049

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Added test_deduplicate_keep_latest_keeps_null_values_of_latest_row to sdk/python/tests/unit/test_aggregation_ops.py. It fails on master (the NaN gets replaced with the older 2.5) and passes with this fix. The rest of test_aggregation_ops.py and the local compute engine unit tests pass, and ruff check/format are clean.

Signed-off-by: ShryukGrandhi <85529990+ShryukGrandhi@users.noreply.github.com>
@ShryukGrandhi
ShryukGrandhi requested a review from a team as a code owner October 20, 2026 08:10

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

deduplicate_keep_latest fills null features of the latest row with values from older rows

1 participant