Skip to content

fix: keep Firestore session events in time order - #1653

Open
innoprej wants to merge 1 commit into
google:mainfrom
innoprej:fix/firestore-event-order
Open

innoprej wants to merge 1 commit into
google:mainfrom
innoprej:fix/firestore-event-order

Conversation

@innoprej

Copy link
Copy Markdown
Contributor

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

2. Or, if no issue exists, describe the change:

Problem:

FirestoreSessionService stores each event's timestamp as Instant.toString(), which leaves out the fraction when it is zero, and orders and filters events by comparing those strings. Within one second, an event at .000 (…05Z) sorts after the later events (…05.400Z), so getSession and listEvents return them out of order. afterTimestamp is compared as text too and with >: a whole-second cursor leaves out the rest of that second, a millisecond cursor includes an earlier whole-second event, and an event at exactly the cursor is left out, while InMemorySessionService, VertexAiSessionService and ADK Python keep the events at or after the cursor. #1646 has the details and a reproduction.

Solution:

  • eventToMap writes timestamp with exactly three fraction digits (DateTimeFormatterBuilder.appendInstant(3), for example 2026-10-10T05:00:05.000Z). Event timestamps are whole milliseconds, so the digits are exact and text order is time order. eventFromMap and FirestoreMemoryService read both forms, as before.
  • getSession rounds afterTimestamp up to whole milliseconds, writes it the same way and filters with whereGreaterThanOrEqualTo. An event at millisecond m is at or after the cursor t exactly when m >= ceil(t), which is what InMemorySessionService keeps. The query still orders and filters on the one field, so no new index is needed.
  • getSession and listEvents sort the events they read by timestamp, as VertexAiSessionService.filterEvents does. Events stored before this change keep their old strings, and this keeps those sessions in time order too.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

./mvnw -pl contrib/firestore-session-service test: 55 run, 2 failures, both in FirestorePropertiesTest (shouldReturnHardcodedDefaultIfPropertyFileHasNoEntry, shouldLoadDevPropertiesWhenEnvIsSetToDev), which fail only on Windows, the same way on main, and pass in the Ubuntu CI (local run: Microsoft Build of OpenJDK 17.0.19, Windows 11). The changed and new tests are in FirestoreSessionServiceTest:

Test On main With this change
appendEvent_storesTimestampsWithThreeFractionDigits: events at 05.000, 05.400 and 06.000 fails: stored as …05Z, …05.400Z, …06Z passes
getSession_withAfterTimestamp_queriesAtOrAfterRoundedCursor (replaces getSession_withAfterTimestamp_appliesFilterToQuery, which used Instant.now()): cursors …05Z, …05.200Z and …05.200300Z fails: whereGreaterThan with …05Z passes: whereGreaterThanOrEqualTo with …05.000Z, …05.200Z and …05.201Z
getSession_eventsStoredOutOfOrder_returnsEventsInTimeOrder: documents …05.400Z and …05Z, in Firestore's text order fails: 05.400 first passes
listEvents_eventsStoredOutOfOrder_returnsEventsInTimeOrder: the same for listEvents fails: 05.400 first passes

Rounding the cursor down instead of up fails the second test (…05.200Z instead of …05.201Z).

Manual End-to-End (E2E) Tests:

Not run against a real Firestore database or the emulator. The tests check the strings and the query the service sends to a mocked Firestore; the order and the result sets then follow from Firestore's documented ordering of strings by UTF-8 bytes. With this commit, #1643 and #1650 applied together, all 38 tests in FirestoreSessionServiceTest pass.

Checklist

  • I have read the CONTRIBUTING.md document.
  • My pull request contains a single commit.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

  • Not covered: events with the same millisecond (point 2 in FirestoreSessionService compares event timestamps as text, reordering or skipping same-second events #1646) still have no tie-breaker, since that needs an extra field or time-ordered document IDs, and ordering on a second field needs a composite index. The sort above is stable, so tied events keep the order Firestore returns. I can open a separate issue for it, so that this PR can close FirestoreSessionService compares event timestamps as text, reordering or skipping same-second events #1646.
  • Events stored before this change keep their old strings, and the sort only fixes the order they are returned in. Firestore still selects them by text, so within one second numRecentEvents can return an old …05Z event instead of a later …05.400Z one, and afterTimestamp can include an old …05Z event although the cursor is later in that second. It no longer leaves out events at or after the cursor.
  • timestamp stays a string, a deliberate difference from ADK Python noted in the TIMESTAMP_FORMATTER Javadoc. ADK Python stores a Firestore timestamp, but Firestore sorts timestamps before strings, so sessions holding both would be misordered unless the stored events are migrated. If you would prefer the timestamp type with a migration, I can change this.
  • FirestoreMemoryService returns the stored string as MemoryEntry.timestamp, so entries for events at a whole second now read …05.000Z instead of …05Z. Both are ISO 8601.
  • The session's updateTime is still written with toString(); it is only parsed back, never compared in a query.

FirestoreSessionService stored event timestamps with
Instant.toString(), which leaves out the fraction when it is zero, and
ordered and filtered events by comparing those strings. Within one
second, an event at .000 sorted after the later events, a whole-second
afterTimestamp cursor left out the rest of that second, and
afterTimestamp used > where InMemorySessionService,
VertexAiSessionService and ADK Python keep events at or after the
cursor.

- Event timestamps are written with exactly three fraction digits.
- afterTimestamp is rounded up to whole milliseconds, written the same
  way and matched with >=.
- getSession and listEvents sort the events they read by timestamp, so
  events stored in the old format also come back in time order.

The field stays a string. ADK Python stores a Firestore timestamp, but
Firestore orders timestamps before strings, so sessions with events
stored before this change would be misordered.
@hemasekhar-p

Copy link
Copy Markdown
Contributor

Hi @innoprej, We greatly appreciate your contribution and the effort you put into this pull request. It is currently under review by our team. We will notify you if any additional details are needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FirestoreSessionService compares event timestamps as text, reordering or skipping same-second events

2 participants