feat(ai): Implement session resumption and context window compression for live api#9795
feat(ai): Implement session resumption and context window compression for live api#9795
Conversation
🦋 Changeset detectedLatest commit: d397d01 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
There was a problem hiding this comment.
Code Review
This pull request adds support for session resumption and context window compression in Live sessions. Key changes include the implementation of LiveSession.resumeSession(), the addition of SessionResumptionConfig and ContextWindowCompressionConfig interfaces, and updated server message handling for resumption updates. The review feedback identifies a logic error in the session connection handshake where the wrong setup message was transmitted and private state was incorrectly mutated due to a shallow copy. A leftover debugging log was also identified for removal.
| } | ||
|
|
||
| // @beta | ||
| export interface SlidingWindow { |
There was a problem hiding this comment.
I had to lookup what a Sliding Window is within this context.
There was a problem hiding this comment.
me too.
Could we possibly provide a brief description of sliding window here? (or wherever you think it makes the most sense so that folks see the description when they need to understand this concept)
I found this description: https://ai.google.dev/api/live#slidingwindow. But let's not link to here, just pull the important info out of the description.
| } | ||
|
|
||
| // @beta | ||
| export interface SlidingWindow { |
There was a problem hiding this comment.
me too.
Could we possibly provide a brief description of sliding window here? (or wherever you think it makes the most sense so that folks see the description when they need to understand this concept)
I found this description: https://ai.google.dev/api/live#slidingwindow. But let's not link to here, just pull the important info out of the description.
| > This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment. | ||
| > | ||
|
|
||
| Allows external code to await the opening of the websocket connection. |
There was a problem hiding this comment.
nit - let's be consistent throughout with the capitalization of "Websocket". It's an official name of a protocol, so WDYT about capitalizing it throughout?
There was a problem hiding this comment.
Done, hopefully caught them all.
|
|
||
| | Property | Type | Description | | ||
| | --- | --- | --- | | ||
| | [targetTokens](./ai.slidingwindow.md#slidingwindowtargettokens) | number | <b><i>(Public Preview)</i></b> The session reduction target, i.e., how many tokens we should keep. | |
There was a problem hiding this comment.
for this instance of i.e. -- do you mean "for example" or "that is"?
We avoid latin abbreviations like e.g. and i.e. b/c it's unclear which is meant. Could this instance (and any others throughout) pls be changed to standard English phrases?
e.g.,=>for example,ORlikeORsuch asORfor instance,i.e.,=>specificallyORthat is
There was a problem hiding this comment.
I went through and think I got all the public facing ones.
|
|
||
| | Property | Type | Description | | ||
| | --- | --- | --- | | ||
| | [targetTokens](./ai.slidingwindow.md#slidingwindowtargettokens) | number | <b><i>(Public Preview)</i></b> The session reduction target, i.e., how many tokens we should keep. | |
There was a problem hiding this comment.
"we" -- who is "we" referring to here?
Can we say something like "the model" here?
Based on #9624
Moved connection initiation logic from LiveGenerativeModel.connect() into the class so that the LiveSession.resumeSession() method can more easily access it. Since the LiveGenerativeModel.connect() method needs to wait for the connection to be opened successfully, and a constructor can't be async, the class provides a
connectionPromiseproperty for external code to tell when the connection has finished opening.