Skip to content

[event memory recovery 1/2] Give EventMemory a store's shape, with a handle per partition - #1839

Open
edwinyyyu wants to merge 12 commits into
MemMachine:feat/horizontal-scalingfrom
edwinyyyu:feat/event-memory-partition-handle
Open

edwinyyyu wants to merge 12 commits into
MemMachine:feat/horizontal-scalingfrom
edwinyyyu:feat/event-memory-partition-handle

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the change

#1840 keeps event memory's two stores consistent through failed ingests and forgets. Its background work finishes what a failed request left in any partition, from one EventMemory the server starts at startup, whether or not a session is open. An EventMemory built around one session's two partitions, as it is before this PR, can't do that.

This PR gives EventMemory the shape of the stores it drives, so that #1840's diff holds the recovery protocol and the move to one EventMemory per deployment, and deletes a session's storage by key:

  • EventMemory is built from the segment store, the vector store, and the models (segmenter, deriver, embedder, reranker, metrics factory). It has no data operations:
    • open_or_create_partition(key) opens or creates the key's partition in both stores, the vector store's first, and returns a handle.
    • delete_partition(key) deletes the key's partition from both stores.
    • The formatting helpers stay on it as static methods.
  • EventMemoryPartition is the handle: encode_events, query, forget_events, and similarity_metric. EventMemory.open_or_create_partition builds it on one key's partition in each store, and it holds the two privately.
  • Schema. A vector store answers store_schema, the schema it was built with. EventMemory checks once, when it is built, that its vector store's schema declares the fields it requires, and its handles take the declared fields and the similarity metric from that schema. VectorStorePartition no longer has a store_schema property; each store keeps its partitions' schema privately.
  • Wiring. Until [event memory recovery 2] Keep event memory's two stores consistent through failed ingests and forgets #1840 moves the segmenter, deriver, and reranker to the deployment, each session chooses its own. So the service locator still builds an EventMemory whenever a session is opened, from the deployment's stores and embedder and the session's segmenter, deriver, and reranker, and opens the session's partition through it. LongTermMemory takes the session's handle, in place of both stores, both partitions, and the models.
  • LongTermMemory takes the session's memory itself: a DeclarativeMemory, or an EventMemoryPartition with the episode storage that turns its results into episodes and the reranker that sets its score direction. Each request matches on the memory's type, and an EventMemoryPartition without episode storage is refused when the LongTermMemory is built. DeclarativeBackendParams, EventBackendParams, and their union go. The long-term service locator's long_term_memory_from_config builds the LongTermMemory in the arm that matches the configuration's backend, the declarative backend's DeclarativeMemory included. LongTermMemory has no lifecycle methods: drop_session_partition goes, and so does close, whose body did nothing. Its one search is query, as EpisodicMemory.query_memory above it and EventMemoryPartition.query below it are; it was search_scored.
  • Deletion by key. LongTermMemory held the stores only to delete the session's partitions, so deleting a session built its whole memory, which opened, and could create, its partitions. Now the long-term service locator deletes a session's storage from its configuration, with a function that matches on the backend as its build function does: the event backend deletes the session's partitions by key through EventMemory.delete_partition, opening neither, and the declarative backend deletes the episodes its DeclarativeMemory of the session matches. EpisodicMemoryManager.delete_episodic_session closes a cached idle instance, then deletes the long-term storage the stored configuration names whenever that configuration builds long-term memory, and builds no EpisodicMemory. EpisodicMemory.delete_session_episodes goes; its short-term half only reset in-memory state. The evaluation scripts build the DeclarativeMemory they pass in, and delete a session through the same function.

Behavior that changes:

Otherwise the stores are opened and deleted in the same order with the same arguments, and every data operation runs the same code. The names the two classes share (the reserved timestamp field and the text extraction) move to module level.

Stack

24 open PRs: three independent PRs, and the vector store tree of short parallel branches. Every open PR in the tree has feat/horizontal-scaling as its GitHub base, and the independent PRs have main. The branches are in a fork, and a pull request can target only this repository's branches, so the on column gives the order the PRs build on each other. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

# PR change on
— #1624 Make no memory request create a project main
— #1786 Refuse a filter value of the wrong type for a datetime column, and answer an invalid list filter with 422 (port of #1620) main
— #1792 Refuse property values that some store refuses or alters where an episode enters main

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1675 is closed, superseded by #1627. #1841 is closed: its request packing is dropped, and its cancellation contract is in #1840. #1828 and #1839 sit side by side on the branch; #1839 and #1840 are the event memory recovery stack, and #1684 heads the event memory stack, which its own PRs index, listed because it builds on #1840, its branch also carrying #1828's commits until one of the two merges; #1871 keeps its own Stack section. Until the PRs under it merge, their changes show in a stacked PR's diff.

# PR change on
[vector store scale-out 1/6] #1671 (merged) Remove custom sharding from the Qdrant store (port of #1654) main
[vector store scale-out 2/6] #1733 (merged into feat/horizontal-scaling) Answer vector store queries with record UUIDs and scores, and refuse invalid inputs feat/horizontal-scaling
[vector store scale-out 3/6] #1734 (merged into feat/horizontal-scaling) Arbitrate vector store collections in a SQL registry, with an incarnation per collection life and a purge feat/horizontal-scaling
[vector store scale-out 4/6] #1735 (merged into feat/horizontal-scaling) Move the Qdrant store onto the collection registry feat/horizontal-scaling
[vector store scale-out 5/6] #1736 (merged into feat/horizontal-scaling) Move the Milvus store onto the collection registry, against a Milvus server feat/horizontal-scaling
— #1775 (merged into feat/horizontal-scaling) Accept attempts exhausted in the lifecycle churn contract, and say which Qdrant operations filter on the incarnation feat/horizontal-scaling
— #1779 (merged into feat/horizontal-scaling) Classify Qdrant errors by status code alone feat/horizontal-scaling
— #1813 (merged into feat/horizontal-scaling) Create Qdrant collections with strict mode off feat/horizontal-scaling
— #1820 (merged into feat/horizontal-scaling) List only active projects in projects/list feat/horizontal-scaling
[user properties 1/2] #1702 (merged into feat/horizontal-scaling) Keep undeclared properties out of the vector store feat/horizontal-scaling
[user properties 2/2] #1670 (merged into feat/horizontal-scaling) Remove per-project filterable properties (port of #1606) feat/horizontal-scaling
— #1824 (merged into feat/horizontal-scaling) Drop the unread logical key from the purge queues feat/horizontal-scaling
— #1788 Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract feat/horizontal-scaling
— #1631 (closed) Superseded by #1733–#1736, which hold its changes split in four, with review changes since —
[vector store scale-out 6] #1627 (merged into feat/horizontal-scaling) Give each vector store a fixed schema and string-keyed partitions, built from the deployment's configuration feat/horizontal-scaling
[declared schema 1/2] #1628 (merged into feat/horizontal-scaling) Make a vector store filter only on the properties it declares feat/horizontal-scaling
[sqlite write transactions 1] #1844 (merged into feat/horizontal-scaling) Take SQLite's write lock at BEGIN in every store on a relational SQLite engine feat/horizontal-scaling
[session storage 1/2] #1622 Create a session's storage with the session, never on a request feat/horizontal-scaling
[session storage 2/2] #1625 Look a partition up with get_partition, and remove open-or-create from both stores and close from the segment store #1622
[selective filter 1] #1871 Route filtered queries by selectivity, and search allowlists exhaustively feat/horizontal-scaling
[search results, deferred] #1663 Score every vector search by cosine similarity, and name scores for it (port of #1598's cosine half) #1628
[declared schema 2/2] #1616 Close the filter union, and make negation the complement on every backend feat/horizontal-scaling
[sqlite store fixes 1/7] #1460 Publish vector index files atomically (but not durably) feat/horizontal-scaling
[sqlite store fixes 2/7] #1469 Never reuse a row id in SQLiteVectorStore #1460
[sqlite store fixes 3/7] #1672 Own the search engine's concurrency in the store, not in each engine (port of #1612) #1469
[sqlite store fixes 4/7] #1673 Serialize a partition's writes so the engine sees them in order (port of #1607) #1672
[sqlite store fixes 5/7] #1674 Refuse a pending row replay cannot honor, instead of dropping it (port of #1608) #1673
[sqlite store fixes 6/7] #1675 (closed) Superseded by #1627, which takes SQLite's write lock at BEGIN where the engine is created —
[sqlite store fixes 7/7] #1676 Give every write a fresh row id, so a key names one version (port of #1610) #1674
[sqlite store fixes 8] #1843 Hold the engine lock until a cancelled engine call's thread finishes #1676
[qdrant options] #1618 Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization feat/horizontal-scaling
[milvus options] #1741 Let a deployment tune a Milvus collection's vector index and its searches #1618
— #1841 (closed) Its request packing is dropped, and its cancellation contract is in #1840 —
[property keys] #1828 Refuse an illegal metadata key before any store writes, and reserve one key prefix feat/horizontal-scaling
[event memory recovery 1/2] #1839 (this PR) Give EventMemory a store's shape, with a handle per partition feat/horizontal-scaling
[event memory recovery 2] #1840 Keep event memory's two stores consistent through failed ingests and forgets #1839
[event memory 1/6] #1684 Add time bounds, sources and expansion to EventMemory (port of #1597, without session) #1840
[schema migration 1/2] #1825 Add memmachine-server migrate, which creates every configured store's schema before servers start feat/horizontal-scaling
[schema migration 2/2] #1826 Make startup check what migrate created, and create nothing at startup #1825

This PR is its 12 commits, e125c9aeb, a6c1aaf88, 378c38cf6, 0750fb691, b827f3316, b97e81357, 6dc4ad6d5, 8319ec6e1, b3357dd24, 07122e98b, 1b1f8be62, 338428956, directly on feat/horizontal-scaling. #1840 is stacked on it.

Verification

At this PR's head 338428956, on feat/horizontal-scaling's 0fa52b38b:

  • ruff check and ruff format --check are clean, ty check is clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server), and uv lock --check passes.
  • Server unit tests (packages/server/server_tests -m 'not integration'): 2356 passed, 8 skipped. Client unit tests (packages/client/client_tests): 259 passed.
  • Integration tests through testcontainers: the event memory and segment store tests on PostgreSQL, 110 passed. At 8319ec6e1, the declarative LongTermMemory tests against Neo4j, 3 passed, and the vector store tests against Qdrant, Milvus, and the registry's PostgreSQL, 384 passed; the commits after it change neither those tests nor the code they run. The declarative tests' NebulaGraph half needs a running NebulaGraph and was not run.
  • The test that deletes an event session with no open instance fails on feat/horizontal-scaling's 0fa52b38b, where deleting a session builds its memory and so opens its partitions.

🤖 Written by Claude Code (Claude Opus 5.5) on behalf of @edwinyyyu.

This was referenced Oct 20, 2026
@edwinyyyu
edwinyyyu force-pushed the feat/event-memory-partition-handle branch from 0b020e6 to a20dc24 Compare October 20, 2026 21:07
@edwinyyyu
edwinyyyu force-pushed the feat/event-memory-partition-handle branch from a20dc24 to c494900 Compare October 26, 2026 18:17
@edwinyyyu
edwinyyyu force-pushed the feat/event-memory-partition-handle branch 8 times, most recently from e12aa88 to b64ecf3 Compare November 2, 2026 17:57
edwinyyyu and others added 2 commits November 2, 2026 11:30
EventMemory is built from the segment store, the vector store and the
models. It opens or creates a partition by key in both stores, the vector
store's first, and deletes a partition from both. EventMemoryPartition is
the handle that encodes, queries and forgets; it holds its two store
partitions privately, so no caller can pair one key's partition in one
store with another key's partition in the other.

The service locator builds the EventMemory and opens the session's
partition through it. EventBackendParams carries the EventMemory and the
handle, and LongTermMemory deletes the session's partition through
EventMemory. The reserved timestamp field and text extraction, shared by
both classes, move to module level. No behavior changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion

A vector store's schema is fixed when the store is built, so EventMemory
takes it as a parameter, the same schema the composition root built the
vector store with, rather than reading it from each partition it opens.

- EventMemoryParams gains `store_schema`. EventMemory checks once, at
  construction, that the schema declares the fields EventMemory requires,
  and hands each partition handle the declared fields and the similarity
  metric. A handle's `similarity_metric` and its score direction come
  from them.
- The service locator builds the vector store and EventMemory from one
  schema, `event_backend_store_schema`.
- `VectorStorePartition` no longer has a `store_schema` property; each
  store's partitions keep the schema privately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@edwinyyyu
edwinyyyu force-pushed the feat/event-memory-partition-handle branch from b64ecf3 to a6c1aaf Compare November 2, 2026 19:34
edwinyyyu and others added 6 commits November 2, 2026 14:27
Deleting a session built its EpisodicMemory, which opened, and so could
create, the session's event-backend partitions, and then deleted them
through the session's own LongTermMemory, which held the EventMemory only
for that.

The long-term service locator now builds each backend's memory and
deletes a session's storage, both as functions that match on the
configuration's backend: the event backend deletes the session's
partitions by key through EventMemory, opening neither, and the
declarative backend deletes the episodes its DeclarativeMemory of the
session matches. The episodic service locator deletes the storage
whenever the configuration builds long-term memory. EpisodicMemoryManager
closes a cached idle instance, then deletes the stored configuration's
storage, and builds no EpisodicMemory.

LongTermMemory is the one class that routes requests on the backend: it
takes each backend's memory built, holds no EventMemory, partition key,
or deleted state, and has neither drop_session_partition nor close, whose
body did nothing. EpisodicMemory.delete_session_episodes has no caller
and goes; its short-term half only reset in-memory state. The evaluation
scripts build the DeclarativeMemory they pass in, and delete a session
through delete_long_term_memory_storage.

A session saved with `enabled: false` now has its long-term storage
deleted too, where the old path returned before deleting anything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LongTermMemory has had a single search since its unscored `search` went,
so `search_scored` no longer tells it apart from anything. It is `query`,
as the calls above and below it are: EpisodicMemory.query_memory and
EventMemoryPartition.query. Its two backend branches are
`_query_declarative` and `_query_event`. DeclarativeMemory keeps
`search_scored`, which it tells apart from its unscored `search`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`_declarative_params` was one statement and `_event_params` a block with
one caller, each standing between `long_term_memory_params_from_config`'s
match and the work it routes to. The work now sits in the match arms, as
it does in `delete_long_term_memory_storage` beside it. `_declarative_memory`
and `_event_memory` stay, since building and deleting both use them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once the service locator built each backend's memory, the two backend
params differed only in that memory: a DeclarativeMemory, or an
EventMemoryPartition with the episode storage that turns its results into
episodes and the reranker that sets its score direction. LongTermMemory
now takes the memory itself, with those two as keywords, and each request
matches on the memory's type. An EventMemoryPartition without episode
storage is refused when the LongTermMemory is built.

DeclarativeBackendParams, EventBackendParams, their union, the backend
literal, and the session id, which nothing read, go. The service locator's
`long_term_memory_from_config` builds the LongTermMemory in each match arm,
and the episodic service locator takes it from there. The evaluation
helper passes its DeclarativeMemory directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The episodic service locator's `delete_episodic_memory_storage` only
checked whether the stored configuration builds long-term memory and
passed it on. EpisodicMemoryManager.delete_episodic_session now makes that
check and calls `delete_long_term_memory_storage` itself, so deleting a
session goes from the manager straight to the long-term layer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EventMemoryParams carried the schema its vector store was built with,
beside the store itself, so the schema had two sources that the
composition root had to keep in step. A vector store now answers its
own `store_schema`, the schema it was built with, and EventMemory reads
its required fields, declared fields, and similarity metric from it.
EventMemoryParams no longer takes a schema, and the service locator no
longer passes one. Each store returns the schema it already held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@edwinyyyu
edwinyyyu marked this pull request as ready for review November 3, 2026 17:23
@edwinyyyu

edwinyyyu commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Self-reviewed (human).

edwinyyyu and others added 4 commits November 3, 2026 10:12
`event_backend_store_schema` was split out so the vector store and
EventMemory could be built from one schema. EventMemory now reads the
schema from its vector store, so building the store is the schema's one
use, and it is built there again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test helpers gave an EventMemory's stand-in vector store the schema
read back from the partition's private attribute, and the SQLite stores'
`_stored` read the declared properties the same way. The tests build each
schema, so they now hand the stand-in store that schema, and `_stored`
reads the properties its stores are built with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on's storage

The drop_session_partition test checked that the delete path leaves
reclamation to the background sweeper; the test of deleting an event
session's storage by key checks it again. A new manager test deletes a
cached idle session whose configuration builds long-term memory: the
instance is closed and the configured storage deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@marvinyu-memverge marvinyu-memverge left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at 3384289.

I read event_memory.py, long_term_memory.py and its service locator, the manager's delete path, and the evaluation helpers as whole files at head, and diffed each EventMemoryPartition data method against the base EventMemory.

Verified:

  • encode_events, query, forget_events and the methods under them are the base code; the only changes are the names that moved to module level and the similarity metric read from the store's schema instead of the partition.
  • open_or_create_partition opens the vector store first and the segment store second, and delete_partition deletes in the same order as the base drop_session_partition; both stores document delete_partition as idempotent, so deleting a session that never created its partitions is safe.
  • delete_episodic_session no longer builds an EpisodicMemory: it closes a cached idle instance (whose short-term close is the same reset clear_memory did), then deletes the long-term storage from the stored config. The session row is still there when this runs, since the queued deletion removes it only after the TaskGroup.
  • The declarative delete is the same get_matching_episodes + delete_episodes as before; the event delete builds no reranker.
  • LongTermMemory's score direction uses the same reranker object the locator hands EventMemory, so the two can't disagree.
  • Nothing in src/ or evaluation/ still calls search_scored on LongTermMemory, delete_session_episodes, or the removed params classes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants