[event memory, deferred] Add sessions to EventMemory (port of #1597's session half) - #1715

Event memory bounds when its vector writes are sent by cancelling an
upsert at a deadline, which needs a written guarantee of what cancelling
does. VectorStorePartition.upsert now states that cancelling the call
stops it from sending further requests to the backend, and that a
request already sent may still be applied, so a cancelled upsert may
have written some of the records. RegistryBackedVectorStorePartition's
`_upsert` hook states the same obligation for its implementations.

Every implementation honors it:

- RegistryBackedVectorStorePartition awaits the registration check,
  `_upsert`, and the check again in sequence, so a cancellation stops at
  the await it lands on.
- Qdrant and Milvus await each request, and send the halves of a batch
  the server refuses as too large one after another, so a cancellation
  raised in a request ends the call. Neither client catches
  CancelledError: qdrant-client's REST path catches Exception only, and
  pymilvus's retry decorators catch grpc.RpcError, MilvusException, and
  Exception, so a cancelled RPC surfaces as CancelledError and is not
  retried.
- sqlite-vec writes in one transaction. A statement already queued to
  aiosqlite's thread finishes there; SQLAlchemy treats the cancellation
  as a disconnect and invalidates the connection, so the transaction
  never commits and nothing more is executed on it.
- The engine-backed SQLite store commits its records and pending
  operations, then applies them to the search engine, then marks them
  applied. A cancellation in the engine step leaves the operations
  pending, and startup replays them.
- The in-memory test partition has no await in its upsert.

Tests cancel an upsert mid-call and check that nothing more goes out:
for Qdrant and Milvus with a fake client that refuses the batch as too
large and holds its first half (the second half is never sent, and the
registration is not checked after it), and against the server (the
request in flight; for Milvus counted at the gRPC stub, so a pymilvus
retry would show); for both SQLite stores, cancelling at the upsert's
first statement, and for the engine-backed one also while its search
engine applies the records, no further statement executes.

Verification: ruff, ruff format, and ty clean; the vector store unit
tests, 401 passed, 5 skipped. The fake-client tests fail when the
halving swallows the cancellation and sends the second half. The
integration tests need Docker and were not run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…forgets

The segment store and the vector store share no transaction, so an ingest
that stopped after storing its events left them unsearchable, and a forget
racing an ingest or a late vector write could leave vectors of forgotten
events behind.

The segment store now holds the events themselves, and two queues of work
the vector store still owes:

- Pending derivatives: an ingest stores its events, segments, derivative
  links, and one pending record per event in one transaction, upserts the
  vectors within a write budget, and deletes the record. A record left
  behind is claimed by any process once the budget has passed, and
  written again within a budget of its own.
- Tombstones: a forget locks its events, deletes their pending records,
  records the derivatives it unlinked in one tombstone, and deletes the
  events in one transaction. The vectors are deleted at once and again
  after the tombstone retention, which outlasts any write sent within
  its budget.

Claims hold a lease, back off doubling on failure, and stop after
repeated failures, which is logged. A search drops a hit with no link
and deletes its vector once, best effort. Partitions are deleted from
the segment store first. Each purge call stays within its bound across
segments, events, and the derivatives tombstones name.

design/event_memory_ingest_and_deletion.md gives the protocol and why
it is safe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

EventMemory's background work claims pending derivatives and tombstones
from the whole segment store database, and its records name no vector
store or embedder: it writes and deletes with its own. That is right only
if one EventMemory serves every partition in the database, and only if it
runs whether or not a session is opened.

The resource manager now builds the deployment's EventMemory from the
episodic memory's long-term memory configuration, starts its background
work, and shuts it down on close. The server builds it at startup when
that configuration names a segment store. A session opens its partition
on it, and the deployment's segmenter, deriver, and reranker serve every
session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…deployment

The deployment's one EventMemory serves every event session with its own
segmenter, deriver, and reranker, so the copies a session saved no longer
apply. The session's event configuration no longer has those fields, and
rows saved earlier load with them ignored; creating a project checks a
reranker only on the declarative backend.

A project's `reranker` applies to the declarative backend only, as its
description now says, and an event project's response names none, as for
`embedder`. The OpenAPI document is regenerated for these fields only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Adding events and claiming pending derivatives and tombstones depend on
RETURNING, which SQLite added in 3.35: the event insert returns the
events it stored, and each claim returns the row it took. The store's
params refuse an older SQLite runtime, after the check that each session
gets its own connection to one database, and a test raises the minimum
past the runtime to see the refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…tamp under it

MemMachine and its users share one property key space. A key beginning
with `memmachine_` now belongs to MemMachine: `reserved_property_key`
builds one as `memmachine_<system>_<field>`, and every key is a valid
vector store identifier, `[a-z0-9_]` and at most 32 bytes.

EventMemory writes its timestamp under `memmachine_em_timestamp`. It
refuses an event whose property key is reserved or not a valid
identifier, through `validate_user_property_key`; a key is used exactly
as given, never rewritten.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

An EventMemory filter, and a segment store's, named a property in two
ways: `m.<key>` for the key itself, and a bare name for `_<name>`, the
layout one consumer gives its own fields. The bare form tied both to
that consumer's keys.

A filter field is now `timestamp` or `m.<key>` (or `metadata.<key>`).
EventMemory refuses any other field before it searches, and the segment
stores refuse it as well. The vector search filters on the timestamp,
under its reserved key, and on the declared properties. The long-term
memory adapter names its system fields as properties when it hands a
filter to EventMemory, so its client filters are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

MemMachine reserved two key namespaces at once. EventMemory refused a
property key beginning with `memmachine_`, while the long-term memory
adapter wrote its own fields under a leading underscore and refused a
caller key beginning with one. So a caller could not use an underscore,
and the adapter's fields sat outside the namespace meant for them.

The adapter now writes its fields under `memmachine_ltm_<field>`, and
refuses a caller key that is not a legal user property key. A filter
still names a system field by its bare name, and refuses a `m.<key>`
in the reserved namespace.

The reserved namespace is divided by the system that writes each key.
EventMemory refuses only its own `memmachine_em_` keys, in an event
and in a filter, so the component that embeds it can write its own
reserved keys. `is_reserved_property_key` takes the system to ask about
one system's keys.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

An episode's metadata keys become event properties, and the event
backend refuses a key that is not a legal user property key. That
refusal came while the episode store and semantic memory were writing
the same episodes in parallel, so the request failed after they had
kept them, and a request over HTTP got a 500.

`MemMachine.add_episodes` now checks every metadata key of every entry
with `validate_user_property_key` before it writes anything, and the
add-memories route answers a ValueError with 422, as the other routes
do. The metadata field's description states the key rule.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

The metadata field's description gives the key rule in words instead
of a pattern. EventMemory's filter check and the adapter's conversion
state what holds for a reserved key, and `MemMachine.add_episodes`
says its refusal comes before any store is written.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

EventMemory and the segment stores spelled out how a client names a
property in a filter, and EventMemory checked a filter's fields against
that spelling before it searched. The spelling belongs to the filter
language and to the API that accepts a client's filter.

EventMemory no longer checks a filter's fields: the long-term memory
adapter checks a client's filter, and the segment stores refuse a field
that names neither a property nor the timestamp. For the vector search,
EventMemory translates the timestamp to its reserved key and a property
to its key, and passes any other name unchanged. The docstrings describe
a filter over the segments' properties and timestamp.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…hine#1597, without session)

The part of MemMachine#1597 that is not session: events and segments carry a
nullable source id; query and expand take since/until and source_ids;
expand walks outward from a seed, which is an address located whatever
the filters say, and returns a Neighborhood that excludes it; query
answers QueryHits (score, seed, neighborhood); rendering is
render_segments with a DateTimeFormat; the segment store answers
get_segments and get_segment_neighborhoods in place of
get_segment_contexts; the v2 adapter lifts timestamp/created_at bounds
and producer_id conjuncts out of the filter into the typed parameters;
the timestamp column holds a UTC instant.

On this base:
- A neighborhood is MemMachine#1713's walk: the partition's order next to the
  seed, filters applied inside its window of 1,000 segments per side.
  The ordering index keeps main's shape.
- The vector stage gets MemMachine#1684's predicates on the reserved keys and,
  joined with AND, the part of the property filter that names declared
  fields alone, as MemMachine#1702's `extract_filter` routing has it; the segment
  store still gets the whole filter, and its seed lookup leaves out a
  seed the filter rejects. A record's declared properties are its
  event's, which every segment carries unchanged, as the segmenter
  contract now states.
- On the event memory recovery PRs (MemMachine#1839, MemMachine#1840), the data operations
  and expand are EventMemoryPartition's, and rerank and render_segments
  are static on EventMemory; the reranker leaves EventMemory's
  parameters, since LongTermMemory reranks with the deployment's. A
  pending derivatives record holds its event's properties once
  (segment_store_pd), and its derivatives their source id and no
  context (segment_store_pd_dv), following Derivative, so encode and
  the background work build a record from the same properties. The
  search's cleanup of unlinked records stays a helper of its own.
- On MemMachine#1828's key rule, the source is EventMemory's own reserved key,
  `memmachine_em_source`, which `source_ids` reaches. A timestamp
  condition in the filter still reaches the vector search under its
  reserved key, beside the since/until predicates.
- Episode uids are UUIDs since MemMachine#1707: the search path reads each hit's
  episode uid back as a UUID, and the tests key their episodes by
  `_uid(name)` as main's do.
- Session and block kind are split out: the block kind column,
  parameter and record key go to the kinds PR, which is their first
  consumer, and session goes to its own PR.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE

Mechanical: the store now holds events, segments, links, pending
derivatives and tombstones, so its name follows its dependent.
`SegmentStore` and every derived name become `EventMemoryStore` and
theirs, the module and test directories move with them, the tables and
indexes take the `event_memory_store_` prefix, and the configuration key
`segment_store` becomes `event_memory_store` in the API spec, the client,
the docs, the sample configs, the Helm chart and the compose files. No
behavior changes; tables are recreated. Applied by the same
substitutions to this tree rather than rebased.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE

Ported from the agentic_expansion branch, where it was cosine only.
EventMemory takes an optional EvictionOptions: a score threshold, by
the vector store partition's similarity metric, at or better than which
eviction is considered, the number of stored derivatives at or better
than it fetched per new derivative, and how many of a new derivative
and those stored ones to keep when there are more. None keeps every
derivative and issues no query.

Every metric has one rule: a derivative is a near-duplicate when its
score is at least as good as the threshold, in the metric's direction
and units. The batch scores its own pairs by the partition's metric
(cosine similarity or dot product, Euclidean or Manhattan distance), so
the batch's decision and the store's neighbor query agree. Under dot
product over vectors that are not normalized, a vector's length shifts
what a fixed threshold means; no metric is refused.

At encode, after embedding, `_decide_eviction` settles the whole
decision before anything is written: the batch's predecessors of each
derivative are computed from the batch's own embeddings, earlier
indices only, so a batch evicts what serial ingestion would; one
batched neighbor query fetches the stored derivatives at or better
than the threshold; the stored ones' timestamps come from their segments
through the store's lookup, since the vector store answers uuids and
scores only, and a neighbor whose segment is gone is not a member;
then, per derivative, the members over target_size are trimmed from
the temporal middle, the earliest target_size // 2 and the latest
remainder kept. The batch is sorted by timestamp first, so the
predecessor rule matches serial order.

On the event memory recovery write path (MemMachine#1840), the decision only
changes what the encode writes. A skipped batch derivative is left out
of its event's pending derivatives record, so it is neither linked nor
written, by the encode or by the background work. A displaced stored
derivative keeps its link and its segment; its record leaves the
vector store after the batch's records are written and its pending
records deleted, best effort, and a delete that fails leaves the record
searchable until a later eviction or a forget removes it. The store
gains no operation for it.

Tests: the branch's eviction tests on the partition handle, a batch
and serial ingestion trimming alike under every metric, and a later
batch displacing stored records while their links stay.

The surviving records carry their event's declared properties, as every
record does, and the eviction's neighbor query and deletes go to the
vector store partition.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE

…ives

An ingest decides eviction before its transaction and carries it out
after it: skipped derivatives are never linked or written, and displaced
records are deleted after the pending derivatives. A stranded ingest's
derivatives were written by the background work as they were, with no
eviction, so a cluster could grow past its target through recovery.

The background work now decides eviction for the derivatives it writes,
against what is stored then, through the same decision the ingest uses,
which moves to module level so both share it. A record the stranded
write may have stored is the derivative itself, so the claim's own
derivatives are not counted as neighbors, and the neighbor query asks
for that many more so `search_limit` others still fit. Skipped
derivatives are not written. Once the claim is finished, their records,
which the stranded write may have stored, are deleted with the
displaced ones, best effort, as the ingest deletes its displaced
records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Moved here from MemMachine#1684's port, where every block was text and the column
would have held one value with nothing dispatching on it; this PR's
kind-keyed segmenter and deriver tables are its first consumer. The
segment row carries its block's kind in a column of its own, filled by
the store from the block, since the codec's bytes are opaque to SQL;
query, expand, get_segments and get_segment_neighborhoods take
block_kinds; and the vector record carries the kind under its reserved
key, which the vector stage filters with an In predicate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE

…s (speedkick)

The second half of the event memory handoff, on top of the
source/expansion/eviction change: the content model and the processing
steps become registered families keyed by kind.

Blocks. `Block` is an ABC with `kind` as the discriminator every
registered family uses and `render(options) -> str | None`; the union
stays closed over `TextBlock`; the segment row's `block_kind` reads
`block.kind`.

Context. `Context = Mapping[str, ContextPart]`, keyed by part kind,
never None; `Author` is the first registered kind, an unregistered
kind decodes to `UnknownPart` and round-trips unchanged; `with_part` composes one. `ProducerContext`, `NullContext` and the
discriminated union go; the codec encodes a context as `{kind:
fields}`. The server puts the producer id in an `Author` part beside
`source_id`, so rendering keeps its `producer: text` shape.

Segmenter and deriver. `Segmenter` and `Deriver` are tables from block
kind to handler, built from handlers in order with a later handler
replacing an earlier one for its kind. `BlockSegmenter[B]` and `BlockDeriver[B]` are the one-kind
handler contracts, typed by the block class: `split(event, block) ->
list[Piece]` and `derive(segment, block) -> list[str]`. The table
builds every envelope; a handler decides pieces or texts and nothing
else. A kind with no handler passes through as one segment and derives
nothing; nothing raises on a kind. `TextSegmenter`, `WholeTextDeriver`
and `SentenceTextDeriver` keep their names as `text` handlers;
`PassthroughSegmenter` and its configuration name go: one segment per
block is the identity segmentation, and a default gets no name, so an
omitted `segmenter` means it.

Composition. A derivative is text (`Derivative.text`, the text to
embed, plus the segment's `block_kind` for the record), so every
derivative gets the same context processing. `format_header` is the
one composition point for the embedded text and the rendered header:
the timestamp, then the context parts `parts` names in that order
(default `("author",)`), then the content. Parts carry no order; a part
not listed contributes nothing; a new kind is placed by listing it.
`parts` is a parameter of the composers, the handler and `render`,
not a field of `DatetimeFormat`, which writes a timestamp and nothing
else. A handler owns the composition of what it embeds, its
`datetime_format` and `parts`, so a kind or a handler can embed under
its own; the Deriver table composes each derivative's text with them;
rendering for display takes the caller's per call.

Tests: table dispatch, override order, envelope copying; text handlers returning bare
content; composition order, unlisted parts, kind-name validation, and
the anchor over a derivative; context and block round-trips including
an unregistered part kind.

On this base events carry no session, so no segment, derivative or
record copies one; the test harness builds records on the vector store
partition; main's splitting-segmenter test compares the default
segmenter with a text handler; and the event sample configuration main
added documents the text handler as the other sample configurations do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE

RegisteredBlock as the field type left three type errors where a
handler's or the codec's Block was assigned to the field. The fields
are typed Block, decoded through the registered union by a
before-validator and serialized as the concrete kind; the handler
tests keep a typed block in hand instead of narrowing the field.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE

A context was a mapping the caller keyed by hand, and the rule that
the key equals the part's kind lived in prose. Context is a class now:
built from parts, read by kind, so there is no key to get wrong; two
parts of one kind are rejected at construction. It owns its wire form
through a Pydantic core schema, so a model field of the type accepts
an instance or `{kind: fields}` and serializes to `{kind: fields}`,
and the per-model validators, `with_part`, `encode_context` and
`decode_context` go.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE

The kind-keyed tables removed its config, its service-locator branch
and its test, and the module survived with no consumer. The long-term
memory's over-fetch comment named it too; it now says why a segmenter
can carry one episode several times. The expand_context comment main
added names it as well; it now says "no segmenter handler".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE

A coding agent's transcript carries more than messages: the tool calls
the agent makes, what those tools return, and text that entered the
conversation without a user typing it. Each is a block kind of its own,
registered so the v1 events route accepts it and the timeline holds it.

`tool_call` carries the tool's name and its input. The input is a JSON
object rather than a mapping of property values: a tool's arguments
nest, and a block is content the event carries, not something the event
is filtered by. `tool_result` carries the tool's name, its output, and
whether the tool failed instead of returning; a result says nothing
about failure unless it says so. `injected` carries the text and what
put it there: a hook, a skill, a compaction, a reminder, a command, or
something else.

Each kind renders on one line, through the same `render` a timeline
reader already gets for a message: a call's input as compact JSON, a
failed result behind an `[error]` marker, injected text behind its
source. A window mixing messages and tool events reads as one timeline.

None of the three declares a segmenter or a deriver, so the tables'
fallbacks decide: one segment per block, however long, and no
derivative and no vector record. A tool event is on the timeline
and off the search surface, reached by expanding from a
message, which is the capture policy the coding-agent integration
settled on.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE

The kinds' round trip through the codec, their encoded form, and what
each renders: a call as its tool and its input on one line, a result as
its tool and its output with an `[error]` marker when it failed, and
injected text as its source and its text. An empty or overlong tool
name, an unlisted injection source and an unregistered kind are
rejected.

Segmentation and derivation are asserted where the tables decide them:
a long tool result is one segment while the text handler splits the
message beside it, and the text deriver derives nothing from any of the
three. End to end, an event carrying a message, a call and a result is
three segments and one vector record, the query reaches the message,
and expanding from it returns the two tool events.

The route's tests go with the v1 route, which sits above this change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE

Capture holds everything the session log holds, never what a model
selected, so the agent's reasoning is a kind of its own rather than
something dropped on the way in: `{"kind": "thinking", "text": ...}`,
holding the reasoning as the transcript recorded it.

It is treated as the tool events are. It declares no segmenter and no
deriver, so reasoning is one segment however long and yields no
derivative and no vector record: it is on the timeline and off the
search surface, reached by expanding from a message. It renders as
`thinking: <text>`, one line of the same timeline a message and a tool
event render into.

The v1 route, which sits above this change, accepts it through the
registered union and names it with the other kinds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE

An event belongs to a session, a conversation's own timeline, and a
walk never leaves it. Event, Segment and Derivative carry a required,
bounded session_id; the segment row holds it in a NOT NULL column, the
ordering index's second, after the incarnation
(event_memory_store_sg__in_se_ts_ev_ix_of), and
a neighborhood's walk pins the seed's session, on SQLite as a bound
parameter per seed and on PostgreSQL through the lateral join, inside
the same context window. query, expand and get_segments take
session_ids, an empty list keeping nothing and None every session; the
vector record carries the session under its reserved key, so the vector
stage evaluates session_ids too; expand raises LookupError for a seed
outside the named sessions, which it checks through get_segments. The v2
adapter has no conversation id to give, so every event it ingests is in
one reserved session, memmachine_default.

On this base the session goes onto the event memory store's names, the
segment rows an encode adds, the pending derivatives the background
work writes from, and the kind-keyed segmenter and deriver tables,
which copy it into every segment and derivative they build.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE