Skip to content

[property keys] Refuse an illegal metadata key before any store writes, and reserve one key prefix - #1828

Open
edwinyyyu wants to merge 7 commits into
MemMachine:feat/horizontal-scalingfrom
edwinyyyu:fix/metadata-key-rule-main
Open

edwinyyyu wants to merge 7 commits into
MemMachine:feat/horizontal-scalingfrom
edwinyyyu:fix/metadata-key-rule-main

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the change

Every property key now follows one rule. A key under memmachine_ belongs to MemMachine, and each system writes its own keys as memmachine_<system>_<field>. Every other key belongs to the user and must be lowercase letters, digits, and underscores, at most 32 bytes. The rule is checked before any store writes. Fixes #1796.

Why one reserved prefix. The design reasoning is in The reserved key namespace (design/event_model.md, #1579).

New with this change, compared with the base:

  • MemMachine's keys cannot collide with an application's. On the base, the adapter's fields and EventMemory's timestamp both used a leading underscore, and EventMemory refused only the exact name _timestamp. An application that embeds EventMemory and writes its own _ keys could pick a name MemMachine later uses. Now every key MemMachine writes into an event starts with memmachine_, each component writes under its own part of it (memmachine_em_, memmachine_ltm_), and a field added later is reserved without any list to update.
  • Users can write _ keys. On the base, the event backend refused every caller key that started with _. Now only memmachine_ is reserved.
  • An illegal key is refused before anything is written. On the base, the event backend refused a _ key only after the episode store and semantic memory had written the episode, and the request failed with 500 (Event-backend sessions refuse metadata keys outside the vector stores' key set after the episode store and semantic memory have written the episode #1796). Now MemMachine.add_episodes checks every key first and answers 422.
  • Both v2 backends accept the same keys. On the base, the declarative backend accepted any key and the event backend any key that did not start with _, so a key such as userId was kept but could never be declared for indexing. Now both refuse the same keys, userId among them, and every accepted key is a valid field name on every vector store, so any key can be declared for indexing.
  • A filter names a system field one way. On the base, an event-backend filter could reach the adapter's stored field through m._producer_id as well as through producer_id. Now only the bare name works, and m.<key> with a reserved key is refused.

Kept from the base:

  • A caller cannot write a field MemMachine stamps, such as another producer's id. The event backend now refuses memmachine_ keys instead of _ keys. The declarative backend still stores user keys under metadata., so they never meet its own fields.

  • The prefix memmachine_ is reserved. common/property_keys.py builds a reserved key with reserved_property_key(system, field) and checks a user key with validate_user_property_key. EventMemory writes its timestamp under memmachine_em_timestamp instead of _timestamp, and it refuses an event whose property key is reserved or illegal.

  • Metadata keys are checked before anything is written. MemMachine.add_episodes checks every metadata key of every entry with validate_user_property_key before it writes. The add-memories route answers the refusal with 422, as the other routes answer a ValueError.

    • Without the check, EventMemory would refuse such a key while the episode store and semantic memory wrote the same episodes in parallel, and the request would fail with 500 after those stores had kept the episodes.
    • The rule holds for every v2 session, so a session on the declarative backend refuses userId and user-id too.
    • A layer above MemMachine that offers a wider key set keeps its own mapping of its keys to legal ones ([Feat]: Limit characters allowed in field/property names; reserve field/property names beginning with underscore #1383). A UUID is a legal key in its 32-character hex form (uuid.hex), not in its 36-character hyphenated form.
    • The description of the metadata field in the API docs states the rule.
  • The long-term memory adapter writes its fields under memmachine_ltm_. It wrote _episode_uid, _session_key, _producer_id, and five more fields under a leading underscore, and refused any caller key that began with one.

    • The adapter now writes memmachine_ltm_<field>, so _x is an ordinary user key.
    • It refuses a caller key that is not a legal user key.
    • A client still filters a system field by its bare name, such as producer_id. A filter field m.<key> whose key is in the reserved namespace is refused.
  • EventMemory refuses only its own keys. It refuses keys under memmachine_em_ in an event. It keeps any other key under the prefix, since that key belongs to the component that embeds it. is_reserved_property_key(key, system=...) answers for one system's keys.

  • No bare filter name stands for _<name>. EventMemory and the segment stores mapped a bare filter name to _<name>, which is the adapter's layout.

    • The adapter maps its bare system names onto its reserved keys before it calls EventMemory.
    • A segment store refuses a field that names neither a property nor the timestamp, context.* included, instead of matching nothing.
    • The vector search still filters on the timestamp, under its reserved key.
    • EventMemory and the segment stores leave the filter syntax to the filter language: their docstrings keep the base's wording, and EventMemory checks no filter fields. Resolving a filter's fields to typed references before they reach a store is a separate change.

Data written before this change. Event-backend segments carry the adapter's underscore keys, and vector records carry those and _timestamp; neither EventMemory nor the adapter reads or writes them any more. A search skips a segment without memmachine_ltm_episode_uid, so it returns none of the old episodes. A vector store's tables and collections keep the properties declared when they were created, and nothing compares those with the current declaration: on SQLite, an old partition opens and answers queries, and an upsert into it raises OperationalError for the missing memmachine_em_timestamp and memmachine_ltm_* columns. Event-backend data from before needs migrating or recreating.

Commits

  1. 57297b1f7 Reserve the memmachine_ property key prefix, with EventMemory's timestamp under it. This is [event memory 1/6] Add time bounds, sources and expansion to EventMemory (port of #1597, without session) #1684's key-rule part: common/property_keys.py and its tests, _timestamp renamed to memmachine_em_timestamp, where a timestamp filter still reaches the vector store, and EventMemory's key check through validate_user_property_key.
  2. fdcbc4d6c Name an EventMemory filter's properties as m. alone. EventMemory, the segment stores, and the in-memory fake name properties m.<key> and refuse other fields; the vector search filters on the timestamp and the declared properties. The adapter maps its bare system names onto its underscore keys.
  3. 295e70ea7 Keep the event backend's fields under the reserved key prefix. This adds memmachine_ltm_<field>, the user-key check on the adapter's caller keys, the refusal of reserved keys in a client filter, and EventMemory's narrowed check with is_reserved_property_key(..., system=...).
  4. f8ae5a302 Refuse an illegal metadata key before any store writes, with 422. This adds the check in MemMachine.add_episodes, the route's 422, and the metadata field's description in the API docs and docs/openapi.json.
  5. 4fc4aab97 State the metadata key rule in words, and two key docstrings plainly. The metadata field's description and three docstrings are reworded; no code changes.
  6. b6b75ab02 Leave the filter syntax to the filter language and its API. EventMemory no longer checks a filter's fields, and its docstrings and the segment stores' no longer spell the syntax; the timestamp's routing to its reserved key stays.
  7. 013346b9f Describe filters without singling out the timestamp. The filter docstrings of EventMemory, the segment stores, and the adapter's mapping return to the base's wording, changed only where it no longer holds; no code changes.

Position: on feat/horizontal-scaling directly, parallel to #1839 and #1840. #1684 builds on both, so its chain sits on this PR's commits rebuilt onto #1840, and whichever of this PR and #1840 merges second rebases onto feat/horizontal-scaling. #1684 adds the source key, the time and source filters, and expansion under this rule.

Stack

24 open PRs: three independent PRs, and the vector store tree of short parallel branches. Every open PR in the tree has feat/horizontal-scaling as its GitHub base, and the independent PRs have main. The branches are in a fork, and a pull request can target only this repository's branches, so the on column gives the order the PRs build on each other. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

# PR change on
— #1624 Make no memory request create a project main
— #1786 Refuse a filter value of the wrong type for a datetime column, and answer an invalid list filter with 422 (port of #1620) main
— #1792 Refuse property values that some store refuses or alters where an episode enters main

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1675 is closed, superseded by #1627. #1841 is closed: its request packing is dropped, and its cancellation contract is in #1840. #1828 and #1839 sit side by side on the branch; #1839 and #1840 are the event memory recovery stack, and #1684 heads the event memory stack, which its own PRs index, listed because it builds on #1840, its branch also carrying #1828's commits until one of the two merges; #1871 keeps its own Stack section. Until the PRs under it merge, their changes show in a stacked PR's diff.

# PR change on
[vector store scale-out 1/6] #1671 (merged) Remove custom sharding from the Qdrant store (port of #1654) main
[vector store scale-out 2/6] #1733 (merged into feat/horizontal-scaling) Answer vector store queries with record UUIDs and scores, and refuse invalid inputs feat/horizontal-scaling
[vector store scale-out 3/6] #1734 (merged into feat/horizontal-scaling) Arbitrate vector store collections in a SQL registry, with an incarnation per collection life and a purge feat/horizontal-scaling
[vector store scale-out 4/6] #1735 (merged into feat/horizontal-scaling) Move the Qdrant store onto the collection registry feat/horizontal-scaling
[vector store scale-out 5/6] #1736 (merged into feat/horizontal-scaling) Move the Milvus store onto the collection registry, against a Milvus server feat/horizontal-scaling
— #1775 (merged into feat/horizontal-scaling) Accept attempts exhausted in the lifecycle churn contract, and say which Qdrant operations filter on the incarnation feat/horizontal-scaling
— #1779 (merged into feat/horizontal-scaling) Classify Qdrant errors by status code alone feat/horizontal-scaling
— #1813 (merged into feat/horizontal-scaling) Create Qdrant collections with strict mode off feat/horizontal-scaling
— #1820 (merged into feat/horizontal-scaling) List only active projects in projects/list feat/horizontal-scaling
[user properties 1/2] #1702 (merged into feat/horizontal-scaling) Keep undeclared properties out of the vector store feat/horizontal-scaling
[user properties 2/2] #1670 (merged into feat/horizontal-scaling) Remove per-project filterable properties (port of #1606) feat/horizontal-scaling
— #1824 (merged into feat/horizontal-scaling) Drop the unread logical key from the purge queues feat/horizontal-scaling
— #1788 Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract feat/horizontal-scaling
— #1631 (closed) Superseded by #1733–#1736, which hold its changes split in four, with review changes since —
[vector store scale-out 6] #1627 (merged into feat/horizontal-scaling) Give each vector store a fixed schema and string-keyed partitions, built from the deployment's configuration feat/horizontal-scaling
[declared schema 1/2] #1628 (merged into feat/horizontal-scaling) Make a vector store filter only on the properties it declares feat/horizontal-scaling
[sqlite write transactions 1] #1844 (merged into feat/horizontal-scaling) Take SQLite's write lock at BEGIN in every store on a relational SQLite engine feat/horizontal-scaling
[session storage 1/2] #1622 Create a session's storage with the session, never on a request feat/horizontal-scaling
[session storage 2/2] #1625 Look a partition up with get_partition, and remove open-or-create from both stores and close from the segment store #1622
[selective filter 1] #1871 Route filtered queries by selectivity, and search allowlists exhaustively feat/horizontal-scaling
[search results, deferred] #1663 Score every vector search by cosine similarity, and name scores for it (port of #1598's cosine half) #1628
[declared schema 2/2] #1616 Close the filter union, and make negation the complement on every backend feat/horizontal-scaling
[sqlite store fixes 1/7] #1460 Publish vector index files atomically (but not durably) feat/horizontal-scaling
[sqlite store fixes 2/7] #1469 Never reuse a row id in SQLiteVectorStore #1460
[sqlite store fixes 3/7] #1672 Own the search engine's concurrency in the store, not in each engine (port of #1612) #1469
[sqlite store fixes 4/7] #1673 Serialize a partition's writes so the engine sees them in order (port of #1607) #1672
[sqlite store fixes 5/7] #1674 Refuse a pending row replay cannot honor, instead of dropping it (port of #1608) #1673
[sqlite store fixes 6/7] #1675 (closed) Superseded by #1627, which takes SQLite's write lock at BEGIN where the engine is created —
[sqlite store fixes 7/7] #1676 Give every write a fresh row id, so a key names one version (port of #1610) #1674
[sqlite store fixes 8] #1843 Hold the engine lock until a cancelled engine call's thread finishes #1676
[qdrant options] #1618 Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization feat/horizontal-scaling
[milvus options] #1741 Let a deployment tune a Milvus collection's vector index and its searches #1618
— #1841 (closed) Its request packing is dropped, and its cancellation contract is in #1840 —
[property keys] #1828 (this PR) Refuse an illegal metadata key before any store writes, and reserve one key prefix feat/horizontal-scaling
[event memory recovery 1/2] #1839 Give EventMemory a store's shape, with a handle per partition feat/horizontal-scaling
[event memory recovery 2] #1840 Keep event memory's two stores consistent through failed ingests and forgets #1839
[event memory 1/6] #1684 Add time bounds, sources and expansion to EventMemory (port of #1597, without session) #1840
[schema migration 1/2] #1825 Add memmachine-server migrate, which creates every configured store's schema before servers start feat/horizontal-scaling
[schema migration 2/2] #1826 Make startup check what migrate created, and create nothing at startup #1825

This PR is its 7 commits, 57297b1f7, fdcbc4d6c, 295e70ea7, f8ae5a302, 4fc4aab97, b6b75ab02, 013346b9f, directly on feat/horizontal-scaling.

Verification

On 2026-11-03, per Edwin's "Don't change the docstrings to single out the timestamp", 013346b9f restores the base's filter docstrings, changed only where they no longer hold: an event memory base property translates to its reserved key, any other name passes unchanged, and a segment store refuses an unknown field. Docstrings only. At 013346b9f: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean. The copy for #1684's chain, on #1840 a82fc6e65, has the same commit as 4fe221b12.

On 2026-11-02 Edwin set this PR parallel to #1839 and asked that EventMemory and the segment stores not know how the filter language is written; his May 8 ruling that EventMemory takes the client's m. and bare names is replaced. b6b75ab02 drops EventMemory's filter field check and the docstrings' spelling of the syntax. EventMemory's change from feat/horizontal-scaling is now the timestamp key and its routing, the bare names no longer mapped to _<name>, and the event key check. At b6b75ab02: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean; the server unit suite passes, 2371 tests (8 skipped), three fewer for EventMemory's dropped filter check. No integration tests ran. The same commit on the copy for #1684's chain, on #1840 cda5face2, is 70508eb8d; its server unit suite passes, 2431 tests (8 skipped).

Re-ported on 2026-11-02 onto feat/horizontal-scaling 0fa52b38b, below #1839, after Edwin's "#1839 on #1828": the key rule needs nothing #1839 or #1840 adds. On feat/horizontal-scaling, EventMemory has no EventMemoryPartition, so the reserved key's set and the event key check stay on EventMemory, and the tests use its event_memory fixture; test_event_memory_recovery.py, which #1840 adds, drops out of the first commit. Otherwise each file's added and removed lines match 53f030efe's. The commits are now 57297b1f7, fdcbc4d6c, 295e70ea7, f8ae5a302, and 4fc4aab97, with the same messages, authors, and dates, and signed; the last two keep their patch-ids. At 4fc4aab97: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean; the server unit suite passes, 2374 tests (8 skipped). No integration tests ran. The SQLite old-data results above hold at 4fc4aab97.

Rebuilt on 2026-11-02 on #1840 cda5face2, below #1684, after Edwin's "please reorder" the same day: reserving one key prefix and adding event metadata to event memory are separate ideas. The first commit is #1684's key-rule part. The other four are the commits from 8ae3e117f, re-expressed on #1840's code, where everything they change already exists: the bare-name mapping to _<name>, the adapter's underscore fields and its check, the segment store's field mapping, and the API boundary. With #1684 stay memmachine_em_source, the since, until, and source predicates, expansion, and this PR's docstrings on them. For the new base: the vector search keeps filtering on the timestamp under its reserved key, which on #1684 the since and until predicates do; the in-memory segment store fake evaluates a timestamp condition; and the tests name #1840's get_segment_contexts, ProducerContext, and the timestamp key where they named #1684's methods, its author helper, and the source key. The six files #1684 does not change are identical to 8ae3e117f's. At 53f030efe: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean; the server unit suite passes, 2434 tests (8 skipped). No integration tests ran. The SQLite old-data results hold, measured with #1840's declaration of _timestamp and the adapter's _<field> keys: an old partition opens and answers queries, an upsert into it raises OperationalError (no column p_memmachine_em_timestamp), and a new partition takes the upsert.

Restacked on 2026-11-02 onto #1684 574566083, on #1840 cda5face2, whose base no longer packs vector store upserts and takes the cancellation contract into #1840. None of this PR's files change beneath it. The commits are now 5960c8ab7, cbc3475da, 86a20bdf4, and 8ae3e117f, with the same patch-ids, messages, authors, and dates, and signed. The head's change from 2cfd65cbc is #1684's change from ffe4374ba. At 8ae3e117f: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean.

Restacked on 2026-11-02 onto #1684 ffe4374ba, on #1840 96097c2c0, on feat/horizontal-scaling with [sqlite write transactions 1] (#1844) merged. The base changes the segment store's engine check and imports, in the file whose filter-field mapping this PR changes, and the replay was clean. The commits are now d8cb9872d, 37f23f72a, f8c3805ed, and 2cfd65cbc, with the same patch-ids, messages, authors, and dates, and signed. At 2cfd65cbc: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean. The unit tests of event memory, long-term memory, the property keys, MemMachine, and the v2 router pass, 527 tests.

Restacked on 2026-10-30 onto #1684 c9270336a, on #1840 004e6c79f, on feat/horizontal-scaling 690116e37, which has [declared schema 1/2] (#1628), #1831, and #1858 merged. #1831 and #1858 change three files this PR also changes, memmachine.py, the v2 router, and test_memmachine_mock.py, and the replay was still clean. The commits are now 952edd32f, da734bfdf, 187ddf713, and 99aff2611, with the same patch-ids, messages, authors, and dates, and signed. At 99aff2611: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean. The unit tests of event memory, long-term memory, the property keys, MemMachine, and the v2 router pass, 527 tests; the six more are #1831's MemMachine deletion tests.

Restacked on 2026-10-28 onto #1684 e3bde9748, on #1840 f4b487018, whose base takes #1628's df716e47b: is_valid_identifier's docstring, and the SQLite records table built without extend_existing. The commits are now 246689971, e3ca4e1a3, cf01bedd8, and 0ccfdf4a1, with the same patch-ids, messages, authors, and dates, and signed. The head's change from 321f9bb41 is #1684's change from e9556d9af. At 0ccfdf4a1: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean.

Restacked on 2026-10-27 onto #1684 e9556d9af, on #1840 b631cd84c, whose base takes #1628's second review round: is_valid_identifier moves from the vector store's utils.py into its data_types.py, and the interface's Raises sections are sorted. The commits are now 74277a4b8, a265d75f0, 80679112e, and 321f9bb41, with the same messages, authors, and dates, and signed. The first, third, and fourth keep their patch-ids. The second's one changed line is event_memory.py's import of is_valid_identifier, now from memmachine_server.common.vector_store.data_types. At 321f9bb41: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean. The unit tests of event memory, long-term memory, the property keys, MemMachine, and the v2 router pass, 521 tests.

Restacked on 2026-10-27 onto #1684 384dfec58, on #1840 cff96f208, whose base takes #1628's review changes: the vector stores evaluate the whole filter language, so supported_filter_nodes, UnsupportedFilterError, and filter_nodes are gone, the declared-property checks live in utils.py, and the SQLite stores' property module is sqlite_properties.py. This PR names none of them and touches none of the files the base changed. The commits are now 86b5b5189, bf5efd35e, 015013700, and 3104dfe5e, with the same patch-ids, messages, authors, and dates, and signed. At 3104dfe5e: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean. The unit tests of event memory, long-term memory, the property keys, MemMachine, and the v2 router pass, 521 tests.

Restacked on 2026-10-27 onto #1684 2122252a3, on #1840 94b3588a6, after #1844 moved above #1628 beneath it; #1684's tree is 127f18617's. The commits are now a1f0c9fbb, cbe1a9774, 166d4a648, and f2a020a3c, with the same patch-ids, messages, authors, and dates, and signed. The head's tree is a6c3fbcf8's, so the checks and tests below stand.

Restacked on 2026-10-27 onto #1684 127f18617, on #1840 75982b908, after [vector store scale-out 6] (#1627) merged into feat/horizontal-scaling as 805ea2177, a squash whose tree is 8c2351dbe's. The commits are now 8d7db9f46, 48ce752ad, a765a8a99, and a6c3fbcf8, with the same patch-ids, messages, authors, and dates, and signed. The head's change from 51cf91c30 is #1684's change from c847f0b53 to 127f18617: a longer request timeout for two Milvus integration tests. At a6c3fbcf8: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean. The 521 unit tests that passed at 51cf91c30 cover this head, since the change touches integration tests only.

Restacked on 2026-10-26 onto #1684 c847f0b53, on #1840 5b16f04f5, whose base gives a vector store its schema as one VectorStoreSchema and names EventMemory's declaration expected_vector_store_indexed_properties. The commits are now c117e5df2, 61867d7d2, b96006335, and 51cf91c30, with the same messages, authors, and dates, and signed. The third and fourth keep their patch-ids, and the first two keep their source changes line for line. Only their tests changed: the test fakes take store_schema, the shared test partition declares episode_uid, the declaring fixture declares d1, d2, and d3, and two test modules import VectorStoreSchema beside reserved_property_key. At 51cf91c30: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean. The unit tests of event memory, long-term memory, the property keys, MemMachine, and the v2 router pass, 521 tests; the base now marks one segment store test as an integration test. The SQLite old-data results above hold at 51cf91c30, measured on a store declared as the deployment declares its event store.

Restacked on 2026-10-20 onto #1684 ee58642bc, which now sits on #1840 and #1839: EventMemory serves each partition through an EventMemoryPartition handle. The commits are now cacda52fb, 81189acb8, 6dad028b3, and 4227ccd58, with the same messages, authors, and dates, and signed; the third and fourth keep their patch-ids. The first two conflicted where the base moved or renamed code, and each resolution keeps the commit's change on the base's layout. The filter check, _vector_store_filter, and the key check are now EventMemoryPartition's, with _TIMESTAMP_FIELD and _EVENT_MEMORY_SYSTEM beside the base's module-level _RESERVED_PROPERTY_SCHEMA, and encode_events's Raises keeps the base's two new entries. The adapter's changes have the same added and removed lines as before. The tests use the event_memory_partition fixture, and the segment store's test adds its segment through _add_segments. At 4227ccd58: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean. The unit tests of event memory, long-term memory, the property keys, MemMachine, and the v2 router pass, 522 tests. The SQLite old-data results above hold at 4227ccd58.

Restacked on 2026-10-20 onto #1684 29c22631c, whose base takes #1627's comment, docstring, and documentation accuracy fixes. The commits are now 11d733b58, 80f332c89, 7e0004419, and 423ac84e3, with the same patch-ids, messages, authors, and dates, and signed. The head's change from fbfeae54b has the same patch-id as #1684's change from ab4714322 to 29c22631c. At 423ac84e3: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean.

Restacked on 2026-10-20 onto #1684 ab4714322, whose base takes #1627's second review's test, doc, and docstring fixes. The commits are now 3a687e95e, bf9606c17, b7314dab9, and fbfeae54b, with the same patch-ids, messages, authors, and dates, and signed. The head's change from 4b25288f8 has the same patch-id as #1684's change from 387d56131 to ab4714322. At fbfeae54b: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean.

Restacked on 2026-10-19 onto #1684 387d56131, whose base takes #1627's review fixes to the database manager and the partition registry. The commits are now 3f584035a, e87b72361, b68d07cea, and 4b25288f8, with the same patch-ids, messages, authors, and dates, and signed. The head's change from 0cb33663c has the same patch-id as #1684's change from 84a000bc2 to 387d56131. At 4b25288f8: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean.

Restacked on 2026-10-19 onto #1684 84a000bc2, which no longer sits on #1663: event memory scores by the partition's similarity metric, and query takes score_threshold. The commits are now 235d8ca1c, 0d3c08f3f, 7d5651306, and 0cb33663c, with the same messages, authors, and dates, and signed. Commits 3 and 4 have the same patch-ids. Commits 1 and 2 conflicted with the new base in two places, and each resolution keeps both sides: the event memory tests' make_partition keeps its similarity_metric parameter and takes this PR's docstring and declared episode_uid; EventMemory.query's Raises lists the non-finite score_threshold beside this PR's filter field refusals. At 0cb33663c: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean; the event memory, long-term memory, and property key unit tests pass (310). Also at 0cb33663c, the SQLite check of a partition created with _episode_uid declared gives the same results as at d432bf8fa.

Restacked on 2026-10-19 onto #1684 4bac1cbfe, whose base rewords one line of design/vector_store_partition_registry.md. The commits are now f942fa924, ebb6e884a, 367ccebb3, and 425fd4a39, with the same patch-ids, messages, authors, and dates, and signed. The head's change from 322af628e has the same patch-id as #1684's change from 45b75a446 to 4bac1cbfe. At 425fd4a39: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean.

Restacked on 2026-10-19 onto #1684 45b75a446, whose base takes #1824 from feat/horizontal-scaling. The commits are now 6aaceac22, 5afe6aa86, d080e6470, and 322af628e, with the same patch-ids, messages, authors, and dates, and signed. The head's change from 9824b4cb0 has the same patch-id as #1684's change from 4241c994e to 45b75a446. At 322af628e: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean.

Restacked on 2026-10-16 onto #1684 4241c994e, whose base renames the configuration's tombstone retention check. The commits are now 226c77809, 4ab38bd1e, 7f3156d7a, and 9824b4cb0, with the same patch-ids, messages, authors, and dates, and signed. The head's change from 401477980 has the same patch-id as #1684's change from b6a4f6285 to 4241c994e. At 9824b4cb0: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean.

Restacked on 2026-10-16 onto #1684 b6a4f6285, whose base renames the vector store's raising checks to validate_*. The commits are now f7964d7b2, ce79c7e8a, 884dbfdef, and 401477980, with the same patch-ids, messages, authors, and dates, and signed. The head's change from 9310c2b68 has the same patch-id as #1684's change from 9a035e168 to b6a4f6285. At 401477980: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean.

Restacked on 2026-10-16 onto #1684 9a035e168, whose base renames the vector store's validate_identifier to is_valid_identifier. The commits are now e1172686a, 7870865f3, 284f3c8a5, and 9310c2b68, with the same messages, authors, and dates, and signed. Commits 1, 3, and 4 have the same patch-ids; commit 2's only change is that event_memory.py imports and calls is_valid_identifier. At 9310c2b68: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean. Also at 9310c2b68, the SQLite check of a partition created with _episode_uid declared gives the same results as at d432bf8fa.

Restacked on 2026-10-15 onto #1684 539f52a98, whose base drops the vector store's per-partition schema record. The commits are now 1fea085fb, 437e1c5ed, 941ffbfe3, and d432bf8fa, with the same patch-ids, messages, authors, and dates, and signed. The head's change from 39075cdf1 has the same patch-id as #1684's change from aae191241 to 539f52a98. At d432bf8fa: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean. Also at d432bf8fa, on SQLite: a partition created with _episode_uid declared, reopened with memmachine_ltm_episode_uid declared, opens and answers a query; an upsert into it raises OperationalError (no column p_memmachine_ltm_episode_uid); a partition created after takes the same upsert.

Restacked on 2026-10-15 onto #1684 aae191241, whose base names the vector store's lookup open_partition and keeps the segment store's open_partition, close_partition, and open_or_create_partition. The commits became adb72e43a, c01e726ca, 9c9836cce, and 39075cdf1, with the same patch-ids, messages, authors, and dates, and signed. The head's change from 6a71ec3da has the same patch-id as #1684's change from 392d436cf to aae191241. At 39075cdf1: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean.

Restacked on 2026-10-15 onto #1684 392d436cf, whose tree is c0235df55's without the segment store's close_partition, which #1627 then removed. The commits became 76b202fdb, b104ea5fd, 962ef6061, and 6a71ec3da, with the same patch-ids, messages, authors, and dates, and signed. The head's change from f66ac07d4 has the same patch-id as #1684's change from c0235df55 to 392d436cf. At 6a71ec3da: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean.

Restacked on 2026-10-15 onto #1684 c0235df55, whose tree equals ebb27742d's; only #1627's history beneath it was rewritten. The commits became 27c47d49b, 099c93a70, 2035ec164, and f66ac07d4, with the same patch-ids, messages, authors, and dates, and signed. f66ac07d4's tree equals a4ef42919's, so the checks below at a4ef42919 held for it.

Restacked on 2026-10-15 onto #1684 ebb27742d, on feat/horizontal-scaling cf7ae1579. The commits became 0c674aa36, 87cd7815f, 6f512c697, and a4ef42919. The replay was clean, and all four keep their patch-ids, messages, authors, and dates, and are signed. The head's change from a14919f2e has the same patch-id as #1684's change from 22f48998f to ebb27742d. At a4ef42919: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean.

Restacked on 2026-10-14 onto #1684 22f48998f. The commits became d1a97995e, 0f1b5196b, 786412423, and a14919f2e, with the same messages, authors, and dates, and signed. The first, third, and fourth keep their patch-ids. The second keeps its own line where #1670 renamed normalize_filter_field's unused result to _, since the commit uses that result. The head differs from 8d7408d69 only by #1820's patch, the same patch-id as its merge 21f06ddbc. At a14919f2e: ruff check, ruff format --check, ty check as CI runs it, and uv lock --check clean.

Before the restack, at 8d7408d69, on 2026-10-14, the suites ran on its tree, as an unpushed commit a275bf393 with the same tree: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The server and client suites pass: unit, 2,460 passed (7 skipped); integration, 1,691 passed (342 skipped). At 2d045afdf and 0e222b023, the property key and episodic memory unit tests pass (375 and 381); at 26c02a718, the full unit suite passes (2,460, 7 skipped). All four commits are signed.

🤖 Generated with Claude Code

This was referenced Oct 14, 2026
@edwinyyyu
edwinyyyu force-pushed the fix/metadata-key-rule-main branch from 8d7408d to a14919f Compare October 15, 2026 05:53
@xiongzubiao
xiongzubiao force-pushed the feat/horizontal-scaling branch from 101182c to cf7ae15 Compare October 15, 2026 15:31
@edwinyyyu
edwinyyyu force-pushed the fix/metadata-key-rule-main branch 5 times, most recently from 39075cd to d432bf8 Compare October 16, 2026 00:21
edwinyyyu and others added 5 commits November 2, 2026 14:28
…tamp under it

MemMachine and its users share one property key space. A key beginning
with `memmachine_` now belongs to MemMachine: `reserved_property_key`
builds one as `memmachine_<system>_<field>`, and every key is a valid
vector store identifier, `[a-z0-9_]` and at most 32 bytes.

EventMemory writes its timestamp under `memmachine_em_timestamp`. It
refuses an event whose property key is reserved or not a valid
identifier, through `validate_user_property_key`; a key is used exactly
as given, never rewritten.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An EventMemory filter, and a segment store's, named a property in two
ways: `m.<key>` for the key itself, and a bare name for `_<name>`, the
layout one consumer gives its own fields. The bare form tied both to
that consumer's keys.

A filter field is now `timestamp` or `m.<key>` (or `metadata.<key>`).
EventMemory refuses any other field before it searches, and the segment
stores refuse it as well. The vector search filters on the timestamp,
under its reserved key, and on the declared properties. The long-term
memory adapter names its system fields as properties when it hands a
filter to EventMemory, so its client filters are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
MemMachine reserved two key namespaces at once. EventMemory refused a
property key beginning with `memmachine_`, while the long-term memory
adapter wrote its own fields under a leading underscore and refused a
caller key beginning with one. So a caller could not use an underscore,
and the adapter's fields sat outside the namespace meant for them.

The adapter now writes its fields under `memmachine_ltm_<field>`, and
refuses a caller key that is not a legal user property key. A filter
still names a system field by its bare name, and refuses a `m.<key>`
in the reserved namespace.

The reserved namespace is divided by the system that writes each key.
EventMemory refuses only its own `memmachine_em_` keys, in an event
and in a filter, so the component that embeds it can write its own
reserved keys. `is_reserved_property_key` takes the system to ask about
one system's keys.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An episode's metadata keys become event properties, and the event
backend refuses a key that is not a legal user property key. That
refusal came while the episode store and semantic memory were writing
the same episodes in parallel, so the request failed after they had
kept them, and a request over HTTP got a 500.

`MemMachine.add_episodes` now checks every metadata key of every entry
with `validate_user_property_key` before it writes anything, and the
add-memories route answers a ValueError with 422, as the other routes
do. The metadata field's description states the key rule.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The metadata field's description gives the key rule in words instead
of a pattern. EventMemory's filter check and the adapter's conversion
state what holds for a reserved key, and `MemMachine.add_episodes`
says its refusal comes before any store is written.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@edwinyyyu
edwinyyyu force-pushed the fix/metadata-key-rule-main branch from 53f030e to 4fc4aab Compare November 2, 2026 22:33
edwinyyyu and others added 2 commits November 2, 2026 15:36
EventMemory and the segment stores spelled out how a client names a
property in a filter, and EventMemory checked a filter's fields against
that spelling before it searched. The spelling belongs to the filter
language and to the API that accepts a client's filter.

EventMemory no longer checks a filter's fields: the long-term memory
adapter checks a client's filter, and the segment stores refuse a field
that names neither a property nor the timestamp. For the vector search,
EventMemory translates the timestamp to its reserved key and a property
to its key, and passes any other name unchanged. The docstrings describe
a filter over the segments' properties and timestamp.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The docstrings of EventMemory's query and filter translation, the
segment stores' filter, and the adapter's filter mapping named the
timestamp as a special field. They return to their wording on the
base, changed only where it no longer holds: an event memory base
property translates to its reserved key, and a segment store refuses
an unknown field.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@edwinyyyu
edwinyyyu marked this pull request as ready for review November 3, 2026 18:25
@edwinyyyu

Copy link
Copy Markdown
Contributor Author

Self-reviewed (human).

@marvinyu-memverge marvinyu-memverge left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at 013346b.

I read property_keys.py, both memories' paths (EventMemory and the long-term memory adapter), the segment store's field mapping, MemMachine.add_episodes and the add-memories route, as whole files at head.

Observation, no action needed in this PR: the integrations that hand an LLM a free-form metadata record don't state the new key rule. openclaw's memory_store takes metadata as any string-to-string record and merges run_id/user_id into it, and the Strands tool schema just says "Additional metadata as key-value pairs". So a key the model picks, like sourceUrl or source-url, now comes back as a 422 from add-memories. Every in-repo caller I found (the evaluation ingest scripts, crewai, the Strands example, openclaw's own run_id/user_id) already uses legal keys, so nothing in the repo breaks; the tool descriptions are just where an agent would learn the rule. Could be a small follow-up.

Verified:

  • Every key MemMachine writes into an event is under memmachine_: EventMemory's memmachine_em_timestamp and the adapter's eight memmachine_ltm_* fields. The deployment's declared vector store properties (event_backend_indexed_properties) are built from the same constants, so the declaration and the writes can't drift.
  • The longest reserved key, memmachine_ltm_produced_for_id, is 30 bytes, inside the 32-byte budget, and reserved_property_key raises at import if a later field doesn't fit.
  • add_episodes checks every entry's metadata keys before anything is written, and it's the only path into the episode store, episodic memory and semantic memory; the route turns the ValueError into 422.
  • Filters on the event backend: the reserved-key check runs before the adapter maps a bare system name (producer_id) to m.memmachine_ltm_producer_id, so a client's own m.memmachine_* is refused while the mapped name passes. EventMemory demangles it to the vector store key, the segment store reads it from the properties JSON, and timestamp goes to memmachine_em_timestamp and the timestamp column.
  • On the declarative backend the _filterable_metadata_none flag is still an underscore key, but user keys there live under metadata., so it can't collide with the user _ keys this now allows.

The one red check, build-docs (24), is npm not finding @mintlify/link-rot@3.0.1580, not this change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants