[declared schema 1/2] Make a vector store filter only on the properties it declares by edwinyyyu · Pull Request #1628 · MemMachine/MemMachine

edwinyyyu · 2026-09-14T21:39:15Z

Purpose of the change

A partition stored every property of a record and filtered on any key, which made a caller's arbitrary keys part of the store's schema: the SQLite stores kept them in a JSON column and filtered with json_extract, Milvus the keys its schema did not declare in a JSON field, and a filter on a key the store never indexed scanned. Since EventMemory routes a filter on an undeclared key to the segment store ([user properties 1/2]), the vector store need not hold undeclared keys at all.

A partition now stores the properties its store declares and no others. upsert raises UndeclaredPropertyKeyError before anything is sent for a record naming an undeclared key, and PropertyTypeMismatchError for a value of another type than its key declares; query raises UndeclaredPropertyKeyError for a filter naming an undeclared key, and every store evaluates the whole filter language during the search. Both SQLite stores keep one typed, indexed, nullable column per declared key on the records table (sqlite_properties.py), with datetimes as microseconds since the epoch since SQLite has no datetime type; sqlite-vec 0.1.9 rejects NULL in a vec0 metadata column and a declared key is optional per record, so that store keeps the columns on the records table and hands the KNN a rowid IN (SELECT ...) allowlist, evaluating the filter during the search instead of after it. Milvus drops its JSON field and keeps its typed field per declared key; Qdrant, which already wrote one payload field per key, keeps a payload index per declared key.

Since every key a filter may name is now indexed, the Qdrant store creates its collection in strict mode (unindexed_filtering_retrieve and _update false, Qdrant Cloud's default): the server refuses a filter on an unindexed key instead of scanning for it, the same on every server, so a missing index fails a search rather than slowing it. A leaf whose value is of another type than its key declares matches nothing, as on the SQL stores; the Qdrant compiler answers it with a filter no point satisfies, since the server would refuse the condition for the field's index, and Milvus no longer compares an int with a float key. Local mode does not record the setting, so a unit test checks the request and integration tests the server's answer.

declared_schema_contract.py states the contract every backend's test module runs: which records a filtered search admits, over fixtures small enough that every backend searches them exactly, checked after each upsert so an approximate index fails on recall, by name, and not on the filter.

Adaptation for main: the SQLite stores of this stack keep one records table per life of a partition, named by its incarnation ([vector store scale-out 6]), where the origin kept one table per collection with an incarnation column, so the declared columns and their indexes are per table: property_indexes indexes each declared column alone, a store builds a records table's Table once per incarnation from its metadata, and sqlite-vec's KNN statement carries the allowlist alone, with no incarnation or registry clause. On the Qdrant and Milvus stores the checks are the registry-backed base handle's: upsert runs validate_declared_properties where it ran a type check, and query runs validate_supported_filter, which checks that the filter names declared keys only; both checks live in utils.py, where the checks they replace lived. A SQLite datetime column holds the instant, as microseconds since the epoch: the vector stores keep a datetime property's instant only, and the segment store keeps its offset. The Milvus schema test drops its assertion on the JSON field that goes, and the Milvus test for filtering on undeclared keys goes, the contract covering it; the test that a declared datetime is stored as its instant stays. The Qdrant strict-mode tests build their store through the module's _params and create the collection at startup; the origin's TestDeclaredPayloadIndexes is not carried over, since the Qdrant store's tests already check every declared key's payload index. The declared-schema contract gains the lifecycle contract's settle hook, run after each fixture upsert, since the Milvus store reads at Bounded consistency.

Position: directly on feat/horizontal-scaling, which has [vector store scale-out 6] (#1627) merged, beneath [sqlite write transactions 1] (#1844) and [search results, deferred] (#1663), so a query here takes score_threshold under the store's similarity metric; #1663, deferred, would make it cosine. The SQLite fixes are not under this PR, so the SQLite store's upsert is the insert-or-update it was before them, so this PR rewrites every column but the row id on a conflict, which sets a declared key the record no longer holds to NULL (the fixes replace that statement with a delete and a fresh insert); the fixes' tests are not in this branch, and the Qdrant collection is created in strict mode without [qdrant options]' tuning.

Fixes #1790. Part of the fix for #1781.

Stack

27 open PRs: three independent PRs, and the vector store tree of short parallel branches. Every open PR in the tree has feat/horizontal-scaling as its GitHub base, and the independent PRs have main. The branches are in a fork, and a pull request can target only this repository's branches, so the on column gives the order the PRs build on each other. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

#	PR	change	on
—	#1624	Make no memory request create a project	`main`
—	#1786	Refuse a filter value of the wrong type for a datetime column, and answer an invalid list filter with 422 (port of #1620)	`main`
—	#1792	Refuse property values that some store refuses or alters where an episode enters	`main`

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1675 is closed, superseded by #1627. #1839 and #1840 are the event memory recovery stack, and #1684 heads the event memory stack, which its own PRs index; #1684 is listed because it builds on #1840 and #1828 builds on it; #1871 keeps its own Stack section. Until the PRs under it merge, their changes show in a stacked PR's diff.

#	PR	change	on
[vector store scale-out 1/6]	#1671 (merged)	Remove custom sharding from the Qdrant store (port of #1654)	`main`
[vector store scale-out 2/6]	#1733 (merged into `feat/horizontal-scaling`)	Answer vector store queries with record UUIDs and scores, and refuse invalid inputs	`feat/horizontal-scaling`
[vector store scale-out 3/6]	#1734 (merged into `feat/horizontal-scaling`)	Arbitrate vector store collections in a SQL registry, with an incarnation per collection life and a purge	`feat/horizontal-scaling`
[vector store scale-out 4/6]	#1735 (merged into `feat/horizontal-scaling`)	Move the Qdrant store onto the collection registry	`feat/horizontal-scaling`
[vector store scale-out 5/6]	#1736 (merged into `feat/horizontal-scaling`)	Move the Milvus store onto the collection registry, against a Milvus server	`feat/horizontal-scaling`
—	#1775 (merged into `feat/horizontal-scaling`)	Accept attempts exhausted in the lifecycle churn contract, and say which Qdrant operations filter on the incarnation	`feat/horizontal-scaling`
—	#1779 (merged into `feat/horizontal-scaling`)	Classify Qdrant errors by status code alone	`feat/horizontal-scaling`
—	#1813 (merged into `feat/horizontal-scaling`)	Create Qdrant collections with strict mode off	`feat/horizontal-scaling`
—	#1820 (merged into `feat/horizontal-scaling`)	List only active projects in projects/list	`feat/horizontal-scaling`
[user properties 1/2]	#1702 (merged into `feat/horizontal-scaling`)	Keep undeclared properties out of the vector store	`feat/horizontal-scaling`
[user properties 2/2]	#1670 (merged into `feat/horizontal-scaling`)	Remove per-project filterable properties (port of #1606)	`feat/horizontal-scaling`
—	#1824 (merged into `feat/horizontal-scaling`)	Drop the unread logical key from the purge queues	`feat/horizontal-scaling`
—	#1788	Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract	`feat/horizontal-scaling`
—	#1631 (closed)	Superseded by #1733–#1736, which hold its changes split in four, with review changes since	—
[vector store scale-out 6]	#1627 (merged into `feat/horizontal-scaling`)	Give each vector store a fixed schema and string-keyed partitions, built from the deployment's configuration	`feat/horizontal-scaling`
[declared schema 1/2]	#1628 (this PR)	Make a vector store filter only on the properties it declares	`feat/horizontal-scaling`
[sqlite write transactions 1]	#1844	Take SQLite's write lock at BEGIN in every store on a relational SQLite engine	#1628
[session storage 1/2]	#1622	Create a session's storage with the session, never on a request	`feat/horizontal-scaling`
[session storage 2/2]	#1625	Look a partition up with get_partition, and remove open-or-create from both stores and close from the segment store	#1622
[selective filter 1]	#1871	Route filtered queries by selectivity, and search allowlists exhaustively	#1628
[search results, deferred]	#1663	Score every vector search by cosine similarity, and name scores for it (port of #1598's cosine half)	#1628
[declared schema 2/2]	#1616	Close the filter union, and make negation the complement on every backend	#1844
[sqlite store fixes 1/7]	#1460	Publish vector index files atomically (but not durably)	#1844
[sqlite store fixes 2/7]	#1469	Never reuse a row id in SQLiteVectorStore	#1460
[sqlite store fixes 3/7]	#1672	Own the search engine's concurrency in the store, not in each engine (port of #1612)	#1469
[sqlite store fixes 4/7]	#1673	Serialize a partition's writes so the engine sees them in order (port of #1607)	#1672
[sqlite store fixes 5/7]	#1674	Refuse a pending row replay cannot honor, instead of dropping it (port of #1608)	#1673
[sqlite store fixes 6/7]	#1675 (closed)	Superseded by #1627, which takes SQLite's write lock at BEGIN where the engine is created	—
[sqlite store fixes 7/7]	#1676	Give every write a fresh row id, so a key names one version (port of #1610)	#1674
[sqlite store fixes 8]	#1843	Hold the engine lock until a cancelled engine call's thread finishes	#1676
[qdrant options]	#1618	Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization	#1844
[milvus options]	#1741	Let a deployment tune a Milvus collection's vector index and its searches	#1618
—	#1841	Pack vector store upserts by serialized size, and state what cancelling one guarantees	#1844
[event memory recovery 1/2]	#1839	Give EventMemory a store's shape, with a handle per partition	#1841
[event memory recovery 2]	#1840	Keep event memory's two stores consistent through failed ingests and forgets	#1839
[event memory 1/6]	#1684	Add time bounds, sources and expansion to EventMemory (port of #1597, without session)	#1840
[property keys]	#1828	Refuse an illegal metadata key before any store writes, and reserve one key prefix	#1684
[schema migration 1/2]	#1825	Add `memmachine-server migrate`, which creates every configured store's schema before servers start	#1844
[schema migration 2/2]	#1826	Make startup check what migrate created, and create nothing at startup	#1825

This PR is its 16 commits, ec2c66283, 48058c8b7, 659cc0b13, c8ca16bc9, ea2dc2894, f34d16afd, d6972a5fa, 96632d6da, 274264523, 42edae2e2, ac43aedbe, 1664a87e4, df716e47b, 0d8d22d4b, 115cd6d99, aad515896, directly on feat/horizontal-scaling. #1844, #1871, and #1663 are stacked on it, in parallel.

Verification

At this PR's head aad515896, on 2026-10-30, two commits on 0d8d22d4b from xiongzubiao's review, which Edwin asked to address: 115cd6d99 adds the declared-schema contract's check that a re-upsert clears the declared keys it drops, and states it in VectorStorePartition.upsert's docstring (a replaced record holds only the vector and properties it is upserted with); aad515896 tidies the contract tests, naming the node test for what it checks (test_every_node_admits_the_matching_records), removing a helper nothing here calls, and merging an import. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The vector store unit tests pass, 396 tests (5 skipped), the new check among them on both SQLite stores; it runs on Qdrant and Milvus with the integration tests, which did not run here.

Before xiongzubiao's review changes: At this PR's head 0d8d22d4b, on 2026-10-28, one commit on df716e47b from leomem's review, which Edwin asked to address: property_column_names derives the column names from property_column_name, the rule property_columns and property_indexes use, instead of building every declared key's Column to read its name back, once per record a SQLite store writes. The same names, in the declared order. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. Both SQLite stores' unit tests pass, 219 tests.

Before the column names change: At this PR's head df716e47b, on 2026-10-28, a revert on 1664a87e4, at Edwin's request: df716e47b reverts ac43aedbe, since deleting the oversized Milvus test's partition only leaves a tombstone, which the test never purges, so it reclaimed nothing. The tree differs from 42edae2e2 only by 1664a87e4's removal of extend_existing. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before the revert: At this PR's head 1664a87e4, on 2026-10-28, two commits on 42edae2e2 from a code review Edwin approved: ac43aedbe deletes the oversized Milvus upsert test's partition when the test is done, as the module's other tests with a store of their own do, and 1664a87e4 builds a SQLite records table without extend_existing, since the store builds a Table only when none of that name exists. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The vector store and filter unit tests pass, 542 tests (5 skipped); the Milvus test is an integration test and did not run here.

Before the code review's two fixes: At this PR's head 42edae2e2, on 2026-10-28, one commit on 274264523 from Edwin's review: is_valid_identifier's docstring says what the rule is for, a name portable wherever a backend restricts names, in place of describing its pattern. Docstring only. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before the identifier docstring: At this PR's head 274264523, on 2026-10-27, two more commits from Edwin's review, on d6972a5fa: 96632d6da gives the package one dependency direction, with data_types.py at the bottom importing nothing from it, by moving the identifier rule (is_valid_identifier and its two constants) into data_types.py, so utils.py imports the errors by name; and 274264523 sorts the interface's Raises sections by exception name. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 2312 tests (8 skipped). The integration tests did not run here.

Before the second round of review changes: At this PR's head d6972a5fa, on 2026-10-27, four commits on 659cc0b13 from Edwin's review: c8ca16bc9 removes supported_filter_nodes, UnsupportedFilterError, the node half of validate_supported_filter, and filter_nodes, since every store evaluates the whole filter language, which the partition interface now states and the declared-schema contract checks node by node; ea2dc2894 moves validate_declared_properties and validate_supported_filter into utils.py beside the other validators, raising their errors through the data_types module, which imports utils.py; f34d16afd names the SQLite stores' declared-property module sqlite_properties.py; and d6972a5fa says what Qdrant's strict mode is for, in the store's comment and the design document. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 2312 tests (8 skipped). The integration tests did not run here.

Before Edwin's review changes: At this PR's head 659cc0b13, on 2026-10-27, moved beneath [sqlite write transactions 1] (#1844): its three commits replay directly on feat/horizontal-scaling at 805ea2177, and #1844 is rebuilt on it. Its tree is the previous head 135184cd6's without #1844's changes; the one conflict, at test_milvus_vector_store.py's imports, drops enable_sqlite_explicit_begin, which only #1844's change to that module uses, beside the decode_properties import this PR removes. Its commits are re-signed, with their messages and authors unchanged. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The vector store and filter unit tests pass, 542 tests (5 skipped). The integration tests did not run here.

Before #1844 moved above this PR: At this PR's head 135184cd6, on 2026-10-27, replayed onto [sqlite write transactions 1] (#1844) at 2cb305a8e, after [vector store scale-out 6] (#1627) merged into feat/horizontal-scaling as 805ea2177, a squash whose tree is 8c2351dbe's: its tree is the previous head 9ff25a97c's, so the checks and tests at that head stand. Its commits are re-signed, with their messages and authors unchanged.

Before the replay onto the merged #1627: At this PR's head 9ff25a97c, on 2026-10-27, a third commit on 4b1210953: the oversized Milvus upsert test's own store gives its requests longer than the module's request timeout. pymilvus checks a VARCHAR value one character at a time while it builds a row upsert request, inside the request's timeout, so on a slow CI runner building the test's first request, which is over Milvus's size limit, outlasted the timeout before Milvus could refuse it; the store's code is unchanged. The PRs on this one still sit on 4b1210953 and take the commit at their next rebuild. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The Milvus store's unit tests pass, 9 tests; its integration tests did not run here.

Before the oversized upsert test's timeout: At this PR's head 4b1210953, on 2026-10-26, rebuilt onto [sqlite write transactions 1] (#1844) at c9445910f, on #1627's 8c2351dbe: beneath it are #1627's changes since e38e8003e (each store's schema passed as one VectorStoreSchema, store_schema; SQLite partitions named by an incarnation and created as the other stores create theirs; the shared SQL engine setup and write transactions in common/sql.py; and every session's event memory served from the deployment's configuration, in one store) and #1844's SQLite write transactions. This PR's own lines follow them: its checks and compilers read the declared properties from the store's store_schema; a SQLite records table, one per incarnation, is built once per incarnation from the store's metadata; both SQLite stores' upserts validate, then open write_session and check the partition is current, as #1627's do; the in-memory test partition takes a store_schema; the SQLite store's class docstring now says that a declared property is a typed, indexed column, as the store's indexed_properties parameter said before #1627 moved it into the schema; and the declared-schema contract names its fixture partition. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 2317 tests (8 skipped). The integration tests did not run here.

Before the rebuild onto #1627's 8c2351dbe: At this PR's head 0bef1379d, on 2026-10-20, its tree is the previous head b077a3058's with #1627's accuracy fixes, in text only: the partition registry and purge design documents, the database manager's comments and docstrings, the SQLite store's module docstring, the partition registry configuration's descriptions, and other docstrings that still described collections. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 426 tests (5 skipped). No integration tests ran this time, since only comments, docstrings, descriptions, and documentation changed.

Before #1627's accuracy fixes: At this PR's head b077a3058, on 2026-10-19, its tree is the previous head 35e60c922's with #1627's second review's fixes: the Qdrant and Milvus tests that partitions share the store's collection now write through two partitions and read the store's one native collection; the store tests name their partition fixture partition; and docstrings, comments, and the configuration and upgrade documentation are corrected. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 426 tests (5 skipped).

Before #1627's second review's fixes: At this PR's head 35e60c922, on 2026-10-19, its tree is the previous head 0375a60ed's with #1627's review fixes: the database manager answers a repeated (backend, vector store name) with the store it built, without comparing parameters, and keeps a backend's shared SQLite engine until it closes, even when a store's build fails; its test that one SQLite backend's stores share the engine can now fail; the partition registry's parameters check the vector store name with the shared validate_vector_store_name; and semantic memory's local name for the backend key is backend. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 426 tests (5 skipped).

Before #1627's review fixes: At this PR's head 0375a60ed, on 2026-10-19, its tree is the previous head 5d9daf636's with design/vector_store_partition_registry.md listing the lookup as "open", the name it has from #1627 until #1625, where it said "get". Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. Outside that document the tree is the previous head's, so the test runs below stand.

Before the partition registry document's wording fix: At this PR's head 5d9daf636, on 2026-10-19, its tree is the previous head ee830b047's with #1824 beneath it, which drops the purge queues' unread logical key, and #1627 dropping the same key from the partition registry's tombstones and removing the collection registry design document its partition registry document replaces. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 427 tests (5 skipped).

Before the rebase onto #1824's merge: At this PR's head ee830b047, on 2026-10-16, its tree is the previous head ad482226d's with the configuration's check of a provided tombstone_retention_seconds named _validate_tombstone_retention, where it was _require_retention_floor. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 427 tests (5 skipped).

Before the retention check's rename: At this PR's head ad482226d, on 2026-10-16, its tree is the previous head ace37a96d's with every vector store check of a value that raises named validate_, as validate_partition_key already was: validate_declared_types, validate_query_vector, validate_score_threshold, validate_limit, and validate_dimensions from the base, and the PRs' own checks in their turn. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 427 tests (5 skipped).

Before the raising checks took the validate_ prefix: At this PR's head ace37a96d, on 2026-10-16, its tree is the previous head b987f55b1's with #1627's changes: the VectorStore interface drops vector_store_name, vector_dimensions, similarity_metric, and indexed_properties, and the partition handle drops partition_key; a partition key may be any string of up to 255 bytes without NUL, and the SQLite stores name a partition's tables and index file by a SHA-256 digest of it; and the checks that return a bool are is_valid_identifier and is_valid_filter. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 427 tests (5 skipped).

Before the store's properties left its interface: At this PR's head b987f55b1, on 2026-10-15: its tree is the previous head aeb3a5c67's without #1627's per-partition schema record (PartitionSchema, the registry's and the SQLite stores' schema column, and VectorStorePartitionSchemaMismatchError), and with require_partition_key named validate_partition_key; ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The SQLite stores' JSON import goes with the schema column. The unit tests pass, 421 tests.

Before the per-partition schema record went: At this PR's head aeb3a5c67, on 2026-10-15: its tree is the previous head 4c6536039's with the partition lookup named open_partition and the segment store's close_partition kept, since #1625 renames the one and removes the other; ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before the lookup's rename moved into #1625: At this PR's head 4c6536039, on 2026-10-15: its tree is the previous head cc0ae967a's without the segment store's close_partition; ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before #1627's first commit took the removal of close_partition: At this PR's head cc0ae967a, on 2026-10-15, restacked on #1627's rewritten history: its tree is the previous head 097bd3c89's; ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before #1627's history was rewritten: At this PR's head 097bd3c89, on 2026-10-15, rebased onto feat/horizontal-scaling at cf7ae1579, which carries main through #1774: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before the rebase onto cf7ae1579: At this PR's head 6f925f7c9, on 2026-10-15, rebased onto feat/horizontal-scaling with #1702 and #1670 merged (101182c18): ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before the rebase onto #1670's merge: At this PR's head 4452e6f18, on 2026-10-13, rebuilt on #1702's 6631b5c77: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean; the episodic memory and filter tests pass, 478 tests (1 skipped).

Before the rebuild on #1702's filter routing: At this PR's head d6d9b5651, on 2026-10-09, on feat/horizontal-scaling with #1813 merged: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before the rebase onto #1813's merge: At this PR's head 6420e9096, on 2026-10-09: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean; the vector store, filter, and episodic memory suites without integration tests pass, 789 tests. The vector store, resource manager, episodic memory, and semantic storage suites, unit and integration together, pass, 1549 tests (21 skipped), against PostgreSQL 16, Neo4j 5.23, Qdrant 1.19.1, and Milvus 2.6.24 in containers.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

A partition stored every property of a record and filtered on any key, which made a caller's arbitrary keys part of the store's schema: the SQLite stores kept them in a JSON column and filtered with json_extract, Milvus the keys its schema did not declare in a JSON field, and a filter on a key the store never indexed scanned. Since EventMemory routes a filter on an undeclared key to the segment store, the vector store need not hold undeclared keys at all. A partition now stores the properties its store declares and no others. `upsert` raises UndeclaredPropertyKeyError before anything is sent for a record naming an undeclared key, and PropertyTypeMismatchError for a value of another type than its key declares; `query` raises UndeclaredPropertyKeyError for a filter naming an undeclared key and UnsupportedFilterError for a node outside the partition's `supported_filter_nodes`. Both SQLite stores keep one typed, indexed, nullable column per declared key on the records table (sql_columns.py); sqlite-vec 0.1.9 rejects NULL in a vec0 metadata column and a declared key is optional per record, so that store keeps the columns on the records table and hands the KNN a `rowid IN (SELECT ...)` allowlist, evaluating the filter during the search instead of after it. Qdrant drops the JSON copy and keeps a payload field per declared key; Milvus drops its JSON field and keeps its typed field per declared key. Datetimes are stored as microseconds since the epoch where a backend has no datetime type. Since every key a filter may name is now indexed, the Qdrant store creates its collection in strict mode (`unindexed_filtering_retrieve` and `_update` false, Qdrant Cloud's default): a filter on an unindexed key is refused by the server instead of scanned for. A leaf whose value is of another type than its key declares matches nothing, as on the SQL stores; the Qdrant compiler answers it with a filter no point satisfies, since the server would refuse the condition for the field's index, and Milvus no longer compares an int with a float key. Local mode does not record the setting, so a unit test checks the request and integration tests the server's answer. `declared_schema_contract.py` states the contract every backend's test module runs: which records a filtered search admits, over fixtures small enough that every backend searches them exactly, checked after each upsert so an approximate index fails on recall, by name, and not on the filter. On the registry-backed stores, the checks are the base handle's: `upsert` runs validate_declared_properties in place of the type check it ran, and `query` runs validate_supported_filter against the subclass's `supported_filter_nodes`, which each subclass now implements. A datetime column on the SQLite stores has a `tz_<key>` column beside it holding the UTC offset in seconds, written with the value and read by no filter, so a stored datetime is the value written, as on Milvus, Qdrant and the segment store; the microseconds column alone would keep only the instant. The Qdrant and Milvus design documents describe the declared-only properties and Qdrant's strict mode. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

A declared datetime on the SQLite stores had a `tz_<key>` column beside its microseconds column, holding the UTC offset in seconds that no filter reads. The vector stores keep a datetime property's instant only, as MemMachine#1736 now does for Milvus and MemMachine#1788 for Qdrant, and the segment store keeps the offset, so the column, `offset_column_name`, and the value written to it go: each declared property is one column, and a datetime stays microseconds since the epoch. The tests read a stored datetime back as its instant in UTC, and the roundtrip test checks that the stored instant equals the written one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

pymilvus checks a VARCHAR value one character at a time while it builds a row upsert request, and the request's timeout covers that build. The test's batch, over Milvus's request size limit, is mostly one declared VARCHAR property, so on a slow runner building the first request outlasts the module's request timeout before Milvus can refuse the request as too large. The test's own store gives its requests longer; the halving it checks is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Every store declared all six filter nodes, the whole closed union, so the node check behind `supported_filter_nodes` could not fail, and nothing outside the stores read the property. The sqlite-vec store evaluates the filter in SQL on the records table and hands vec0 a rowid allowlist, so vec0's limits on its own filter constraints never applied. The abstract property and its implementations go, with `UnsupportedFilterError`, the node half of `validate_supported_filter`, and `filter_nodes`. The partition interface states that every store evaluates the whole filter language during a search, and the declared-schema contract checks that each node does. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…tors `validate_declared_properties` and `validate_supported_filter` are checks of the same kind as `validate_partition_key`, `validate_query_vector` and `validate_limit`, and their predecessors lived in utils.py, so they move there and declared_properties.py goes. utils.py raises the errors through the data_types module, since data_types imports utils.py and its errors are not defined yet while utils.py loads. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The module holds the declared-property columns, indexes, values, and filter compiler of the two SQLite stores, and only theirs: its datetimes are integers because SQLite has no datetime type. sql_columns.py named neither SQLite nor properties. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The store filters only on indexed keys, so strict mode changes no search it sends; what it does is make the server refuse a filter on an unindexed key instead of scanning for it, the same on every server, so a missing index fails a search rather than slowing it. The comment and the design document said only that the server may refuse such a filter, which read like a speed-up. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The vector store package has one dependency direction: data_types.py, the types, their errors, and the rules defining their valid values, is the bottom and imports nothing from the package; utils.py's argument checks sit above it, then the interfaces, then the stores. Every import followed it but one, data_types.py importing `is_valid_identifier` from utils.py, which made utils.py reach the errors through a module import. The identifier rule says which property keys and store names the types accept, so `is_valid_identifier` and its two constants move into data_types.py; utils.py imports the errors by name, and the stores and the partition registry take the identifier length bound, which sizes their columns, from data_types.py. `validate_partition_key` stays in utils.py, since a partition key is a call argument, not a field of any type. data_types.py's docstring states the rule. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

They followed the order the implementations run their checks, which a reader of the interface cannot see; by name, every section in the file reads the same way. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

`is_valid_identifier`'s docstring described its pattern. It now states what the rule is: a value portable as a name wherever a backend restricts names, which is why it allows only lowercase letters, digits, and underscores, and at most 32 bytes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The test builds a store and a partition of its own and left the partition behind, so each run kept its 1,200 records of 60,000 bytes in the session's Milvus container. It deletes the partition at the end, as the module's other tests with a store of their own do. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The store returns a records table already in its metadata, so it builds a `Table` only when none of that name exists, and the flag that let a rebuild redefine one has nothing left to do. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…s done" This reverts commit ac43aed. Deleting a partition only unregisters it and leaves a tombstone; its entities are reclaimed by a purge once the tombstone's retention has passed, and the test, whose registry lives in its own temporary database, never purges. The delete reclaimed nothing, so it only suggested a cleanup that does not happen. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

edwinyyyu · 2026-10-28T23:15:35Z

Self-reviewed.

leomem

I checked each finding with a small failing test, run on this PR's head (df716e4). Docker wasn't available, so the contract tests ran against SQLite, sqlite-vec and Qdrant in local mode (AsyncQdrantClient(location=":memory:")), and the Milvus findings were checked against _filter_expression directly. Server-mode Qdrant and Milvus still need confirming in CI.

The contract tests are written as methods on DeclaredSchemaContract, so they can be added to declared_schema_contract.py and run on every backend. The int-bound-on-a-float-key regression (Milvus and Qdrant) and the mistyped != disagreement are the two I'd fix before merging.

leomem · 2026-10-29T00:34:32Z

-        target = f"{_DECLARED_FIELD_PREFIX}{expr.field}"
-        type_check = ""
-        render = _declared_property_value_literal
+    values = [value for value in values if type(value) is declared[expr.field]]


An int bound on a float key now matches nothing. The filter parser reads score > 1 as an int ('1'.isdigit()), type(1) is float is false, and the leaf compiles to _FALSE_EXPR. Before this PR, _is_comparable_with_type explicitly let an int compare with a float field. Qdrant has the same regression at qdrant_vector_store.py:114, where the old code sent a range filter on the float index.

Verified: the contract test below fails on SQLite, sqlite-vec and Qdrant (local mode), each admitting 0 of 1 records. Milvus compiles the leaf to id in [].

@pytest.mark.asyncio async def test_an_int_bound_on_a_float_key_compares_numerically(self, partition): high = _record([1.0, 0.0, 0.0], score=1.5) low = _record([0.0, 1.0, 0.0], score=0.5) await self._store(partition, [high, low]) # `score > 1` parses to an int bound. assert await _admitted( partition, Comparison(field="score", op=">", value=1) ) == {high.uuid}

Accepting an int (but not a bool) wherever a float is declared, in each backend's type check, would restore the old behavior.

This is intended in this PR, and #1616 fixes it. #1616 is stacked on this PR (#1628 → #1844 → #1616), and both go into feat/horizontal-scaling before that branch reaches main. Edwin ruled on Oct 7 that a filter is bound to the declared schema before any store compiles it. An int for a float key becomes the float it equals, and any other mismatch raises PropertyTypeMismatchError. #1616's bind_filter (vector_store/utils.py) does this in one place for every store, so score > 1 on a float key admits 1.5 on every backend.

🤖 Generated with Claude Code

leomem · 2026-10-29T00:34:32Z

+        """Convert a FilterExpr tree into a Qdrant Filter over the declared keys."""
+        build = QdrantVectorStorePartition._build_qdrant_filter
        if isinstance(expr, FilterComparison):
+            if type(expr.value) is not indexed_properties[expr.field]:


With a value of the wrong type, != short-circuits to _NO_MATCH, so age != "x" matches no record, while not (age = "x") wraps _NO_MATCH in must_not and matches every record. SQLite does the same (_compile_leaf returns false() for !=). Milvus rewrites != as negated equality, so on Milvus both forms match every record. The same filter therefore gives different answers depending on the backend and on how it's written.

Verified on SQLite, sqlite-vec and Qdrant (local mode): != admitted 0 records and not (=) admitted 2. On Milvus, _filter_expression gives (not (id in [])) || (_p_age is null) for the != leaf.

@pytest.mark.asyncio async def test_mistyped_not_equal_agrees_with_negated_equality(self, partition): held = _record([1.0, 0.0, 0.0], age=30) lacking = _record([0.0, 1.0, 0.0], name="bob") await self._store(partition, [held, lacking]) not_equal = await _admitted(partition, Comparison(field="age", op="!=", value="x")) negated = await _admitted(partition, Not(Comparison(field="age", op="=", value="x"))) assert not_equal == negated

Before this PR, Qdrant's must_not match returned every record here.

Fixed by #1616, which is stacked on this PR. Its parser reads != as Not(Equals(...)), so age != "x" and not (age = "x") are the same expression. Its bind_filter raises PropertyTypeMismatchError for a value whose type differs from the key's declared type, before any backend compiles the filter. Both forms are therefore refused on every backend, and no store keeps a branch where a mismatch matches nothing.

🤖 Generated with Claude Code

leomem · 2026-10-29T00:34:32Z

+    if isinstance(expr, IsNull):
+        return column.is_(None)
+    if isinstance(expr, In):
+        return column.in_(expr.values) if type(expr.values[0]) is declared else false()


expr.values[0] has no guard for an empty list, so In(field="name", values=[]) raises IndexError from query() on both SQLite stores. The removed _compile_properties_json_leaf returned false() for an empty IN, and Milvus and Qdrant still match nothing for it.

Verified: fails on SQLite and sqlite-vec with IndexError: list index out of range at this line, and passes on Qdrant.

@pytest.mark.asyncio async def test_an_empty_in_matches_nothing(self, partition): await self._store(partition, [_record([1.0, 0.0, 0.0], name="alice")]) assert await _admitted(partition, In(field="name", values=[])) == set()

Fixed by #1616, which is stacked on this PR: the SQLite compiler returns false() for an empty In, matching Milvus and Qdrant, which already match nothing. The filter parser itself can't produce an empty In, since an IN (...) list takes at least one value.

🤖 Generated with Claude Code

leomem · 2026-10-29T00:34:33Z

+    if isinstance(expr, IsNull):
+        return column.is_(None)
+    if isinstance(expr, In):
+        return column.in_(expr.values) if type(expr.values[0]) is declared else false()


This IN branch passes expr.values to column.in_() as they are. A datetime column is a BigInteger of epoch microseconds, and only the Comparison branch converts the bound. In.__post_init__ normalizes datetime members, so it does expect them at runtime, but on SQLite they never match.

Verified: SQLite and sqlite-vec admit 0 of 1 records. Qdrant fails differently: MatchAny rejects the datetime with a pydantic ValidationError at qdrant_vector_store.py:223.

@pytest.mark.asyncio async def test_in_matches_a_datetime_member(self, partition): record = _record([1.0, 0.0, 0.0], created_at=_INSTANT) await self._store(partition, [record]) assert await _admitted( partition, In(field="created_at", values=[_INSTANT]) ) == {record.uuid}

If datetimes aren't meant to be IN members (the annotation says list[int] | list[str]), rejecting them up front would also work. Right now the result depends on the backend.

Datetimes aren't meant to be In members, so #1616, which is stacked on this PR, rejects them up front. There, In.values is typed tuple[int, ...] | tuple[str, ...], the datetime normalization in __post_init__ is gone, and the parser builds an IN list only from ints or strings.

🤖 Generated with Claude Code

leomem · 2026-10-29T00:34:33Z

        if isinstance(expr, FilterIn):
+            if (
+                expr.values
+                and type(expr.values[0]) is not indexed_properties[expr.field]


For IN, Qdrant and SQLite check only the type of values[0] and then send the whole list, while Milvus filters each member by type. In(field="name", values=[1, "alice"]) matches nothing on Qdrant and SQLite, but Milvus drops the 1 and matches "alice".

Verified: SQLite, sqlite-vec and Qdrant (local mode) admit 0 of 1 records, while Milvus compiles it to _p_name in ["alice"].

@pytest.mark.asyncio async def test_a_mistyped_in_member_does_not_hide_the_others(self, partition): alice = _record([1.0, 0.0, 0.0], name="alice") await self._store(partition, [alice]) assert await _admitted( partition, In(field="name", values=[1, "alice"]) ) == {alice.uuid}

Filtering members the way Milvus does, in one shared place, would make all three agree.

Fixed by #1616, which is stacked on this PR. Three checks now apply:

In takes all ints or all strings, and its __post_init__ raises TypeError for a mixed list or a bool.

The parser refuses a mixed list (Mixed types in IN list).

bind_filter checks every member against the key's declared type before any backend compiles the filter.

So In(field="name", values=[1, "alice"]) is refused before it reaches a backend.

🤖 Generated with Claude Code

leomem · 2026-10-29T00:34:33Z

+    ) -> models.Filter:
+        """Convert a FilterExpr tree into a Qdrant Filter over the declared keys."""
+        build = QdrantVectorStorePartition._build_qdrant_filter
        if isinstance(expr, FilterComparison):


An ordering comparison on a str key passes this type check (str is str) and then reaches _build_qdrant_comparison, which raises TypeError: Range filter on 'name' requires a numeric or datetime value. The new VectorStorePartition docstring says "Every store evaluates the whole filter language during a search", and SQLite and Milvus do evaluate name > "b".

Verified: passes on SQLite and sqlite-vec, and raises the TypeError on Qdrant (local mode).

@pytest.mark.asyncio async def test_a_str_key_orders(self, partition): alice = _record([1.0, 0.0, 0.0], name="alice") bob = _record([0.0, 1.0, 0.0], name="bob") await self._store(partition, [alice, bob]) assert await _admitted( partition, Comparison(field="name", op=">", value="b") ) == {bob.uuid}

Qdrant's keyword index has no range support, so the honest fix may be to narrow the docstring and reject this up front with a clear error.

Edwin decided that ordering comparisons take numbers and dates only. Booleans have no order, and a string's order depends on how a store encodes it. #1616, which is stacked on this PR, implements that:

Ordering.value is typed int | float | datetime, and its __post_init__ rejects a bool.

The parser refuses name > "b" with a FilterParseError, telling the caller to use a number or a date().

With that, the "whole filter language" in the docstring holds on Qdrant as well.

🤖 Generated with Claude Code

leomem · 2026-10-29T00:34:33Z

-                # instead, and a server may turn it on for new collections by
-                # default, so it is turned off here.
-                strict_mode_config=models.StrictModeConfig(enabled=False),
+                strict_mode_config=QdrantVectorStore._STRICT_MODE,


Strict mode is set only when create_collection succeeds. A collection created by the previous code (with StrictModeConfig(enabled=False)) hits the already-exists branch, and startup never calls update_collection, so strict mode stays off there. The guarantee that a missing index fails the search only holds for new collections, and existing deployments keep falling back to scans.

Verified with a local-mode client whose create_collection answers 409 as a server would: startup made no update_collection call.

@pytest.mark.asyncio async def test_startup_turns_strict_mode_on_for_an_existing_collection( self, monkeypatch, registry_engine ): client = AsyncQdrantClient(location=":memory:") await client.create_collection( collection_name=VECTOR_STORE_NAME, vectors_config=models.VectorParams(size=VECTOR_DIM, distance=models.Distance.COSINE), strict_mode_config=models.StrictModeConfig(enabled=False), ) updated = [] original = client.update_collection async def existing_create_collection(*args, **kwargs): raise UnexpectedResponse(409, "Conflict", b"already exists", httpx.Headers()) async def recording_update_collection(*args, **kwargs): updated.append(kwargs.get("strict_mode_config")) return await original(*args, **kwargs) monkeypatch.setattr(client, "create_collection", existing_create_collection) monkeypatch.setattr(client, "update_collection", recording_update_collection) await QdrantVectorStore(await _params(client, registry_engine)).startup() assert any(config is not None and config.enabled for config in updated)

Only collections created by feat/horizontal-scaling builds between #1813 and this PR can be in that state. #1813 never reached main, so no released deployment has one. On such a collection, startup still creates a payload index for every declared key, because the index loop runs after the already-exists guard. Filters therefore use indexes and return the same results; the only thing missing is the loud failure if an index were ever absent. Recreating such a development collection covers it, or turning strict mode on once with update_collection, so this PR leaves the startup path as it is.

🤖 Generated with Claude Code

leomem · 2026-10-29T00:34:33Z

+    ]
+
+
+def property_column_values(


property_column_values calls property_column_names, which calls property_columns and builds a new SQLAlchemy Column for every declared key. That happens once per record on every upsert, just to read the column names. Computing {key: property_column_name(key)} once, or reading table.c, would avoid it.

Verified: upserting 100 records into a SQLite partition called property_columns 100 times.

@pytest.mark.asyncio async def test_upsert_does_not_build_columns_per_record(self, monkeypatch, partition): from memmachine_server.common.vector_store import sqlite_properties calls = 0 original = sqlite_properties.property_columns def counting_property_columns(*args, **kwargs): nonlocal calls calls += 1 return original(*args, **kwargs) monkeypatch.setattr(sqlite_properties, "property_columns", counting_property_columns) await partition.upsert(records=[ Record(uuid=uuid4(), vector=[1.0, 0.0, 0.0], properties={"age": i}) for i in range(100) ]) assert calls <= 1

Fixed in 0d8d22d. property_column_names now derives each name from property_column_name(key), the rule property_indexes already uses, instead of building a Column per key to read its name. Names and order are unchanged, and nothing is cached. With EventMemory's schema at this PR, property_column_values went from 2.3 µs to 0.4 µs per record. I added no test, since behavior is unchanged and a call count would test the implementation.

🤖 Generated with Claude Code

leomem · 2026-10-29T00:34:33Z

+    raise TypeError(f"Unsupported filter expression type: {type(expr)}")
+
+
+def _compare(column: ColumnElement, op: str, bound: object) -> ColumnElement[bool]:


_compare duplicates the operator table in common/filter/sql_filter_util.py (_COMPARISON_OPS and _get_op), so _get_op(op)(column, bound) could replace it.

More broadly, each backend has its own copy of the rule that a value of the wrong type matches nothing (Milvus, Qdrant and here), and the copies have already drifted apart, which is where the != and mixed-IN differences in the other comments come from. Rewriting mistyped leaves once, in validate_supported_filter or in the registry-backed query, before any backend compiles the filter, would leave one rule to maintain. No test for this one, since it's about structure.

#1616, which is stacked on this PR, removes the per-backend copies of the mismatch rule. Its bind_filter (vector_store/utils.py) binds every filter to the declared schema once, raising on a mismatch before any backend compiles the filter, so the backends can no longer drift apart on it. #1616 also moves the operator table into sql_filter_util.py as ORDERING_OPS, which the SQLite compiler uses in place of _compare.

🤖 Generated with Claude Code

`property_column_names` built every declared key's `Column` and type to read back their names, once per record a SQLite store writes. It derives the names from the naming rule, `property_column_name`, which `property_columns` and `property_indexes` use too: the same names, in the declared order. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

edwinyyyu · 2026-10-29T03:07:16Z

Thanks for checking each of these with a test. Here is where the nine comments stand:

Fixed by [declared schema 2/2] Close the filter union, type each filter field when it is parsed, check records at upsert, and make negation the complement on every backend but NebulaGraph #1616: the int bound on a float key, the mistyped !=, the empty and datetime In, the mixed In, ordering a str key, and the per-backend copies of the mismatch rule. [declared schema 2/2] Close the filter union, type each filter field when it is parsed, check records at upsert, and make negation the complement on every backend but NebulaGraph #1616 is stacked on this PR ([declared schema 1/2] Make a vector store filter only on the properties it declares #1628 → [sqlite write transactions 1] Take SQLite's write lock at BEGIN in every store on a relational SQLite engine #1844 → [declared schema 2/2] Close the filter union, type each filter field when it is parsed, check records at upsert, and make negation the complement on every backend but NebulaGraph #1616). It binds every filter to the declared schema once, in bind_filter, before any backend compiles it, so both fixes you'd want before merging land in feat/horizontal-scaling before the branch reaches main.
Fixed here: column names for upserts, in 0d8d22d.
No change: strict mode on existing collections. Only unreleased development builds can have such a collection, and it still gets every payload index. The reply on that thread has the details.

🤖 Generated with Claude Code

marvinyu-memverge

Approving at 0d8d22d.

What I checked at this head, beyond the diff:

Callers. EventMemory is the only caller that sends properties, and it already writes only _timestamp and the declared keys (_build_derivative_record) and hands the vector store only the declared part of a filter (extract_filter(..., allowed_fields=...)). VectorStoreSemanticStorage sends neither. So no server path raises the new UndeclaredPropertyKeyError today.
SQLite. The per-key indexes are now Index objects on the records Table rather than CREATE INDEX statements, and both stores create the table with create_all(tables=[records_table]), which creates them with it. Datetimes go in as epoch microseconds on both the write and the filter bound. The contract's test_the_filter_is_evaluated_during_the_search (one admitted record behind five nearer excluded ones, limit 1) is what shows sqlite-vec takes the rowid IN (SELECT ...) allowlist into the KNN rather than filtering after it. The last commit's property_column_names gives the same names in the same order as property_columns.
Qdrant strict mode. Every declared type maps to a payload index type, the incarnation keeps its index, and the purge's scroll and delete filter on the incarnation only, so the server has nothing of ours to refuse.
Milvus. 60 properties on 2.6 checks out with the JSON field gone (64 fields minus id, record_uuid, partition_key and vector).

xiongzubiao

Reviewed at 0d8d22d. #1790 is fixed, and the declared-schema checks run before any I/O on all four stores. One test gap and three nits, inline; none blocks.

A record's declared properties are columns or fields now, not one value replaced whole, so a store keeps a dropped key's old value unless its update sets every declared key. The contract checks that an upserted record holds only the properties it carries, and the partition interface says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The node test checks which records each node admits, not that the filter runs during the search, so its name says that. `_presence_records` has no caller here and goes to the change whose tests use it. The `timezone` import joins the module's other datetime imports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

xiongzubiao

All four addressed at aad5158, thanks. The re-upsert test passes on both SQLite stores here (221 passed across the two suites).

edwinyyyu force-pushed the feat/vector-store-declared-properties-speedkick branch 5 times, most recently from a297965 to a1e7fd0 Compare September 14, 2026 23:16

edwinyyyu changed the title ~~[vector store 9/13] Make a vector store filter only on the properties it declares (speedkick)~~ [vector store 9/12] Make a vector store filter only on the properties it declares (speedkick) Sep 14, 2026

edwinyyyu force-pushed the feat/vector-store-declared-properties-speedkick branch 2 times, most recently from a297965 to 21ccc01 Compare September 15, 2026 17:26

edwinyyyu mentioned this pull request Sep 15, 2026

[vector store scale-out] Mint an incarnation per collection life in a SQL-arbitrated registry, so any process may serve any Qdrant or Milvus collection #1631

Closed

edwinyyyu changed the title ~~[vector store 9/12] Make a vector store filter only on the properties it declares (speedkick)~~ [vector store 10/13] Make a vector store filter only on the properties it declares (speedkick) Sep 15, 2026

edwinyyyu mentioned this pull request Sep 15, 2026

Add session, source and expansion to EventMemory (speedkick) #1597

Closed

edwinyyyu force-pushed the feat/vector-store-declared-properties-speedkick branch 7 times, most recently from 447de33 to df75c26 Compare September 15, 2026 20:19

edwinyyyu and others added 12 commits October 27, 2026 12:13

edwinyyyu mentioned this pull request Oct 28, 2026

[selective filter 1] Route filtered queries by selectivity, and search allowlists exhaustively #1871

Draft

leomem reviewed Oct 29, 2026

View reviewed changes

marvinyu-memverge approved these changes Oct 29, 2026

View reviewed changes

xiongzubiao reviewed Oct 30, 2026

View reviewed changes

edwinyyyu and others added 2 commits October 30, 2026 14:07

xiongzubiao approved these changes Oct 30, 2026

View reviewed changes

		raise TypeError(f"Unsupported filter expression type: {type(expr)}")


		def _compare(column: ColumnElement, op: str, bound: object) -> ColumnElement[bool]:

Conversation

edwinyyyu commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the change

Stack

Verification

Uh oh!

edwinyyyu commented Oct 28, 2026

Uh oh!

leomem left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edwinyyyu commented Oct 29, 2026

Uh oh!

marvinyu-memverge left a comment

Choose a reason for hiding this comment

Uh oh!

xiongzubiao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xiongzubiao left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

edwinyyyu commented Sep 14, 2026 •

edited

Loading