[declared schema 2/2] Close the filter union, type each filter field when it is parsed, check records at upsert, and make negation the complement on every backend but NebulaGraph - #1616

Purpose of the change

The filter language becomes a closed union: Equals, Ordering, In, IsNull, n-ary And/Or, Not (common/filter/filter_expression.py); every compiler is an exhaustive match. != and <> parse as Not(Equals), NOT IN as Not(In), IS NOT NULL as Not(IsNull), so a negation means one thing. An ordering comparison takes a number or a date: field > 'string' and field > true, which parsed before, are now rejected by the parser.

A leaf on a field holding no value is false, IsNull on it is true, and Not is the complement over the record set on every vector store and the segment store. The SQL compilers render it as NOT COALESCE(x, FALSE): sql_filter_util.py compiled ~inner, three-valued, and a negated filter silently dropped rows holding no value on every SQL-backed store. Milvus already pushes Not to the leaves by De Morgan, since [vector store scale-out 5/6] (#1736), as its own not excludes entities lacking the field, and Qdrant's must_not already admits them. The Neo4j graph store and the Neo4j semantic storage render Not as NOT coalesce((x), false), the complement, and the NebulaGraph graph store keeps its NOT over a comparison three-valued (#1918), since its tests need a NebulaGraph Enterprise 5.0 server. A filter is bound to the store's declared schema before the store compiles it (bind_filter), and a record before the store writes it (bind_record): an int for a float key becomes the float it equals, and a float for an int key, a bool for a number, a non-finite value, or an int outside 64 signed bits raises PropertyTypeMismatchError, whose message says which, so no compiler meets a value of another type than its key declares. EventMemory binds an event's declared properties in _validate_events, before anything is written, so the segment store and the vector store hold the same value. An In over no values admits nothing: every compiler renders it as false, so EventMemory's empty id lists keep meaning "nothing", and its complement admits everything.

A filter's fields are resolved when it is parsed, so nothing below the parser reads how a client spelled a field. Each leaf names a SystemField(name) or a Property(key, type) (filter_expression.py). The parser turns m.<key> and metadata.<key> into a property and any other name into a system field, and normalize_filter_field, mangle_user_metadata_key, demangle_user_metadata_key, is_user_metadata_key and the USER_METADATA_* prefixes go. Each memory maps its own system fields: the event backend's to its stored keys, and the episode store's, the semantic storages' and short-term memory's to their columns and attributes. EventMemory takes only SystemField("timestamp") and properties, the segment store maps the timestamp to its column and a property to its JSON key, and the vector stores and graph stores refuse a system field.

A literal is typed by its field when the filter is parsed. parse_filter takes the target memory's system field types (EVENT_BACKEND_FILTER_FIELD_TYPES, EPISODE_STORAGE_FILTER_FIELD_TYPES, SEMANTIC_STORAGE_FILTER_FIELD_TYPES), and each store's resolver takes its field names from the same map. An int for a float field parses as the float it equals. A string for a datetime field (a datetime is written date('...')), a float for an int field, a bool for a number, or a number for a string field is a parse error, which the API answers with 422. A system field's name matches exactly: a mixed-case name such as Producer_ID, which the episode store matched before, is an unknown field. A user property's type is dynamic, and its numbers compare by value, an int and a float as one numeric type, exactly beyond 2⁵³ on SQLite and PostgreSQL.

An In takes no float and no datetime on either side: a membership test of floats, datetimes or bools, or on a float or datetime field, is refused, where an int test on a float key was a disjunction of float equalities. An ordering comparison (<, <=, >, >=) takes an int, a float or a datetime. An equality on a float or a datetime is exact on every store: Qdrant compiles it as a closed Range or DatetimeRange, since its MatchValue takes no float.

A vector store refuses an upsert batch that names a record UUID twice, with ValueError before anything is sent, where the stores disagreed: the SQLite store and Qdrant kept the last version, the sqlite-vec store failed partway with sqlite3's UNIQUE constraint error, and Milvus refused the batch with its own error. A store keeps a datetime property's instant, taking a naive datetime as UTC, and may drop its UTC offset; a filter compares datetimes by their instant (the VectorStorePartition docstring). Qdrant now writes a datetime in UTC: it keeps a written datetime's offset only to the minute, so one at an offset with a seconds component was stored up to 59 seconds off its instant, and an equality on that instant missed it. These come from [vector store write validation] (#1788), folded in here; its non-finite check is not, since bind_record already refuses a non-finite value at every store's upsert.

Last in the stack, because the store work is the priority and this change touches many files mechanically; the slices under it are written against the nodes main has today. The complement law is checked over thirteen filters in declared_schema_contract.py, across the column, JSON and properties encodings of the SQL stores.

Adaptation for main: the routing of [user properties 1/2] is rewritten on the closed union (And.operands; Equals for Comparison with =), and its extract_filter moves to filter_expression.py, a match over the n-ary nodes exported from common.filter, with its tests and random filters on the closed union; the speedkick-era commit 8ca5f06 was written on #1597's tree, and its rewrites of #1597's _system_predicates, _lift_typed_filters and their tests have nothing to apply to on main. The event memory test spells the field context.producer, main's name for it. The tests the rebuilt base adds use the closed union too, and the segment store's random context-read test models the new semantics: a condition on a property with no value is false and a negation is the complement, where the model followed SQL's three-valued logic. Through 3176c1a92, the same 60 files as the origin; the commits after it are new here, with no origin counterpart.

Position: directly on feat/horizontal-scaling, which has [declared schema 1/2] (#1628) merged, in parallel with the SQLite fixes; the fixes' tests, which the origin also converts to the closed union, are not in this branch.

Fixes #1780, #1781, #1783, and #1784.

Stack

24 open PRs: three independent PRs, and the vector store tree of short parallel branches. Every open PR in the tree has feat/horizontal-scaling as its GitHub base, and the independent PRs have main. The branches are in a fork, and a pull request can target only this repository's branches, so the on column gives the order the PRs build on each other. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1675 is closed, superseded by #1627. #1841 is closed: its request packing is dropped, and its cancellation contract is in #1840. #1828 and #1839 sit side by side on the branch; #1839 and #1840 are the event memory recovery stack, and #1684 heads the event memory stack, which its own PRs index, listed because it builds on #1840, its branch also carrying #1828's commits until one of the two merges; #1871 keeps its own Stack section. Until the PRs under it merge, their changes show in a stacked PR's diff.

This PR is its 16 commits, 2fd7f67f9, 197757ec7, b0f862135, 9a3efe165, 3176c1a92, 38bfc9e58, 79cf89dc4, e39f219a8, 2cc7d0fa9, 602386499, 3a0e53fd9, 04c252b9d, 4c0e9cbec, c093a94df, f09a4f7be, afac73b12, directly on feat/horizontal-scaling.

Verification

At this PR's head 2486d2d73, on 2026-11-03, its tree is afac73b12's with three commits from #1788 on top: an upsert batch naming a record UUID twice refused, Qdrant's datetimes written in UTC, and the datetime property contract stated. #1788's non-finite check is dropped: every store's upsert (the registry-backed base for Qdrant and Milvus, the SQLite store and the sqlite-vec store) binds each record with bind_record first, whose _validate_value refuses a non-finite float, and declared_schema_contract.py checks that on every store. The new contract tests run on every store: a batch repeating a UUID raises and stores nothing, and fails without the check on all five variants (the SQLite store and Qdrant over REST and gRPC raise nothing, the sqlite-vec store raises sqlite3's OperationalError, Milvus 2.6.24 its MilvusException); a datetime written at +05:30 compares by its instant at other offsets; a datetime written at -00:44:30 equals its instant, given at that offset or in UTC, and fails on Qdrant over REST and gRPC without the write in UTC. Qdrant's read-back test finds a datetime written at +05:30 held in UTC. Checks at the head: ruff check and ruff format --check clean; ty check clean as CI runs it; uv lock --check clean. The unit tests pass, 2596 tests (8 skipped), and the vector store integration tests pass against containers the tests start (Qdrant 1.19.1, Milvus 2.6.24): Qdrant 356, Milvus 149, and the partition registry 56.

Before #1788's commits: At this PR's head afac73b12, on 2026-11-03, its tree is 3176c1a92's with eleven commits on top: field references resolved at parse, the tests' nodes built from them, the refused-value wording, the In and equality rules, literals typed at parse, the tests' field types, numbers compared by value, events bound before any write, Neo4j's complement, each store's field names taken from its field types, and exact field names. Checks at the head: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass at the head, 2590 tests (8 skipped). At c093a94df, the ninth of them, the unit tests pass, 2552 tests (8 skipped), and the integration tests pass against containers the tests start: Qdrant 348; the SQL filter, segment store and episode store tests on PostgreSQL 216; Neo4j 27 (graph store and declarative memory); semantic storage 1044 (pgvector, Neo4j and in-memory, 3 skipped by their Neo4j-specific skips, the two LLM-backed modules not run); and Milvus 146. The two commits after it were checked on what they touch. At f09a4f7be: the episode store, semantic, main and filter unit tests (790 passed, 2 skipped), and the episode store and semantic integration tests (1145 passed, 15 skipped by Neo4j-specific skips). At afac73b12: the episode store, main and server unit tests (453), and the episode store's integration tests on PostgreSQL (91). The NebulaGraph tests did not run, since they need an external NebulaGraph Enterprise 5.0 server.

Before the filter-reference commits: At this PR's head 3176c1a92, on 2026-11-02, rebased onto feat/horizontal-scaling at 0fa52b38b, where [sqlite write transactions 1] (#1844) is merged since 690116e37. Its own lines follow the base: the SQLite stores import FilterExpr from the filter package beside #1844's validate_engine_connections. Its commits are re-signed, with their messages and authors unchanged. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 2479 tests (8 skipped). The integration tests did not run here.

Before the rebase onto #1844's merge: At this PR's head b566a2c15, on 2026-10-30, restacked directly on feat/horizontal-scaling at 690116e37, where [declared schema 1/2] (#1628) is merged as 5c9d3ae99, with #1831 before it and #1858 after it; [sqlite write transactions 1] (#1844) is no longer beneath it, since this PR uses none of #1844's code. Its own lines follow the base: the declared-schema contract keeps both #1628's re-upsert test and this PR's own tests where they meet, and keeps _presence_records, which #1628's last commit removed and this PR's tests call; with_has_set_ids, which #1831 made public, returns FilterExpr | None as this PR has it; and #1831's semantic storage test builds its membership filter with a tuple, as this PR's In holds. Its commits are re-signed, with their messages and authors unchanged. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 2470 tests (8 skipped). The integration tests did not run here.

Before the restack onto feat/horizontal-scaling after #1628's merge: At this PR's head 526b3d52e, on 2026-10-28, restacked on [sqlite write transactions 1] (#1844) at 4f5d4430c, on #1628's df716e47b: its tree is the previous head cb0da1545's with #1628's two changes since 274264523 beneath it (is_valid_identifier's docstring, and the SQLite store's records table built without extend_existing), and its own lines are unchanged. Its commits are re-signed, with their messages and authors unchanged. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before the restack onto #1628's df716e47b: At this PR's head cb0da1545, on 2026-10-27, rebuilt on [sqlite write transactions 1] (#1844) at 0bcbf8b3a, with #1628's second round of review commits (the identifier rule in data_types.py, which imports nothing from the package, and the interface's Raises sections sorted by name) beneath it. Its own lines follow them: utils.py imports the errors, Record, and the int range from data_types.py by name, so bind_record annotates with Record itself; MIN_INT_VALUE and MAX_INT_VALUE sit in data_types.py with no import from the package; and the Raises entries this PR adds to upsert and query take their places by name. Its commits are re-signed, with their messages and authors unchanged. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 2446 tests (8 skipped). The integration tests did not run here.

Before #1628's second round of review changes: At this PR's head 85d44a9a2, on 2026-10-27, rebuilt on [sqlite write transactions 1] (#1844) at 69e292d8a, with #1628's four commits from Edwin's review (no supported_filter_nodes, its checks in utils.py, sqlite_properties.py, and the strict-mode comment) beneath it. Its own lines follow them: bind_filter and bind_record, with their helpers, live in utils.py beside the other validators and raise their errors through the data_types module, which imports utils.py; bind_filter takes no supported nodes, since every store evaluates the whole filter language; and filter_expression.py and the filter package's exports drop filter_nodes. The first two commits' messages name sqlite_properties.py and no longer mention the node check. Its commits are re-signed, with their authors unchanged. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 2446 tests (8 skipped). The integration tests did not run here.

Before #1628's review changes: At this PR's head 3a803816e, on 2026-10-27, moved onto [sqlite write transactions 1] (#1844) at 497a89ef1, which now sits on #1628: its tree is the previous head 08c6dc525's, so the checks and tests at that head stand. Its commits are re-signed, with their messages and authors unchanged.

Before #1844 moved above this PR: At this PR's head 08c6dc525, on 2026-10-27, replayed onto the replayed #1628's 135184cd6, after [vector store scale-out 6] (#1627) merged into feat/horizontal-scaling as 805ea2177, a squash whose tree is 8c2351dbe's: its tree is the previous head ac7c6eda7's with #1628's test-only commit beneath it, which gives the oversized Milvus upsert test's own store a longer request timeout. Its commits are re-signed, with their messages and authors unchanged. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The integration tests did not run here.

Before the replay onto the merged #1627: At this PR's head ac7c6eda7, on the rebuilt #1628, on 2026-10-26: beneath it are #1627's changes since e38e8003e (each store's schema passed as one VectorStoreSchema, store_schema; SQLite partitions named by an incarnation and created as the other stores create theirs; the shared SQL engine setup and write transactions in common/sql.py; and every session's event memory served from the deployment's configuration, in one store) and #1844's SQLite write transactions. This PR's own lines follow them: the stores bind filters and records against store_schema.indexed_properties; the two SQLite store tests #1627 added with a Comparison filter use Equals; and the Qdrant store drops the Mapping and PropertyType imports with the compiler's schema argument. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 2446 tests (8 skipped). The integration tests did not run here.

Before the rebuild onto #1627's 8c2351dbe: At this PR's head 8c76235b1, on 2026-10-20, its tree is the previous head 6d67208b1's with #1627's accuracy fixes, in text only: the partition registry and purge design documents, the database manager's comments and docstrings, the SQLite store's module docstring, the partition registry configuration's descriptions, and other docstrings that still described collections. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 1355 tests (8 skipped). No integration tests ran this time, since only comments, docstrings, descriptions, and documentation changed.

Before #1627's accuracy fixes: At this PR's head 6d67208b1, on 2026-10-19, its tree is the previous head 5ddf3a290's with #1627's second review's fixes: the Qdrant and Milvus tests that partitions share the store's collection now write through two partitions and read the store's one native collection; the store tests name their partition fixture partition; and docstrings, comments, and the configuration and upgrade documentation are corrected. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 1355 tests (8 skipped), and the vector store and segment store integration tests (the registry on PostgreSQL and SQLite, both SQLite stores, Qdrant, Milvus, and the SQLAlchemy segment store) pass, 642 tests.

Before #1627's second review's fixes: At this PR's head 5ddf3a290, on 2026-10-19, its tree is the previous head 5a9d2d9eb's with #1627's review fixes: the database manager answers a repeated (backend, vector store name) with the store it built, without comparing parameters, and keeps a backend's shared SQLite engine until it closes, even when a store's build fails; its test that one SQLite backend's stores share the engine can now fail; the partition registry's parameters check the vector store name with the shared validate_vector_store_name; and semantic memory's local name for the backend key is backend. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 1355 tests (8 skipped).

Before #1627's review fixes: At this PR's head 5a9d2d9eb, on 2026-10-19, moved onto [declared schema 1/2] (#1628): its tree is the previous head 18ef07b12's without [search results] (#1663) beneath it. This PR's own lines keep SimilarityMetric beside its filter imports. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 1355 tests (8 skipped), and the vector store and segment store integration tests (the registry on PostgreSQL and SQLite, both SQLite stores, Qdrant, Milvus, and the SQLAlchemy segment store) pass, 642 tests.

Before the move off [search results] (#1663): At this PR's head 18ef07b12, on 2026-10-19, its tree is the previous head 3ae24ee56's with design/vector_store_partition_registry.md listing the lookup as "open", the name it has from #1627 until #1625, where it said "get". Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. Outside that document the tree is the previous head's, so the test runs below stand.

Before the partition registry document's wording fix: At this PR's head 3ae24ee56, on 2026-10-19, its tree is the previous head 2af6ac444's with #1824 beneath it, which drops the purge queues' unread logical key, and #1627 dropping the same key from the partition registry's tombstones and removing the collection registry design document its partition registry document replaces. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 1333 tests (8 skipped).

Before the rebase onto #1824's merge: At this PR's head 2af6ac444, on 2026-10-16, its tree is the previous head 291a4a103's with the configuration's check of a provided tombstone_retention_seconds named _validate_tombstone_retention, where it was _require_retention_floor. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 1333 tests (8 skipped).

Before the retention check's rename: At this PR's head 291a4a103, on 2026-10-16, its tree is the previous head 85a4b756a's with every vector store check of a value that raises named validate_, as validate_partition_key already was: validate_declared_types, validate_query_vector, validate_score_threshold, validate_limit, and validate_dimensions from the base, and the PRs' own checks in their turn. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 1333 tests (8 skipped).

Before the raising checks took the validate_ prefix: At this PR's head 85a4b756a, on 2026-10-16, its tree is the previous head a7950f283's with #1627's changes: the VectorStore interface drops vector_store_name, vector_dimensions, similarity_metric, and indexed_properties, and the partition handle drops partition_key; a partition key may be any string of up to 255 bytes without NUL, and the SQLite stores name a partition's tables and index file by a SHA-256 digest of it; and the checks that return a bool are is_valid_identifier and is_valid_filter. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 1333 tests (8 skipped), and the vector store integration tests (the registry on PostgreSQL and SQLite, both SQLite stores, Qdrant, and Milvus) pass, 512 tests.

Before the store's properties left its interface: At this PR's head a7950f283, on 2026-10-15: its tree is the previous head 150acbc8e's without #1627's per-partition schema record (PartitionSchema, the registry's and the SQLite stores' schema column, and VectorStorePartitionSchemaMismatchError), and with require_partition_key named validate_partition_key; ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The unit tests pass, 1327 tests.

Before the per-partition schema record went: At this PR's head 150acbc8e, on 2026-10-15: its tree is the previous head 2b82dbde4's with the partition lookup named open_partition and the segment store's close_partition kept, since #1625 renames the one and removes the other; ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before the lookup's rename moved into #1625: At this PR's head 2b82dbde4, on 2026-10-15: its tree is the previous head 580538da1's without the segment store's close_partition; ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before #1627's first commit took the removal of close_partition: At this PR's head 580538da1, on 2026-10-15, restacked on #1627's rewritten history: its tree is the previous head f61920cf6's; ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before #1627's history was rewritten: At this PR's head f61920cf6, on 2026-10-15, rebased onto feat/horizontal-scaling at cf7ae1579, which carries main through #1774: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean; the MemMachine mock and session deletion tests pass, 48 tests (3 skipped).

Before the rebase onto cf7ae1579: At this PR's head 04943f87d, on 2026-10-15, rebased onto feat/horizontal-scaling with #1702 and #1670 merged (101182c18): ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before the rebase onto #1670's merge: At this PR's head ad48f4407, on 2026-10-13, rebuilt on #1702's 6631b5c77: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean; the episodic memory and filter tests pass, 503 tests (1 skipped). Each of seven deliberately broken versions of extract_filter fails these tests, and the table of expected parts alone covers every line and branch but the closed match's unreachable fall-through.

Before the rebuild on #1702's filter routing: At this PR's head 13d104245, on 2026-10-09, on feat/horizontal-scaling with #1813 merged: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean; the Qdrant store's, the registry-backed store's, and the database manager's tests, unit and integration, pass, 386 tests, against Qdrant 1.19.1 defaulting new collections to strict mode and Milvus 2.6.24 in containers.

Before the rebase onto #1813's merge: At this PR's head df21dcdda, on 2026-10-09: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. Its tree is the previous head e9b2fc25c's: the server suite without integration tests last passed at d491433ce, on 2026-10-08, 2235 tests, and the integration tests of the vector stores, the resource manager, episodic memory, and semantic storage at d0ba0b451, on 2026-10-08, 840 tests, against PostgreSQL 16, Neo4j, Qdrant 1.19.1, and Milvus 2.6.24 in containers. The SQLite stores' tests last passed at 372d94164, on 2026-10-09, 270 tests, after the SQLite offset column was dropped.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn