[user properties 1/2] Keep undeclared properties out of the vector store - #1702

…ore (#1702)

* Keep undeclared properties out of the vector store

The event backend wrote every property of an event into its vector record
and mapped the caller's whole filter onto the vector store, so a user key
that a deployment never declared was both stored and filtered there: on
Qdrant and Milvus as unindexed payload a filtered query scans for. The
segment store already holds every property and already receives the whole
filter for the context windows, so the vector side only duplicated work the
segment store does anyway.

The vector record now carries the keys the collection declares:
EventMemory's reserved timestamp, the `_`-prefixed system properties an
adapter stamps on the event, and the keys a project's `properties_schema`
declares. The vector store is queried with the conjuncts of the filter
that name only such fields; a conjunct is dropped whole when any field
under it is undeclared, so dropping only ever widens the vector search,
and the segment store narrows it back on the windows. An undeclared key
never reaches the vector store, so a tenant's undeclared properties cannot
shape what it stores or scans.

`filter_fields` joins the filter parser: every field name a tree addresses.

Rebased onto #1631, where the vector record no longer carries the segment
uuid (the segment store maps a derivative to its segment).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

* Send the vector store the part of a filter it can evaluate, at any depth

The vector store got the conjuncts of a filter's top-level AND that name
only declared fields, so an OR or a NOT with an undeclared field anywhere
under it was dropped whole, and the vector search ran wider than it had
to.

It now gets the part of the whole tree on declared fields, the way a
database planner uses an index for part of a condition: an AND keeps
every operand that has such a part, an OR only when every operand has
one, and a NOT applies the opposite rule to its operand. So
`(a = 1 AND u = 1) OR (a = 2 AND u = 2)` gives `a = 1 OR a = 2`, and
`NOT (u = 1 OR a = 1)` gives `NOT a = 1`. Every record that matches the
filter still matches the part, and the segment store, which gets the
whole filter, narrows the result back as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Extract a filter's part on allowed fields in the filter module

Taking the part of a filter that names only some fields depends on
nothing but the filter and the fields, so it moves next to
`filter_fields` and `map_filter_fields` as `extract_filter`, the name
PostgreSQL, DataFusion, and Spark give the same operation.
`EventMemory._vector_store_filter` had one caller, so `query` now maps
the filter's fields and extracts its part on the declared ones itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test filter extraction against an evaluator, and queries with and without it

Rewriting a filter is checked the ways database engines check a
planner's rewrites, against the in-memory stores' filter evaluator:

- A table of filters and their expected parts covers every node kind
  under both parities of negation, with each operand kept or dropped,
  and alone reaches every line and branch of `extract_filter`.
- Random filters up to four levels deep, over allowed and other fields
  and every kind of leaf: every record a filter matches, its part
  matches, and the part names only allowed fields; a filter on allowed
  fields alone is its own part; and when each other field appears
  once, a record matches the part exactly when some values of the other
  fields would make it match the filter.
- Random filters through EventMemory, with the vector store getting the
  part and with it getting no filter: when the search reaches every
  record, both answer the filter exactly, and when it reaches fewer, the
  part only ever adds hits.

The random filters and records come from a shared test module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* State that a condition on an undeclared field applies to the searched seeds

A vector search returns at most `vector_search_limit` seeds, and a
condition on a field the vector store does not declare is applied to
those seeds afterward, so a selective one can leave fewer seeds than the
limit, or none, while matching segments exist further down the ranking.
That bounds a query's cost by the limit. `query`'s docstring now says so,
and a test pins it with an embedder that ranks the matching event below
the limit: the condition on an undeclared field finds nothing, the same
condition finds the event when the search reaches every seed, and a
condition on a declared field finds it within the limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Say that a seed the filter rejects has no context, and make the fake agree

EventMemory leaves to the segment store every condition the vector
store cannot evaluate, so it relies on `get_segment_contexts` leaving
out a seed that fails the filter. The SQL store does, but the contract
only said the filter applies to the segments, and the in-memory fake
kept a rejected seed whenever a neighbor in its window matched. The
contract now says a seed that does not match has no entry, the fake
checks the seed, and a test pins it through EventMemory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Say what the vector search filters on, in EventMemory's docstrings

`query` said only conditions on undeclared fields wait for the seeds,
but a declared condition in an OR with an undeclared one, or in a
negated AND with one, waits too: the vector search applies only the
part of the filter that names declared fields alone. And the expected
collection schema said declaring a property makes it efficiently
filterable, when a property left undeclared is not on the vector record
at all, so the search cannot filter on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Use the declared fields EventMemory already holds

Building a vector record and extracting the vector store's part of a
filter both read the collection's declared schema again, once per
derivative for the record. `_schema_fields` already holds those keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test a declared user key on the record and in the search

The fixture's collection declares only system keys, so no test had a
user key reaching the vector store; a wrong mapping of `m.<key>` would
have dropped its condition from the search with every test passing. A
test now declares `color` and checks that the record carries it and
that `m.color` reaches the search as `color`. The three tests that
spied on the vector store's queries share one helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>