Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion deployments/helm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ resources:
databases:
db_postgres: { provider: postgres, config: { host, port, user, password: $POSTGRES_PASSWORD, ... } }
db_neo4j: { provider: neo4j, config: { uri, username: $NEO4J_USER, password: $NEO4J_PASSWORD, pool, ... } }
event_vector_store: { provider: qdrant, config: { host, port, grpc_port, prefer_grpc, https, collection_registry, api_key: $QDRANT_API_KEY (only when a key is configured) } }
event_vector_store: { provider: qdrant, config: { host, port, grpc_port, prefer_grpc, https, partition_registry, api_key: $QDRANT_API_KEY (only when a key is configured) } }
embedders:
default_embedder: { provider, config: { model, api_key: $OPENAI_API_KEY, base_url, dimensions } }
language_models:
Expand Down
2 changes: 1 addition & 1 deletion deployments/helm/templates/memmachine-configmaps.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ data:
grpc_port: {{ .Values.qdrant.grpcPort }}
prefer_grpc: {{ .Values.qdrant.prefer_grpc }}
https: {{ .Values.qdrant.https }}
collection_registry: db_postgres
partition_registry: db_postgres
{{- if include "memmachine.qdrantApiKeySecret" . }}
api_key: $QDRANT_API_KEY
{{- end }}
Expand Down
60 changes: 27 additions & 33 deletions design/milvus_vector_store.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,21 @@
# Milvus vector store

How the Milvus store meets the shared contracts: [collection
registry](vector_store_collection_registry.md),
How the Milvus store meets the shared contracts: [partition
registry](vector_store_partition_registry.md),
[purge](vector_store_purge.md), [consistency](vector_store_consistency.md),
[isolation](vector_store_isolation.md).

## Layout

- **One native collection per namespace and configuration**, named
`memmachine_{namespace}__{sha256(config)}`, the digest over the
configuration's JSON. It holds every logical collection of that namespace
and configuration, one incarnation each. Admitting a collection creates
nothing in Milvus unless its configuration is new.
- **One native collection per store**, named `sys_` followed by the vector
store name. It holds every partition of the store, one incarnation each.
Creating a partition creates nothing in Milvus.
- **Fields:** `id` (VARCHAR primary key, `"{incarnation}:{record_uuid}"`),
`record_uuid` (VARCHAR), `partition_key` (VARCHAR, the incarnation,
`is_partition_key`), `vector` (FLOAT_VECTOR), `properties` (JSON), and one
nullable typed field per declared property, `_p_<name>`. Dynamic fields
are off, so each property is stored once.
`is_partition_key`), `vector` (FLOAT_VECTOR), and one nullable typed field
per declared property, `_p_<name>`. Dynamic fields are off, and a record or
a filter naming an undeclared key is refused, so every property is a typed
field.
- **Tenancy:** partition-key multi-tenancy with `partitionkey.isolation`: each
segment builds its vector index per group of tenants, so a search filtered
on one incarnation searches only its group. Milvus documents isolation for
Expand All @@ -36,14 +35,13 @@ registry](vector_store_collection_registry.md),
a TIMESTAMPTZ field holding its instant, written in UTC: a filter compares
only instants, a search answers with record UUIDs and scores, and Milvus
refuses an offset with a seconds component. Milvus caps a collection at
`proxy.maxFieldNum` fields (64 on 2.6, 256 on 3.0), so a configuration
declares at most 59 properties on 2.6.
Undeclared properties go in the JSON field, still filterable by path.
Negation is the complement, as on Qdrant: a negated condition holds where
the property has no value, which Milvus's SQL-style null evaluation does not
give on its own.
- **Scores** are the server's (cosine similarity, inner product, and the
square root of Milvus's squared Euclidean distance).
`proxy.maxFieldNum` fields (64 on 2.6, 256 on 3.0), so a store declares at
most 60 properties on 2.6. A condition whose value is of another type than
its key declares matches nothing. Negation is the complement, as on Qdrant:
a negated condition holds where the property has no value, which Milvus's
SQL-style null evaluation does not give on its own.
- **Scores** are the server's: a COSINE index answers the cosine similarity
as a hit's distance.
- **Server-configured limits stay the server's.** A search `limit` reaches the
server, which refuses one above `quotaAndLimits.limits.topK`. A declared
string's VARCHAR length is `max_varchar_length` (65,535 unless configured,
Expand All @@ -52,13 +50,11 @@ registry](vector_store_collection_registry.md),
`quotaAndLimits.limits.maxQueryResultWindow`); both are settings, not
constants. A native collection keeps the VARCHAR length it was created
with, so a changed `max_varchar_length` applies to native collections
created afterward. A record's undeclared properties share one JSON field,
which the server refuses above `common.JSONMaxLength` bytes (65,536 unless
configured). A declared property's field name is its key, at most 32
created afterward. A declared property's field name is its key, at most 32
bytes, under `_p_`, within `proxy.maxNameLength` (255 unless configured).
- **Creation converges:** the collection and its indexes (named by their
fields) are created only when missing, and the collection is loaded, a
no-op when it is loaded already.
- **Startup converges:** the collection and its indexes (named by their
fields) are created at startup only when missing, and the collection is
loaded, a no-op when it is loaded already.
- **A delete raises unless Milvus accepted every key sent.** Milvus counts the
primary keys a delete accepts, present or not (milvus-io/milvus#51566), and
pymilvus's async client returns that count for a delete Milvus rejected, so
Expand Down Expand Up @@ -144,8 +140,8 @@ throughput unchanged.

The incarnation is in the primary key because Milvus's upsert deletes by
primary key in every partition (`AllPartitionsID`) before inserting: with the
bare record UUID as the key, one collection's upsert would delete another
collection's record of the same UUID. With the composite key, two collections'
bare record UUID as the key, one partition's upsert would delete another
partition's record of the same UUID. With the composite key, two partitions'
records of one UUID are two entities, and a reused UUID's record is simply
stored, which meets the [isolation](vector_store_isolation.md) guarantee. The
record UUID is also kept in its own field, which reads return. Measured
Expand All @@ -168,10 +164,7 @@ variation.

Bounded batches. A round lists up to `purge_batch_size` of the incarnation's
primary keys, by a query on the incarnation field at the store's read level,
and deletes them by key. It first creates the native collection's missing
indexes and loads it, as a creation does: a creation that failed partway can
leave it unindexed and unloaded, and only a loaded collection answers the
listing. Measured (Milvus 3.0.2; 2.11M points in this layout,
and deletes them by key. Measured (Milvus 3.0.2; 2.11M points in this layout,
dead incarnations of 10k to 1M among live tenants under search, upsert, and
scroll traffic; 2 CPUs / 4 GB): rounds stayed flat at about 100 ms to the end
of a 1M purge, with at most a 0.3 s stall for other tenants; one filter-delete
Expand Down Expand Up @@ -233,7 +226,7 @@ behind, the read waits rather than reads staler, and fails over to another
replica if the node's tsafe stalls for 3 s (`queryNode.waitTsafeStallTimeout`,
from 2.6.15).

**The store reads at Bounded**: it creates each native collection at Bounded
**The store reads at Bounded**: it creates its native collection at Bounded
by name, rather than at whatever level pymilvus defaults to, and names no
level on a read, so every read runs at the collection's level. The level is not configurable:
the stated delay of at most `common.gracefulTime` and the tombstone retention
Expand Down Expand Up @@ -296,7 +289,8 @@ both.

## Consequences

- An existing native Milvus collection created with the earlier schema is not
usable by this store and has to be dropped.
- A native Milvus collection from an earlier release has another name than
this store's, so the store neither reads nor purges it; dropping it frees
its space.
- A deployment that raises `common.gracefulTime` lengthens the store's read
delay, which the tombstone retention must still exceed by far.
54 changes: 30 additions & 24 deletions design/qdrant_vector_store.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,38 @@
# Qdrant vector store

How the Qdrant store meets the shared contracts: [collection
registry](vector_store_collection_registry.md),
How the Qdrant store meets the shared contracts: [partition
registry](vector_store_partition_registry.md),
[purge](vector_store_purge.md), [consistency](vector_store_consistency.md),
[isolation](vector_store_isolation.md).

## Layout

- **One native collection per namespace and configuration**, named
`{namespace}__{sha256(config)}`, the digest over the configuration's JSON.
It holds every logical collection of that namespace and configuration, one
incarnation each. Admitting a collection creates nothing in Qdrant unless
its configuration is new.
- **Vectors:** one unnamed dense vector, the configuration's dimensions and
metric.
- **Graphs per tenant, not per collection:** HNSW `m=0`, `payload_m=16`. No
- **One native collection per store**, named by the vector store name. It
holds every partition of the store, one incarnation each. Creating a
partition creates nothing in Qdrant.
- **Vectors:** one unnamed dense vector of the store's dimensions, compared by
cosine distance.
- **Graphs per tenant, not per store:** HNSW `m=0`, `payload_m=16`. No
collection-wide graph is built; each value of an indexed payload field gets
its own graph once it has enough points in a segment, so a search filtered
on one tenant walks that tenant's graph.
- **The incarnation:** payload field `sys-incarnation`, a keyword index with
`is_tenant=true`, which also has Qdrant keep a tenant's points together.
A handle's searches filter on it, as do the purge's scrolls and deletes. A
handle's deletes name its points' ids instead (see [Point ids](#point-ids)).
- **Declared properties:** a payload index per property of the collection's
schema, typed by the property's type.
- **Creation converges:** the collection and each payload index are created
under separate already-exists guards, so a creation that failed between them
is completed by the next.
- **Declared properties:** a payload index per property of the store's
declared schema, typed by the property's type. A record or a filter naming
any other key is refused.
- **Strict mode:** the collection is created with filtering on unindexed
fields off, for reads and updates alike, so the server refuses a filter on
a field it has not indexed instead of scanning for it. Every key a filter
may name is indexed: the declared keys and the incarnation. A condition
whose value is of another type than its key declares matches nothing; the
store answers it with a filter no point satisfies, since the server would
refuse it for the field's index.
- **Startup converges:** the collection and each payload index are created at
startup under separate already-exists guards, so a startup that failed
between them is completed by the next.
- **Client:** `AsyncQdrantClient`, with `request_timeout_seconds` as its
timeout.
- **Oversized upserts are halved.** Qdrant's REST API refuses a request over
Expand All @@ -48,14 +54,14 @@ in the payload, `sys-record_uuid`. A search returns that one payload field to
answer each match's record UUID; a delete names the derived ids directly;
someone inspecting a collection finds a record by filtering on the field.

The id space is the native collection's, shared by its logical collections,
The id space is the native collection's, shared by the store's partitions,
and Qdrant's upsert replaces a whole point, vectors and payload alike
(`lib/shard/src/update/points/upsert.rs` at v1.19.1). With the record UUID as
the point id, an upsert of a UUID another collection's point held replaced
the point id, an upsert of a UUID another partition's point held replaced
that point and moved it to the writer's incarnation. Derived ids differ
between collections whatever record UUIDs they carry, so the store keeps the
between partitions whatever record UUIDs they carry, so the store keeps the
[isolation](vector_store_isolation.md) guarantee without a rule on callers,
and a collection created again under a name writes ids its dead predecessor's
and a partition created again under a key writes ids its dead predecessor's
points, awaiting purge, cannot collide with.

Measured on Qdrant 1.19.1 (4 CPUs / 4 GB, gRPC; 300 tenants x 1,000 points of
Expand Down Expand Up @@ -85,8 +91,8 @@ of every property's payload; it is not measured.

**Why one-way ids.** The record UUID has to come back from the payload because
a UUIDv5 cannot be inverted, and that is also what makes it safe. An attacker
who writes through the API into one collection, and wants to hide a record of
another collection sharing the native collection, needs a record UUID whose
who writes through the API into one partition, and wants to hide a record of
another partition sharing the native collection, needs a record UUID whose
point id equals the target's. With UUIDv5 that is a second preimage of SHA-1
on 122 bits, about 2^122 work, even for someone who knows both incarnations
and the target's record UUID; known SHA-1 attacks need control of both inputs.
Expand All @@ -100,7 +106,7 @@ the mapping lets anyone holding the incarnations aim it: with the target's
incarnation, their own, and the target's record UUID, an attacker computes the
colliding UUID directly (`record ^ incarnation_A ^ incarnation_B`), and a
write of it through an ingestion path that passes a caller's UUID through
replaces the target's point, which then disappears from its collection.
replaces the target's point, which then disappears from its partition.
Incarnations are not guarded as secrets: the registry logs them in its
warnings and dead-letter errors, and they are in its tables and any backup, as
record UUIDs are in any Qdrant snapshot. So a read-level leak plus an ordinary
Expand Down Expand Up @@ -195,8 +201,8 @@ apply reaches the caller.
replica by default and can miss a write another replica has; and with the
default `weak` write ordering "write operations can be freely reordered",
while `medium` and `strong` serialize them through a leader. The store sets
none of these, and creates collections with the server's default replication
factor. It states no replicated read delay, which is unbounded.
none of these, and creates its native collection with the server's default
replication factor. It states no replicated read delay, which is unbounded.

Measured on a three-node Qdrant 1.19.1 cluster in Docker (1.5 CPUs / 1.5 GB
per node; one shard replicated on all three nodes; write consistency factor 1;
Expand Down
Loading