Skip to content

Pack vector store upserts by serialized size, and state what cancelling one guarantees - #1841

Closed
edwinyyyu wants to merge 3 commits into
MemMachine:feat/horizontal-scalingfrom
edwinyyyu:feat/vector-store-size-based-upserts
Closed

Pack vector store upserts by serialized size, and state what cancelling one guarantees#1841
edwinyyyu wants to merge 3 commits into
MemMachine:feat/horizontal-scalingfrom
edwinyyyu:feat/vector-store-size-based-upserts

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Oct 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Purpose of the change

The Qdrant and Milvus partitions send an upsert as one request. When the server refuses it as too large (Qdrant 400 or 413, Milvus RESOURCE_EXHAUSTED), they halve the batch and resend the halves, so an upsert's time depends on how many halvings it takes, and a refused batch's bytes go out again at each one. Event memory bounds when its vector writes are sent by cancelling an upsert at a deadline, which needs a written guarantee of what cancelling does.

  • Upserts are packed by serialized size. Each record's size is computed as it goes on the wire, and the records are packed, in order, into the fewest requests of at most a new max_request_bytes setting, each sent once (pack_requests in vector_store/utils.py):
    • Qdrant over REST: the point's JSON from qdrant-client's own jsonable_encoder. Over gRPC: its UpsertPoints field after qdrant-client's RestToGrpc conversion. Both are exact; unit tests compare them byte for byte with what a real client sends.
    • Milvus: each value as an element of its field's column, which is exact, plus an upper bound for each field's and the request's own framing.
  • Defaults are the servers' own limits: 32 MiB for Qdrant (service.max_request_size_mb; its gRPC sets no limit) and 64 MiB for Milvus (proxy.grpc.serverMaxRecvSize). The design docs cite the sources at Qdrant v1.19.1 and Milvus v2.6.24.
  • A size refusal is a configuration error. A refused request was within the setting, so the setting exceeds the server's limit: the store raises QdrantConfigurationError or MilvusConfigurationError, chaining the server's error. Qdrant answers any invalid request with 400, which only the message tells apart, so its error says so. A record too large for a request alone raises ValueError before anything is sent.
  • The cancellation guarantee. VectorStorePartition.upsert now states that cancelling the call stops it from sending further requests, and that a request already sent may still be applied. Every implementation already behaves that way; each store gets a test that cancels an upsert mid-call and checks that nothing more goes out.

Behavior that changes:

  • A deployment whose Qdrant sits behind a proxy with a smaller body limit than 32 MiB must set max_request_bytes to that limit. Its refusals now raise instead of being halved.
  • Over REST, each point is serialized twice, once to measure it and once to send it: about 0.1 ms per 1536-dimension point (0.03 ms over gRPC), measured locally.

Stack

25 open PRs: three independent PRs, and the vector store tree of short parallel branches. Every open PR in the tree has feat/horizontal-scaling as its GitHub base, and the independent PRs have main. The branches are in a fork, and a pull request can target only this repository's branches, so the on column gives the order the PRs build on each other. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

# PR change on
— #1624 Make no memory request create a project main
— #1786 Refuse a filter value of the wrong type for a datetime column, and answer an invalid list filter with 422 (port of #1620) main
— #1792 Refuse property values that some store refuses or alters where an episode enters main

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1675 is closed, superseded by #1627. #1839 and #1840 are the event memory recovery stack, and #1684 heads the event memory stack, which its own PRs index; #1684 is listed because it builds on #1840 and #1828 builds on it; #1871 keeps its own Stack section. Until the PRs under it merge, their changes show in a stacked PR's diff.

# PR change on
[vector store scale-out 1/6] #1671 (merged) Remove custom sharding from the Qdrant store (port of #1654) main
[vector store scale-out 2/6] #1733 (merged into feat/horizontal-scaling) Answer vector store queries with record UUIDs and scores, and refuse invalid inputs feat/horizontal-scaling
[vector store scale-out 3/6] #1734 (merged into feat/horizontal-scaling) Arbitrate vector store collections in a SQL registry, with an incarnation per collection life and a purge feat/horizontal-scaling
[vector store scale-out 4/6] #1735 (merged into feat/horizontal-scaling) Move the Qdrant store onto the collection registry feat/horizontal-scaling
[vector store scale-out 5/6] #1736 (merged into feat/horizontal-scaling) Move the Milvus store onto the collection registry, against a Milvus server feat/horizontal-scaling
— #1775 (merged into feat/horizontal-scaling) Accept attempts exhausted in the lifecycle churn contract, and say which Qdrant operations filter on the incarnation feat/horizontal-scaling
— #1779 (merged into feat/horizontal-scaling) Classify Qdrant errors by status code alone feat/horizontal-scaling
— #1813 (merged into feat/horizontal-scaling) Create Qdrant collections with strict mode off feat/horizontal-scaling
— #1820 (merged into feat/horizontal-scaling) List only active projects in projects/list feat/horizontal-scaling
[user properties 1/2] #1702 (merged into feat/horizontal-scaling) Keep undeclared properties out of the vector store feat/horizontal-scaling
[user properties 2/2] #1670 (merged into feat/horizontal-scaling) Remove per-project filterable properties (port of #1606) feat/horizontal-scaling
— #1824 (merged into feat/horizontal-scaling) Drop the unread logical key from the purge queues feat/horizontal-scaling
— #1788 Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract feat/horizontal-scaling
— #1631 (closed) Superseded by #1733–#1736, which hold its changes split in four, with review changes since —
[vector store scale-out 6] #1627 (merged into feat/horizontal-scaling) Give each vector store a fixed schema and string-keyed partitions, built from the deployment's configuration feat/horizontal-scaling
[declared schema 1/2] #1628 (merged into feat/horizontal-scaling) Make a vector store filter only on the properties it declares feat/horizontal-scaling
[sqlite write transactions 1] #1844 (merged into feat/horizontal-scaling) Take SQLite's write lock at BEGIN in every store on a relational SQLite engine feat/horizontal-scaling
[session storage 1/2] #1622 Create a session's storage with the session, never on a request feat/horizontal-scaling
[session storage 2/2] #1625 Look a partition up with get_partition, and remove open-or-create from both stores and close from the segment store #1622
[selective filter 1] #1871 Route filtered queries by selectivity, and search allowlists exhaustively feat/horizontal-scaling
[search results, deferred] #1663 Score every vector search by cosine similarity, and name scores for it (port of #1598's cosine half) #1628
[declared schema 2/2] #1616 Close the filter union, and make negation the complement on every backend feat/horizontal-scaling
[sqlite store fixes 1/7] #1460 Publish vector index files atomically (but not durably) feat/horizontal-scaling
[sqlite store fixes 2/7] #1469 Never reuse a row id in SQLiteVectorStore #1460
[sqlite store fixes 3/7] #1672 Own the search engine's concurrency in the store, not in each engine (port of #1612) #1469
[sqlite store fixes 4/7] #1673 Serialize a partition's writes so the engine sees them in order (port of #1607) #1672
[sqlite store fixes 5/7] #1674 Refuse a pending row replay cannot honor, instead of dropping it (port of #1608) #1673
[sqlite store fixes 6/7] #1675 (closed) Superseded by #1627, which takes SQLite's write lock at BEGIN where the engine is created —
[sqlite store fixes 7/7] #1676 Give every write a fresh row id, so a key names one version (port of #1610) #1674
[sqlite store fixes 8] #1843 Hold the engine lock until a cancelled engine call's thread finishes #1676
[qdrant options] #1618 Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization feat/horizontal-scaling
[milvus options] #1741 Let a deployment tune a Milvus collection's vector index and its searches #1618
— #1841 (this PR) Pack vector store upserts by serialized size, and state what cancelling one guarantees feat/horizontal-scaling
[event memory recovery 1/2] #1839 Give EventMemory a store's shape, with a handle per partition feat/horizontal-scaling
[event memory recovery 2] #1840 Keep event memory's two stores consistent through failed ingests and forgets #1839
[event memory 1/6] #1684 Add time bounds, sources and expansion to EventMemory (port of #1597, without session) #1840
[property keys] #1828 Refuse an illegal metadata key before any store writes, and reserve one key prefix #1684
[schema migration 1/2] #1825 Add memmachine-server migrate, which creates every configured store's schema before servers start feat/horizontal-scaling
[schema migration 2/2] #1826 Make startup check what migrate created, and create nothing at startup #1825

This PR is its 3 commits, 5ab987dfb, 8aec258db, ad2b29dc6, directly on feat/horizontal-scaling. Its request packing is dropped and its cancellation contract moved into #1840, so it is to be closed; nothing is stacked on it.

Verification

At this PR's head ad2b29dc6, on feat/horizontal-scaling's 0fa52b38b:

Before the rebase onto #1628's 4b1210953, the vector store test directory with integration tests ran against qdrant/qdrant:v1.19.1 (REST and gRPC) and milvusdb/milvus:v2.6.24 through testcontainers: 766 passed, 5 skipped (registry tests for the other SQL dialect).

🤖 Written by Claude Code (Claude Opus 5.5) on behalf of @edwinyyyu.

This was referenced Oct 20, 2026
Draft
Draft
Draft
Draft
Merged
Draft
Draft
Merged
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Closed
Draft
Draft
Draft
Open
Draft
Open
Draft
Draft
Draft
Merged
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-size-based-upserts branch from df44c87 to ac84d80 Compare October 26, 2026 18:06
This was referenced Oct 26, 2026
Closed
Draft
Draft
This was referenced Oct 26, 2026
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-size-based-upserts branch 7 times, most recently from ac2727e to 68b34f7 Compare October 31, 2026 00:57
edwinyyyu and others added 3 commits November 2, 2026 09:55
@edwinyyyu @claude
Pack the remote stores' upserts into requests by serialized size
5ab987d 
The Qdrant and Milvus partitions sent an upsert as one request and, when
the server refused it as too large (Qdrant 400/413, Milvus
RESOURCE_EXHAUSTED), halved the batch and resent the halves. A refused
batch's bytes went out again at each halving, and a malformed Qdrant
batch was halved down to single points.

Each record's size is now computed as it goes on the wire, and the
records are packed, in order, into the fewest requests of at most a new
`max_request_bytes` setting, each sent once (`pack_requests` in
vector_store/utils.py):

- Qdrant over REST: the point's JSON from qdrant-client's own
  `jsonable_encoder`, plus its comma, inside the `{"points":[...]}`
  body; over gRPC: the point's `UpsertPoints` field (tag, length, and
  message) after qdrant-client's `RestToGrpc` conversion, plus the
  envelope `UpsertPoints(collection_name, wait=True)`. The transport is
  the client's `init_options["prefer_grpc"]`. Both are exact: unit tests
  compare them with the bytes a real client sends to an httpx mock
  transport and to an in-process gRPC server.
- Milvus (gRPC): pymilvus sends a request's fields as columns, so an
  entity's size is each value as an element of its column (a string with
  its tag and length, a packed bool, int64 varint, double, or 4 bytes
  per vector coordinate) plus a validity flag per nullable declared
  field. The request's own fields and each field's framing are an upper
  bound, 64 bytes per field plus its name and 64 per request plus the
  collection name. Against requests pymilvus 3.0.1 built, the bound was
  713 bytes where the actual framing was 195 to 271.

The defaults are the servers' own default limits:

- Qdrant: 33554432 bytes, `service.max_request_size_mb: 32` in
  config/config.yaml (embedded as the default configuration), applied as
  `max_request_size_mb * 1024 * 1024` to the REST JSON body in
  src/actix/mod.rs, both at v1.19.1. Its gRPC services set
  `max_decoding_message_size(usize::MAX)` (src/tonic/mod.rs), so gRPC
  has no limit.
- Milvus: 67108864 bytes, `proxy.grpc.serverMaxRecvSize` in
  configs/milvus.yaml at v2.6.24 (the code's fallback,
  DefaultServerMaxRecvSize, is 256 MiB), passed to grpc-go's
  MaxRecvMsgSize, which refuses a message longer than it with
  RESOURCE_EXHAUSTED.

A request within the setting that the server refuses as too large means
the setting exceeds the server's limit, so the refusal raises
QdrantConfigurationError or MilvusConfigurationError, chaining the
server's error. Qdrant's 400 is also its status for any invalid request,
which only the message tells apart, so its error says the setting
exceeds the limit unless the response names another fault. A record too
large for a request alone raises ValueError before anything is sent.
Retries, timeouts, order, and the registration checks around `_upsert`
are unchanged.

`max_request_bytes` is on QdrantConf, MilvusConf, and both stores'
params, plumbed by the database manager, and listed in
configuration.mdx; the design documents describe the packing in place of
the halving.

Verification: ruff, ruff format, and ty clean; server unit suite 2281
passed, 8 skipped. New integration tests against qdrant/qdrant:v1.19.1
(REST and gRPC) and milvusdb/milvus:v2.6.24 pass: a packed upsert
stores every record, Milvus's requests on the wire stay within the
setting, a record too large raises before any request, and a request
over the server's limit raises the configuration error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@edwinyyyu @claude
State in the upsert contract what cancelling an upsert guarantees
8aec258 
VectorStorePartition.upsert now states that cancelling the call stops it
from sending further requests to the backend, and that a request already
sent may still be applied, so a cancelled upsert may have written some
of the records. RegistryBackedVectorStorePartition's `_upsert` hook
states the same obligation for its implementations.

Every implementation honors it:

- RegistryBackedVectorStorePartition awaits the registration check,
  `_upsert`, and the check again in sequence, so a cancellation stops at
  the await it lands on.
- Qdrant and Milvus send their packed requests one after another, and a
  cancellation raised in a request ends the loop. Neither client catches
  CancelledError: qdrant-client's REST path catches Exception only, and
  pymilvus's retry decorators catch grpc.RpcError, MilvusException, and
  Exception, so a cancelled RPC surfaces as CancelledError and is not
  retried.
- sqlite-vec writes in one transaction. A statement already queued to
  aiosqlite's thread finishes there; SQLAlchemy treats the cancellation
  as a disconnect and invalidates the connection, so the transaction
  never commits and nothing more is executed on it.
- The engine-backed SQLite store commits its records and pending
  operations, then applies them to the search engine, then marks them
  applied. A cancellation in the engine step leaves the operations
  pending, and startup replays them.
- The in-memory test partition has no await in its upsert.

Tests cancel an upsert mid-call and check that nothing more goes out:
for Qdrant and Milvus with a fake client (no further request, and no
registration check after it) and against qdrant/qdrant:v1.19.1 over
REST and gRPC and milvusdb/milvus:v2.6.24 (the first request in flight;
for Milvus counted at the gRPC stub, so a pymilvus retry would show),
checking that only the first request's records may be stored; for both
SQLite stores, cancelling at the upsert's first statement, and for the
engine-backed one also while its search engine applies the records, no
further statement executes.

Verification: ruff, ruff format, and ty clean; server unit suite 2286
passed, 8 skipped; the vector store test directory with integration
tests, 766 passed, 5 skipped (the registry's dialect-specific tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@edwinyyyu @claude
Give the refused oversized Milvus request time to be built
ad2b29d 
The test that Milvus refuses a request over its size limit sends the
whole batch, mostly one declared VARCHAR property, as one request.
pymilvus checks a VARCHAR value one character at a time while it builds
that request, within the request's timeout, so on a slow runner the
build outlasts the module's request timeout before Milvus can refuse the
request. The test's own store gives its request longer, as the packing
test's store does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-size-based-upserts branch from 68b34f7 to ad2b29d Compare November 2, 2026 17:57
@edwinyyyu edwinyyyu closed this Nov 2, 2026
@edwinyyyu

edwinyyyu commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Packing involves some changes to operator-friendliness and tracking how upstream dependencies encode their requests and check their limits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@edwinyyyu