[milvus options] Let a deployment tune a Milvus collection's vector index and its searches - #1741

Purpose of the change

MilvusConf gains vector_index, the vector index of the store's collection: a plain mapping of index_type, params (build parameters) and search_params (parameters every search passes), as Milvus takes them for that index type, so pymilvus stays optional for configuration parsing; the store's params coerce it into MilvusVectorIndex, whose parameter values are scalars. Unset selects the store's own index, HNSW_SQ (M 18, efConstruction 240, SQ4U codes refined at FP16) searched with refine_k 8, as before. Startup creates the index unless the collection has one, in which case the collection keeps its index; the search parameters apply to every query from startup on. The sample configurations and the configuration docs' parameter table carry the key, and the Milvus design doc says the index and its searches are configurable as one spec.

One guard: params may not set index_type or metric_type, which pymilvus's IndexParam would let override the spec's own index type and the store's metric. Everything else is Milvus's to check: it checks the parameters the index type takes when it creates the index and when a search reaches an indexed segment, and ignores other keys. The collection isolates partitions by key, which Milvus documents for the HNSW family only, so a deployment picks an index of that family; Milvus 2.6.24 accepts another type (IVF_FLAT) on the collection without error, so the store leaves the choice to the deployment.

Two commits: the option, with its docstrings, the configuration page's parameter table row and the design doc's vector index paragraph, and the sample configurations' comments. Adaptation from #1618: pymilvus has no typed models for index or search parameters, so there is nothing to validate each mapping against; the unit of configuration is the whole index spec, so build and search parameters always come from one author for one index type. The option lands on the registry-backed Milvus store of [vector store scale-out 5/6] and [vector store scale-out 6], on its MilvusVectorStoreParams; the index it sets is the one startup creates for the collection's vector field, the partition handles the store builds carry its search parameters, and the tests build their store through the module's _params helper.

Position: on [qdrant options] #1618 directly, so the two options PRs share one edit of the configuration table and the sample configurations.

Stack

24 open PRs: three independent PRs, and the vector store tree of short parallel branches. Every open PR in the tree has feat/horizontal-scaling as its GitHub base, and the independent PRs have main. The branches are in a fork, and a pull request can target only this repository's branches, so the on column gives the order the PRs build on each other. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1675 is closed, superseded by #1627. #1841 is closed: its request packing is dropped, and its cancellation contract is in #1840. #1828 and #1839 sit side by side on the branch; #1839 and #1840 are the event memory recovery stack, and #1684 heads the event memory stack, which its own PRs index, listed because it builds on #1840, its branch also carrying #1828's commits until one of the two merges; #1871 keeps its own Stack section. Until the PRs under it merge, their changes show in a stacked PR's diff.

This PR is its 2 commits, 6150110b1, 8971a264b, stacked on #1618.

Verification

Restacked with the vector store tree, at 8971a264b (on #1618's 9760076f7), on 2026-10-30, with #1618 directly on feat/horizontal-scaling at 690116e37, where [declared schema 1/2] (#1628) is merged as 5c9d3ae99, with #1831 before it and #1858 after it; [sqlite write transactions 1] (#1844) is no longer beneath it, since neither PR uses any of #1844's code: its own lines are unchanged, and its two commits are re-signed, with their messages and author unchanged. Checks: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean; the vector store, configuration, and resource manager unit tests pass, 649 tests (5 skipped). The integration tests did not run here. This head's tree differs from 2c78deb61 only by the changes beneath it: #1702's filter routing, #1820, #1670's review fix, main's #1785, #1803, #1787, #1802, #1522, and #1774, #1824, #1627's own since then (its partition lookup named open_partition in place of get_partition, its per-partition schema record removed, require_partition_key named validate_partition_key, the store's parameters off its interface and partition_key off the partition handle, partition keys of up to 255 bytes with the SQLite stores' names hashed from them, the bool checks named is_valid_identifier and is_valid_filter, its raising checks named validate_, the retention check named _validate_tombstone_retention, its tombstones without the partition key, its partition registry document's wording, the first and second reviews' fixes, the accuracy fixes, and its changes since e38e8003e: each store's schema passed as one VectorStoreSchema, SQLite partitions named by an incarnation, the shared SQL engine setup and write transactions in common/sql.py, and every session's event memory served from the deployment's configuration), #1628's longer request timeout for its oversized Milvus upsert test, #1628's four commits from Edwin's review (no supported_filter_nodes, its checks in utils.py, sqlite_properties.py, and the strict-mode comment), #1628's second round of review commits (the identifier rule in data_types.py, which imports nothing from the package, and the interface's Raises sections sorted by name), #1628's is_valid_identifier docstring and its records table without extend_existing, #1628's last commits before its merge (property_column_names deriving the names from the naming rule, the contract's re-upsert check, and its tidied tests), #1831, and #1858, with [sqlite write transactions 1] (#1844) and [search results] (#1663) no longer beneath it. The database manager's tests last ran at 2c78deb61, on 2026-10-09: the vector store suites and the database manager's tests, unit and integration, 731 passed (5 skipped), against Qdrant 1.19.1 defaulting new collections to strict mode and Milvus 2.6.24 in containers.

Rebased onto #1618's head bb19702a1 (on feat/horizontal-scaling with #1736 merged), at d67f8af76: ruff check, ruff format --check, ty check, and uv lock --check are clean. The rebase resolved conflicts with the Milvus store's changes since 2026-10-02 (the metrics factory beside the vector index in the store's parameters and the database manager, grpc beside the pydantic imports, and the configuration and manager tests' settings beside the vector index), and the not-positive settings test now builds the parameters with model_validate, since the vector index field makes the keyword form ill-typed. The Milvus store, configuration, and database-manager tests pass against Milvus 2.6.24 in a test container (169 passed); the full suites were not rerun at this head.

Before the rebase:

At each of this PR's commits, 42c9c17d5 and d641da2aa, on 2026-10-02: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); the server suite without integration tests passes, 2037 tests (6 skipped). At d641da2aa, the Milvus store's integration tests, against Milvus 2.6.24 in a container, pass (62); the resource manager's tests carry no integration mark and run in the server suite above. The sample configurations' commented block, uncommented, parses into MilvusConf and MilvusVectorIndex in all three files. The behavior the docs state was checked against Milvus 2.6.24: index creation refused an unknown index type and a non-integer M and accepted an unknown build key; a search on an indexed segment refused ef of "abc", of -5 and below limit, and ignored an unknown key; IVF_FLAT was accepted on the collection. Both commits are signed.

🤖 Generated with Claude Code