fix(server): add bounded dependency readiness without changing liveness - #1853

Purpose of the change

The current health endpoint stays healthy while storage is unreachable, so the repository's deployment probes can keep routing traffic to a worker whose memory operations fail. This adds dependency readiness while preserving the existing liveness response.

Description

• Add GET /api/v2/ready: 200 after startup and successful required-storage checks; 503 during startup/shutdown, outages, timeouts, or unsupported required configurations. /api/v2/health keeps its status and body.
• Check session and episode stores, enabled semantic stores, and all configured storage available to enabled episodic long-term memory, including project alternatives. Disabled services do not require otherwise-unused dependencies. Responses expose only component types/ordinals and fixed status codes.
• Coalesce requests into one refresh, expire results one second after refresh starts, observe each resource for at most two seconds, and bound each response to three seconds. Four concurrent reservations and a two-thread executor bound work; timed-out or cancelled requests cannot start replacement operations while underlying work still runs.
• Reuse serving clients without closing or replacing them. Cold probes own separate read-only connections. Include PostgreSQL/SQLite, Qdrant, Neo4j, NebulaGraph, and Milvus probe adapters with explicit unsupported behavior.
• Wire startup, shutdown, and restart state; update API documentation, OpenAPI, Compose healthchecks, and Helm readiness while retaining /health for Kubernetes liveness.

Fixes/Closes

Fixes #1777

Type of change

• Bug fix
• Documentation update

How Has This Been Tested?

• Unit Test

• Integration Test

• Linux: 228 focused and adjacent tests passed, with four existing integration cases deselected. After the independent review's early-stop/restart fix, 60 readiness and lifecycle tests passed, including both new stop-before-start and early-start-failure/retry cases. The final focused file has 14 cases; all 14 also passed on macOS after the lifecycle correction.

• macOS with Docker: one native integration test passed, no skips, and passed again on the final source after the lifecycle correction, using disposable PostgreSQL, Neo4j, and Qdrant containers. Both event and declarative projects perform real memory adds/searches. Pausing each dependency changes readiness to 503 while liveness remains 200; unpausing restores 200 and successful adds/searches, preserves old data, and retains the same serving client/engine/store objects.

• The integration uses real application ASGI routing and native storage clients, rather than a separately listening HTTP server. Only the external embedding provider is replaced by the repository's deterministic FakeEmbedder; the identity reranker is local and real. Containers, ports, passwords, and data are test-owned; cleanup targets those exact containers.

• Ruff check and formatting pass for all changed Python files; ty passes for changed server modules; git diff --check and patch checks pass. Helm lint and four renders pass: default/event, declarative, external stores, and the older-image compatibility override. Compose's healthcheck structure and five-second curl bound were checked locally.

• Independent read-only review found the early-stop/restart edge above; it was fixed, regressed, and re-reviewed. No further blocking findings were reported. Existing Testcontainers/Starlette deprecation warnings remain.

Reproduce the real storage test from the repository root:

uv sync --frozen --package memmachine-server --extra qdrant --group dev
.venv/bin/pytest -o addopts= -m integration -q packages/server/server_tests/memmachine_server/server/api_v2/test_readiness_integration.py

Deployment compatibility and limits

The Helm/Compose readiness default now requires an application image containing /api/v2/ready. Upgrade the image before or together with the probe change. Older images can explicitly select memmachine.readiness.path=/api/v2/health or MEMMACHINE_READINESS_PATH=/api/v2/health; that mode checks liveness only. Old images with the new path remain visibly unready with 404. Roll back image and probe configuration together. Helm chart version increases to 0.3.0; application release metadata is not changed.

Readiness checks lightweight storage reachability/read access, not embedding/LLM providers, migrations, write permissions, or collection/index correctness. Cold Milvus Lite reports unsupported because creating its client would start a local server. Native outage/recovery evidence covers PostgreSQL, Neo4j, and Qdrant; no live NebulaGraph or remote Milvus result is claimed.

Checklist

• I have signed the commit(s) within this pull request
• Code follows the project style guidelines for the changed modules
• Implementation and tests were reviewed
• Non-obvious ownership and timeout behavior is commented
• Corresponding API and deployment documentation is included
• Unit tests cover the fix; the focused and adjacent suites above pass

Maintainer Checklist

• Confirmed all checks passed
• Contributor has signed the commit(s)
• Reviewed the code
• Run, Tested, and Verified the change(s) work as expected

Further comments

Implemented with OpenAI Codex and independently reviewed by a second coding agent. The final source was checked against the reviewed file manifest before submission. The GitHub-verified commit signature and Signed-off-by trailer were checked before opening this PR.