Intermittent deadlock on feast serve startup under gunicorn: feature-freshness thread starts pre-fork and can freeze an import lock forever #6647

Describe the bug

feast serve (running under gunicorn, uvicorn_worker.UvicornWorker) can hang forever at startup with no error, no traceback, and no log output beyond "Waiting for application startup.". It's intermittent — most starts succeed, but occasionally a worker freezes permanently and never becomes ready.

Root cause

feast_metrics.start_metrics_server() is called from feature_server.start_server(), which runs in the gunicorn master process, before FeastServeApplication(...).run() triggers gunicorn's fork() of the worker process(es).

start_metrics_server() correctly defers CPU/memory resource monitoring to after the fork (there's an explicit start_resource_monitoring=not uses_gunicorn check, plus a post_worker_init hook that calls init_worker_monitoring()). However, it does not apply the same treatment to the feature-freshness thread:

if _config.freshness:
    freshness_thread = threading.Thread(
        target=monitor_freshness, args=(store, 30), daemon=True
    )
    freshness_thread.start()

This thread is started unconditionally in the master (whenever feature_server.metrics.freshness is enabled), and its very first action — with no delay — is update_feature_freshness(store) → store.list_feature_views(...), which lazily builds the registry for the first time. For registry backends that need a lazy DBAPI import (e.g. the SQL registry importing pymysql via SQLAlchemy's create_engine()), this means a background thread in the gunicorn master may be mid-import, holding CPython's per-module import lock, at the exact moment gunicorn forks a worker.

POSIX fork() only duplicates the calling thread into the child process — every other thread that existed in the parent (including this freshness thread) simply ceases to exist in the worker. If the fork happens while that thread holds a module's import lock, the lock is left permanently in the "locked"/_initializing state inside the new worker, and there is no longer any thread anywhere that can finish that import and release it. When the worker's own FastAPI startup later tries to build its own registry and hits the same import, it deadlocks forever.

This matches a known general class of hazard (fork() + threads + an in-flight import lock — see e.g. https://bugs.python.org/issue6380), and it explains the intermittency exactly: it only manifests if the fork lands inside that narrow timing window.

We confirmed this live by attaching py-spy to a genuinely frozen worker process and dumping its stack — the main thread was blocked acquiring CPython's per-module import lock inside SQLAlchemy's lazy import_dbapi() (dialects/mysql/pymysql.py), called from SqlRegistry.__init__ → feature_store.py's registry property → refresh_registry() → feature_server.py's async_refresh() → the FastAPI lifespan.

We also verified against CPython's actual importlib._bootstrap._find_and_load that the per-module import lock is only acquired if a module is missing from sys.modules or still mid-initialization — once a module has fully finished importing, every later import statement, from any thread, before or after any number of forks, returns the cached module without ever touching the lock again. This is why the freshness thread's pre-fork timing is what makes it dangerous, and it points at the fix.

To Reproduce

1. Configure a registry backend that performs a lazy import on first use (e.g. registry_type: sql with a MySQL connection string, which lazily imports pymysql via SQLAlchemy).
2. Enable feature_server.metrics.freshness: true in feature_store.yaml.
3. Run feast serve --metrics under gunicorn (Linux, non-Windows path) with 1+ workers.
4. Restart the process repeatedly (e.g. in a container orchestrator that recycles the pod/process on every deploy or crash). Occasionally — timing-dependent, not consistently reproducible on demand — a worker will hang forever at "Waiting for application startup." with zero output.

Expected behavior

The feature-freshness thread (and ideally anything else start_metrics_server() spawns pre-fork that can trigger a lazy import) should be deferred to run after gunicorn forks its workers, the same way resource monitoring already is — e.g. via the existing post_worker_init hook — rather than starting unconditionally in the master before the fork.

Workaround

Force the DBAPI driver (e.g. pymysql) to finish importing once, synchronously, as the very first statement of the process — before feast's CLI runs, before any thread exists, and before gunicorn forks anything. A tiny wrapper around the CLI entrypoint works:

import pymysql  # must be first — see explanation above
import sys
from feast.cli.cli import cli

if __name__ == "__main__":
    sys.exit(cli())

...invoked instead of the feast console script (e.g. python entrypoint.py serve ...). This is unrelated to thread timing luck — once the module is fully imported, the import lock is never touched again for it, regardless of what any other thread does afterward.

Environment

• feast version: 0.62.0
• Python: 3.11
• OS: Linux (containerized), gunicorn + uvicorn_worker.UvicornWorker
• Registry: SQL registry (registry_type: sql, MySQL via pymysql)