fix: Defer feature-freshness thread to post-fork to avoid Gunicorn deadlock - #6648
Merged
jyejare merged 1 commit intoJul 27, 2026
Merged
Conversation
casaar97
force-pushed
the
fix/freshness-thread-postfork-deadlock
branch
from
July 27, 2026 13:30
bb3deed to
597a845
Compare
Member
|
@jyejare ^ |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6648 +/- ##
=======================================
Coverage 45.93% 45.93%
=======================================
Files 414 414
Lines 49999 50006 +7
Branches 7146 7147 +1
=======================================
+ Hits 22965 22972 +7
Misses 25423 25423
Partials 1611 1611
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
…adlock feast_metrics.start_metrics_server() starts a feature-freshness thread in the Gunicorn master process, before Gunicorn forks its worker(s). That thread's first action, with no delay, is update_feature_freshness() -> store.list_feature_views(), which lazily builds the registry for the first time. For registry backends needing a lazy DBAPI import (e.g. the SQL registry importing pymysql via SQLAlchemy's create_engine()), this means a thread in the master can be mid-import, holding CPython's per-module import lock, at the exact moment Gunicorn forks a worker. POSIX fork() only duplicates the calling thread into the child process; every other thread in the parent, including this one, simply ceases to exist in the worker. If the fork lands while that thread holds a module's import lock, the lock stays permanently held in the new worker, since there is no longer any thread that can finish the import and release it. The worker's own later attempt to build its registry then deadlocks forever with no error - the process just hangs at "Waiting for application startup." This is intermittent by nature: it only manifests if the fork lands inside that narrow timing window. Resource monitoring already avoids this correctly (start_resource_monitoring= not uses_gunicorn plus the post_worker_init hook calling init_worker_monitoring()), but the freshness thread was not given the same treatment. This applies the identical pattern: start_metrics_server gains a start_freshness_monitoring flag (deferred exactly like resource monitoring's), and FeastServeApplication's post_worker_init hook now also calls the new init_worker_freshness_monitoring(store) after the fork, instead of feast_metrics.py starting it unconditionally beforehand. Fixes feast-dev#6647 Signed-off-by: Carlos Sánchez <carlos.sancheza@cabify.com>
casaar97
force-pushed
the
fix/freshness-thread-postfork-deadlock
branch
from
July 27, 2026 13:48
597a845 to
5376751
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes an intermittent feast serve startup deadlock under Gunicorn by deferring the feature-freshness monitoring thread until after workers are forked, mirroring the existing “post-fork” pattern already used for resource monitoring.
Changes:
- Add a
start_freshness_monitoringflag tofeast.metrics.start_metrics_server()to prevent starting the freshness thread in the Gunicorn master process. - Introduce
init_worker_freshness_monitoring(store)and invoke it from Gunicorn’spost_worker_inithook (via afunctools.partialcapturing theFeatureStore). - Add unit/regression tests to ensure the worker-init freshness behavior is gated by config and stays in lockstep with resource monitoring deferral.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| sdk/python/feast/metrics.py | Adds post-fork freshness init helper and defers freshness thread startup via a new start_freshness_monitoring flag. |
| sdk/python/feast/feature_server.py | Updates Gunicorn post_worker_init hook wiring to start freshness monitoring per worker after fork. |
| sdk/python/tests/unit/test_metrics.py | Adds tests for init_worker_freshness_monitoring gating and ensures deferral flags remain synchronized. |
| sdk/python/tests/unit/test_feature_server.py | Adds a Gunicorn hook test asserting both resource and freshness monitoring are started post-fork. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
jyejare
approved these changes
Jul 27, 2026
jyejare
added a commit
to opendatahub-io/feast
that referenced
this pull request
Jul 28, 2026
…#166) * feat: Multi-arch publish for feast operator image Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * feat: Allow users to have protected project on shared registry Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> * fix: Defer feature-freshness thread to post-fork to avoid Gunicorn deadlock (feast-dev#6648) feast_metrics.start_metrics_server() starts a feature-freshness thread in the Gunicorn master process, before Gunicorn forks its worker(s). That thread's first action, with no delay, is update_feature_freshness() -> store.list_feature_views(), which lazily builds the registry for the first time. For registry backends needing a lazy DBAPI import (e.g. the SQL registry importing pymysql via SQLAlchemy's create_engine()), this means a thread in the master can be mid-import, holding CPython's per-module import lock, at the exact moment Gunicorn forks a worker. POSIX fork() only duplicates the calling thread into the child process; every other thread in the parent, including this one, simply ceases to exist in the worker. If the fork lands while that thread holds a module's import lock, the lock stays permanently held in the new worker, since there is no longer any thread that can finish the import and release it. The worker's own later attempt to build its registry then deadlocks forever with no error - the process just hangs at "Waiting for application startup." This is intermittent by nature: it only manifests if the fork lands inside that narrow timing window. Resource monitoring already avoids this correctly (start_resource_monitoring= not uses_gunicorn plus the post_worker_init hook calling init_worker_monitoring()), but the freshness thread was not given the same treatment. This applies the identical pattern: start_metrics_server gains a start_freshness_monitoring flag (deferred exactly like resource monitoring's), and FeastServeApplication's post_worker_init hook now also calls the new init_worker_freshness_monitoring(store) after the fork, instead of feast_metrics.py starting it unconditionally beforehand. Fixes feast-dev#6647 Signed-off-by: Carlos Sánchez <carlos.sancheza@cabify.com> Co-authored-by: Carlos Sánchez <carlos.sancheza@cabify.com> --------- Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Signed-off-by: Carlos Sánchez <carlos.sancheza@cabify.com> Co-authored-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Carlos Sánchez <i52saarc@uco.es> Co-authored-by: Carlos Sánchez <carlos.sancheza@cabify.com>
aniketpalu
pushed a commit
to aniketpalu/feast
that referenced
this pull request
Jul 28, 2026
…adlock (feast-dev#6648) feast_metrics.start_metrics_server() starts a feature-freshness thread in the Gunicorn master process, before Gunicorn forks its worker(s). That thread's first action, with no delay, is update_feature_freshness() -> store.list_feature_views(), which lazily builds the registry for the first time. For registry backends needing a lazy DBAPI import (e.g. the SQL registry importing pymysql via SQLAlchemy's create_engine()), this means a thread in the master can be mid-import, holding CPython's per-module import lock, at the exact moment Gunicorn forks a worker. POSIX fork() only duplicates the calling thread into the child process; every other thread in the parent, including this one, simply ceases to exist in the worker. If the fork lands while that thread holds a module's import lock, the lock stays permanently held in the new worker, since there is no longer any thread that can finish the import and release it. The worker's own later attempt to build its registry then deadlocks forever with no error - the process just hangs at "Waiting for application startup." This is intermittent by nature: it only manifests if the fork lands inside that narrow timing window. Resource monitoring already avoids this correctly (start_resource_monitoring= not uses_gunicorn plus the post_worker_init hook calling init_worker_monitoring()), but the freshness thread was not given the same treatment. This applies the identical pattern: start_metrics_server gains a start_freshness_monitoring flag (deferred exactly like resource monitoring's), and FeastServeApplication's post_worker_init hook now also calls the new init_worker_freshness_monitoring(store) after the fork, instead of feast_metrics.py starting it unconditionally beforehand. Fixes feast-dev#6647 Signed-off-by: Carlos Sánchez <carlos.sancheza@cabify.com> Co-authored-by: Carlos Sánchez <carlos.sancheza@cabify.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6647
Summary
feast_metrics.start_metrics_server()starts a feature-freshness thread in the Gunicorn master process, before Gunicorn forks its worker(s). That thread's first action, with no delay, isupdate_feature_freshness()→store.list_feature_views(), which lazily builds the registry for the first time. For registry backends needing a lazy DBAPI import (e.g. the SQL registry importingpymysqlvia SQLAlchemy'screate_engine()), this means a thread in the master can be mid-import, holding CPython's per-module import lock, at the exact moment Gunicorn forks a worker.POSIX
fork()only duplicates the calling thread into the child process; every other thread in the parent, including this one, simply ceases to exist in the worker. If the fork lands while that thread holds a module's import lock, the lock stays permanently held in the new worker, since there is no longer any thread that can finish the import and release it. The worker's own later attempt to build its registry then deadlocks forever with no error - the process just hangs at"Waiting for application startup.". This is intermittent by nature: it only manifests if the fork lands inside that narrow timing window (see #6647 for the full write-up, including a livepy-spystack trace confirming the exact mechanism).Resource monitoring already avoids this correctly (
start_resource_monitoring=not uses_gunicornplus thepost_worker_inithook callinginit_worker_monitoring()), but the freshness thread wasn't given the same treatment. This PR applies the identical, already-established pattern:start_metrics_server()gains astart_freshness_monitoringflag, deferred exactly likestart_resource_monitoringalready is.init_worker_freshness_monitoring(store)mirrorsinit_worker_monitoring().FeastServeApplication'spost_worker_inithook now also callsinit_worker_freshness_monitoring(store)after the fork (via a closure capturingstore, since the freshness thread needs the store reference, unlike resource monitoring).Test plan
TestInitWorkerFreshnessMonitoringcovering that the new function starts/doesn't start the thread based on thefreshnessconfig flag, mirroring the existinginit_worker_monitoringbehavior.test_freshness_monitoring_deferred_same_as_resource_monitoringtoTestMetricsYamlConfig, assertingstart_freshness_monitoringis always passed identically tostart_resource_monitoringfromstart_server()- a regression guard so this can't silently drift out of sync again.sdk/python/tests/unit/test_metrics.py(89 tests) andsdk/python/tests/unit/test_feature_server.py/test_feature_server_utils.py(92 tests) pass locally.ruff check/ruff format --checkpass on all changed files.