Skip to content

[Core] Remove busy loop from idle buffer readers - #28053

Merged
njhill merged 41 commits into
vllm-project:mainfrom
joerunde:new-poll-fix
Mar 4, 2026
Merged

[Core] Remove busy loop from idle buffer readers#28053
njhill merged 41 commits into
vllm-project:mainfrom
joerunde:new-poll-fix

Conversation

@joerunde

@joerunde joerunde commented Nov 4, 2025 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Collaborator

Purpose

IBM bundles vLLM with SystemZ mainframes, and in these systems:

  1. The CPU cores are extremely expensive
  2. The systems are shared across many tenants (think multiple host operating systems and many VMs running on each) so it is important to not use any cpu cycles while not doing work.

This makes the fix from #16226 unacceptable for our customers, so this PR is a similar alternative to #24618 that completely removes the need for a busy loop at idle in the shared memory buffer readers.

This PR uses ZMQ to send write notifications which readers can wait for, however it also uses an in-process socket to wake up the reader on shut down so that there's no need to constantly wake up to check for a cancellation flag. This allows us to sleep indefinitely on a zmq.Poller so that the reader only wakes up when there is work to do, or it needs to shutdown.

I tried to contain the diff here inside of shm_broadcast.py as much as possible, the main change is swapping SpinTimer with SpinCondition. The changes that leaked out are:

  • The MessageQueue.deque() api no longer takes a cancel event, instead it exposes a .shutdown() api that the thread running monitor_parent_death calls.
  • The VLLM_SLEEP_WHEN_IDLE env var has been removed

Test Plan

  • I need to write some little unit tests to cover SpinCondition before taking out of draft
  • Benchmarks need to be run to validate that this doesn't cause a performance regression

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@joerunde
wip
67623aa 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
@joerunde
⚡ functional SpinCondition
3ce3a48 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
@joerunde
🎨 cleanup
21c7b04 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
@joerunde
🎨 fmt
b296ad5 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
@mergify mergify Bot added the v1 label Nov 4, 2025
@mergify

mergify Bot commented Nov 4, 2025

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @joerunde.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 4, 2025
@joerunde
🎨 cleanup
0b84082 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
@joerunde
Merge branch 'main' into new-poll-fix
6a36b18 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
@mergify mergify Bot removed the needs-rebase label Nov 4, 2025
@mergify

mergify Bot commented Nov 10, 2025

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @joerunde.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 10, 2025
joerunde
joerunde commented Nov 13, 2025
View reviewed changes
Comment thread vllm/v1/executor/multiproc_executor.py
active_procs = lambda: [proc for proc in worker_procs if proc.is_alive()]

# Give processes time to clean themselves up properly
if wait_for_termination(active_procs(), 4):

@joerunde joerunde Nov 13, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could shorten these 4s so the total time is not so long in the worst case

@joerunde
Merge branch 'main' into new-poll-fix
8037303 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
@mergify mergify Bot removed the needs-rebase label Nov 13, 2025
joerunde and others added 3 commits November 13, 2025 14:50
@joerunde
💩 WIP unit tests
00c4c3f 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
@tjohnson31415
test: flesh out shm_broadcast tests
f55c68e 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
@tjohnson31415
fix timeout handling and little refactor
0bd12b3 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
@mergify

mergify Bot commented Nov 17, 2025

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @joerunde.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@tjohnson31415
Merge branch 'main' into new-poll-fix
74cc6d5 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
tjohnson31415 and others added 4 commits November 20, 2025 07:46
@tjohnson31415
Merge branch 'main' into new-poll-fix
9c97af5
@tjohnson31415
test: add busy shutdown test
73c3398 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
@joerunde
Merge branch 'main' into new-poll-fix
184db2a
@tjohnson31415
Merge branch 'main' into new-poll-fix
28a4e60 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
@joerunde
joerunde marked this pull request as ready for review December 18, 2025 19:09
lihaokun-2026 pushed a commit to lihaokun-2026/vllm-ascend that referenced this pull request Mar 29, 2026
@menogrey @MrZ20 lihaokun-2026
[main2main] upgrade vllm to 0308 (vllm-project#7213)
a9b621a 
### What this PR does / why we need it?
Update main2main to vllm 0308.
breaks:

* vllm-project/vllm#30681
* vllm-project/vllm#35552 remove
self.cudagraph_batch_sizes
* vllm-project/vllm#35158 clear_metadata ->
defer_finalize
* vllm-project/vllm#36006 remove
CacheConfig.cpu_offload_gb
* vllm-project/vllm#35472
* vllm-project/vllm#34552 attn_metadata_builder
* vllm-project/vllm#30515 profile_seq_lens
* vllm-project/vllm#28053 

- vLLM version: v0.16.0
- vLLM main:
vllm-project/vllm@4034c3d

---------

Signed-off-by: MrZ20 <2609716663@qq.com>
Signed-off-by: menogrey <1299267905@qq.com>
Co-authored-by: MrZ20 <2609716663@qq.com>
chenchuw886 pushed a commit to chenchuw886/vllm-ascend that referenced this pull request Apr 1, 2026
@menogrey @MrZ20
[main2main] upgrade vllm to 0308 (vllm-project#7213)
cd12698 
### What this PR does / why we need it?
Update main2main to vllm 0308.
breaks:

* vllm-project/vllm#30681
* vllm-project/vllm#35552 remove
self.cudagraph_batch_sizes
* vllm-project/vllm#35158 clear_metadata ->
defer_finalize
* vllm-project/vllm#36006 remove
CacheConfig.cpu_offload_gb
* vllm-project/vllm#35472
* vllm-project/vllm#34552 attn_metadata_builder
* vllm-project/vllm#30515 profile_seq_lens
* vllm-project/vllm#28053 

- vLLM version: v0.16.0
- vLLM main:
vllm-project/vllm@4034c3d

---------

Signed-off-by: MrZ20 <2609716663@qq.com>
Signed-off-by: menogrey <1299267905@qq.com>
Co-authored-by: MrZ20 <2609716663@qq.com>
@njhill njhill mentioned this pull request Apr 6, 2026
Merged
5 tasks
@lee-b lee-b mentioned this pull request Apr 8, 2026
Closed
1 task done
yangzhe-2026 pushed a commit to yangzhe-2026/vllm-ascend that referenced this pull request May 6, 2026
@menogrey @MrZ20
[main2main] upgrade vllm to 0308 (vllm-project#7213)
ee06e8d 
### What this PR does / why we need it?
Update main2main to vllm 0308.
breaks:

* vllm-project/vllm#30681
* vllm-project/vllm#35552 remove
self.cudagraph_batch_sizes
* vllm-project/vllm#35158 clear_metadata ->
defer_finalize
* vllm-project/vllm#36006 remove
CacheConfig.cpu_offload_gb
* vllm-project/vllm#35472
* vllm-project/vllm#34552 attn_metadata_builder
* vllm-project/vllm#30515 profile_seq_lens
* vllm-project/vllm#28053 

- vLLM version: v0.16.0
- vLLM main:
vllm-project/vllm@4034c3d

---------

Signed-off-by: MrZ20 <2609716663@qq.com>
Signed-off-by: menogrey <1299267905@qq.com>
Co-authored-by: MrZ20 <2609716663@qq.com>
@maikmerten maikmerten mentioned this pull request May 7, 2026
Closed
1 task done
mystous pushed a commit to mystous/vllm_hybrid that referenced this pull request May 10, 2026
@joerunde @tjohnson31415 @njhill
[Core] Remove busy loop from idle buffer readers (vllm-project#28053)
90065d4 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Travis Johnson <tsjohnso@us.ibm.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
mystous pushed a commit to mystous/vllm_hybrid that referenced this pull request May 10, 2026
@tjohnson31415 @njhill
[Bugfix] Quickfix followups to busy loop removal in vllm-project#28053 (
5e764cb 
vllm-project#36068)

Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
nanxingMy pushed a commit to nanxingMy/vllm-ascend that referenced this pull request May 15, 2026
@menogrey @MrZ20
[main2main] upgrade vllm to 0308 (vllm-project#7213)
26917a6 
### What this PR does / why we need it?
Update main2main to vllm 0308.
breaks:

* vllm-project/vllm#30681
* vllm-project/vllm#35552 remove
self.cudagraph_batch_sizes
* vllm-project/vllm#35158 clear_metadata ->
defer_finalize
* vllm-project/vllm#36006 remove
CacheConfig.cpu_offload_gb
* vllm-project/vllm#35472
* vllm-project/vllm#34552 attn_metadata_builder
* vllm-project/vllm#30515 profile_seq_lens
* vllm-project/vllm#28053 

- vLLM version: v0.16.0
- vLLM main:
vllm-project/vllm@4034c3d

---------

Signed-off-by: MrZ20 <2609716663@qq.com>
Signed-off-by: menogrey <1299267905@qq.com>
Co-authored-by: MrZ20 <2609716663@qq.com>
Signed-off-by: nanxing <1014662416@qq.com>
my-other-github-account pushed a commit to my-other-github-account/vllm that referenced this pull request May 15, 2026
@joerunde @tjohnson31415 @njhill
[Core] Remove busy loop from idle buffer readers (vllm-project#28053)
f843106 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Travis Johnson <tsjohnso@us.ibm.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
my-other-github-account pushed a commit to my-other-github-account/vllm that referenced this pull request May 15, 2026
@tjohnson31415 @njhill
[Bugfix] Quickfix followups to busy loop removal in vllm-project#28053 (
1a4e355 
vllm-project#36068)

Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
This was referenced May 18, 2026
Closed
Merged
@1027866388 1027866388 mentioned this pull request May 18, 2026
Open
1027866388 added a commit to 1027866388/vllm-ascend that referenced this pull request May 18, 2026
@1027866388
[Misc] Remove useless sched_yield patch
8510dd0 
The upstream vLLM has fixed the sched_yield issue:
vllm-project/vllm#28053

The workaround patch is no longer needed.

Fixes: vllm-project#9237
Signed-off-by: a1eva <1027866388@users.noreply.github.com>
wangxiyuan added a commit to vllm-project/vllm-ascend that referenced this pull request May 19, 2026
@wangxiyuan
[Misc] Remove useless patch (#9237)
1e1595f 
patch_sched_yield is useless now. The problem is fixed by
vllm-project/vllm#28053

- vLLM version: v0.20.2
- vLLM main:
vllm-project/vllm@0d4d334

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Tian-Fantasea pushed a commit to Tian-Fantasea/vllm-ascend that referenced this pull request May 19, 2026
@wangxiyuan @Tian-Fantasea
[Misc] Remove useless patch (vllm-project#9237)
71b5c7b 
patch_sched_yield is useless now. The problem is fixed by
vllm-project/vllm#28053

- vLLM version: v0.20.2
- vLLM main:
vllm-project/vllm@0d4d334

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Tian <tt553093031@gmail.com>
Tian-Fantasea pushed a commit to Tian-Fantasea/vllm-ascend that referenced this pull request May 19, 2026
@wangxiyuan @Tian-Fantasea
[Misc] Remove useless patch (vllm-project#9237)
c950921 
patch_sched_yield is useless now. The problem is fixed by
vllm-project/vllm#28053

- vLLM version: v0.20.2
- vLLM main:
vllm-project/vllm@0d4d334

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Tian <tt553093031@gmail.com>
@nwpu-zxr nwpu-zxr mentioned this pull request Oct 22, 2026
Closed
1 of 2 tasks
@Absoler Absoler mentioned this pull request Oct 30, 2026
Open
1 task done
@floatlibai floatlibai mentioned this pull request Nov 2, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments
@cursor cursor[bot] cursor[bot] left review comments
@njhill njhill njhill approved these changes
@chaunceyjiang chaunceyjiang Awaiting requested review from chaunceyjiang
+1 more reviewer
@tjohnson31415 tjohnson31415 tjohnson31415 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@joerunde @tdoublep @njhill @tjohnson31415 @markmc