Repository navigation
[Bugfix][Model] Fix FP8 k_scale/v_scale not loaded for Qwen3-MoE - #35656
Merged
Isotr0py merged 4 commits intoMar 4, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request fixes a bug where FP8 KV cache scales were not loaded correctly for Qwen3-MoE models, leading to accuracy degradation. The fix involves removing k_scale and v_scale from ignore_suffixes and adding an early remapping for scale names, which aligns with the approach in other models. The changes are correct and well-supported by a comprehensive set of new unit tests that cover various checkpoint formats. The fix effectively addresses the reported issue.
Isotr0py
reviewed
Mar 2, 2026
Isotr0py
left a comment
Member
There was a problem hiding this comment.
Can you check if this change will affect modelopt quantized models like nvidia/Qwen3-30B-A3B-NVFP4? IIRC, the ignore suffix is workaround for modelopt quantization.
…m-project#25047) FP8 KV cache scales from llm-compressor checkpoints (e.g. `qkv_proj.k_scale`) were silently dropped during weight loading in Qwen3MoeModel, causing fallback to scale=1.0 and accuracy degradation. Root cause: scale names were caught by `ignore_suffixes` before `maybe_remap_kv_scale_name` could remap them to `attn.k_scale`. Fix: add early scale remapping (matching Llama's approach) between the `get_cache_scale` check and the stacked-params loop, and remove k_scale/v_scale from `ignore_suffixes` so they are no longer blocked. Verified compatible with ModelOpt NVFP4 checkpoints (e.g. nvidia/Qwen3-30B-A3B-NVFP4) — their k_proj.k_scale / v_proj.v_scale tensors are correctly remapped by maybe_remap_kv_scale_name. Made-with: Cursor Signed-off-by: raghavan <oneraghavan@gmail.com>
oneraghavan
force-pushed
the
fix/fp8-kv-scale-qwen3-moe-25047
branch
from
March 2, 2026 09:46
392aad8 to
3e15456
Compare
Extend the fix from the previous commit to qwen3_vl_moe.py, which had the same bug: ignore_suffixes silently dropped k_scale/v_scale before maybe_remap_kv_scale_name could remap them. Also add regression tests for Qwen3-VL-MoE and NVFP4 weight/input scales to confirm they are not affected by the remap. Signed-off-by: Raghavan <raghavan@users.noreply.github.com> Made-with: Cursor Signed-off-by: raghavan <oneraghavan@gmail.com>
Isotr0py
approved these changes
Mar 3, 2026
Isotr0py
enabled auto-merge (squash)
March 3, 2026 03:16
Contributor
Author
|
@Isotr0py Some builds are failing, not sure whats the issue . Please help me how to fix these. |
|
Gentle reminder @mgoin @sighingnow |
Copilot AI
pushed a commit
to machov/vllm
that referenced
this pull request
Mar 10, 2026
…m-project#35656) Signed-off-by: raghavan <oneraghavan@gmail.com>
avinashsingh77
pushed a commit
to avinashsingh77/vllm
that referenced
this pull request
Mar 12, 2026
…m-project#35656) Signed-off-by: raghavan <oneraghavan@gmail.com>
wendyliu235
pushed a commit
to wendyliu235/vllm-public
that referenced
this pull request
Mar 18, 2026
…m-project#35656) Signed-off-by: raghavan <oneraghavan@gmail.com>
mystous
pushed a commit
to mystous/vllm_hybrid
that referenced
this pull request
May 10, 2026
…m-project#35656) Signed-off-by: raghavan <oneraghavan@gmail.com>
my-other-github-account
pushed a commit
to my-other-github-account/vllm
that referenced
this pull request
May 15, 2026
…m-project#35656) Signed-off-by: raghavan <oneraghavan@gmail.com>
0826joyce
pushed a commit
to 0826joyce/vllm-serving-optimization
that referenced
this pull request
May 19, 2026
…m-project#35656) Signed-off-by: raghavan <oneraghavan@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FP8 KV cache scales from llm-compressor checkpoints (e.g.
qkv_proj.k_scale) were silently dropped during weight loading in Qwen3MoeModel, causing fallback to scale=1.0 and accuracy degradation.Root cause: scale names were caught by
ignore_suffixesbeforemaybe_remap_kv_scale_namecould remap them toattn.k_scale.Fix: add early scale remapping (matching Llama's approach) between the
get_cache_scalecheck and the stacked-params loop, and remove k_scale/v_scale fromignore_suffixesso they are no longer blocked.cc : @DarkLight1337