Repository navigation
Conversation
Port the q4_K o4 variant (4 outputs per work item, one shared activation read per k-block) to the q4_0 noshuffle layout. The long-vocab lm_head / embed GEMV (ne01 >= 32768) is the worst q4_0 decode shape (~34 GB/s on Adreno 740); 4 rows per WI doubles the bytes in flight per work item and amortizes the activation read/broadcast across 4 rows. The two 2-row dequantize/accumulate macros are reused unchanged, so each row's accumulation order (and thus the result) is byte-identical to the base GEMV. Default on for ne01 >= 32768; opt-out GGML_OPENCL_Q4_0_O4=0. tg64, gemma models, Adreno 740 (SM8550): gemma-4-E2B q4_0 11.47 -> 12.97 (+13.1%) gemma-4-E4B q4_0 7.90 -> 8.71 (+10.3%) gemma-3-1b q4_0 17.33 -> 17.97 (+3.7%) prefill unchanged (E2B pp256 205.4 -> 206.9); greedy output byte-identical.
|
Hi @EdgeBites, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a 4-output-per-work-item (
o4) GEMV for q4_0 decode, mirroring the existing q4_Ko4variant, for long-vocab lm_head / embed shapes.Motivation
The q4_0 decode GEMV for
ne01 >= 32768(lm_head / embed, e.g. 262144 rows on gemma-4) is the worst q4_0 GEMV shape: each work item computes only 2 output rows, so the per-block activation read + subgroup broadcasts are amortized over just 2 rows. The q4_K path already solves this withkernel_gemv_noshuffle_q4_k_f32_o4("4-output-per-WI, long-vocab lm_head"); q4_0 had no equivalent.Change
kernel_gemv_noshuffle_q4_0_f32_o4: 4 consecutive output rows per work item as two 2-row pairs (a, b) that share one activation read per k-block. The existing 2-rowdequantizeBlockAccum_ns_sgbroadcast_*macros are reused unchanged, so each row's accumulation order — and therefore the result — is byte-identical to the base GEMV.ne1 == 1),ne01 % 4 == 0 && ne01 >= 32768, default on, opt-outGGML_OPENCL_Q4_0_O4=0. Launch{CEIL_DIV(ne01/4,64)*64, 4, 1}; tail quads past the last row are guarded.Benchmarks (gemma models, Adreno 740 / SM8550)
llama-bench --device GPUOpenCL -ngl -1 -fa on -p 1 -n 64 -r 2, interleaved A/B (base = same tree without the change):