Repository navigation
Conversation
use_adreno_kernels() requires ne0 >= 512 && ne1 >= 512 on E031 >= 38.11 compilers, so q4_0 weights with ne1 < 512 (K/V projections, per-layer embedding gates/projections, low-rank adapters) fall back to kernel_mul_mm_q4_0_f32_l4_lm, which runs at ~5-11 GFLOPS vs ~300-500 GFLOPS for the trans-weight GEMM on the same shapes. Lower the threshold to 128 (the value older compilers already use). Verified on Adreno 740 (E031.41.03.64), llama-bench pp256/tg64 r2: gemma-4-E2B pp 45.9 -> 205.3 t/s (+348%) gemma-4-E4B pp 29.2 -> 109.5 (+275%) gemma-3-1b pp 105.9 -> 497.0 (+369%) granite-350m pp 137.8 -> 1066.7 (+674%) LFM2.5-230M pp 2006 -> 2115 (+5%), tg neutral Greedy output is byte-identical between builds.
|
Hi @EdgeBites, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
EdgeBites
marked this pull request as ready for review
October 16, 2026 05:20
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
use_adreno_kernels()gates the Adreno trans-weight GEMM path behindne0 >= 512 && ne1 >= 512for E031 >= 38.11 compilers (older compilers use128). As a result, every q4_0 weight with
ne1 < 512falls back tokernel_mul_mm_q4_0_f32_l4_lm, which on Adreno 740 runs at ~5-11 GFLOPSwhile the trans-weight GEMM reaches ~300-500 GFLOPS on the same shapes —
~50x slower.
The affected weights are exactly the narrow ones: K/V projections
(ne1 = kv_dim), per-layer embedding gates/projections (ne1 = n_embd_altup),
and low-rank adapters (ne1 = rank). On gemma-4-E2B these run once per layer,
and kernel profiling shows the
l4_lmkernel accounts for 79 % of theprefill time (
-DGGML_OPENCL_PROFILING=ON: 4.4 s of 5.6 s per 256-tokenbatch; 92 calls x ~48 ms).
This PR lowers the threshold to 128 — the value the same function already
uses for older compilers. The shape constraints for the transposed layout
(ne0 % 32 == 0, ne1 % 4 == 0, ne2 == ne3 == 1, element count < 2^27) are
still enforced by
enable_adreno_trans_weight().Additional information
Verified on Adreno 740 (E031.41.03.64),
llama-bench --device GPUOpenCL -ngl -1 -p 256 -n 64 -r 2:Greedy 24-token generation is byte-identical between the two builds.
Evidence:
GGML_SCHED_DEBUG=2shows only the q4_0 embedding gathers on CPU;the per-kernel profile (cl_profiling.csv) attributes 4.4 s of 5.6 s to
kernel_mul_mm_q4_0_f32_l4_lmat ~10.8 ms per small matmul.Alternative fix: repair
kernel_mul_mm_q4_0_f32_l4_lmitself (it is ~1000xslower than a straightforward tiled-GEMM estimate); this PR takes the
low-risk route of using the already-fast path.
Requirements
developed with AI assistance (opencode). I understand the change fully and
will maintain it.