Skip to content

opencl: use Adreno trans-weight GEMM for narrow q4_0 weights (a7x) - #30646

Open
EdgeBites wants to merge 1 commit into
ggml-org:masterfrom
EdgeBites:opencl-a7x-narrow-weight-gemm
Open

EdgeBites wants to merge 1 commit into
ggml-org:masterfrom
EdgeBites:opencl-a7x-narrow-weight-gemm

Conversation

@EdgeBites

Copy link
Copy Markdown

Overview

use_adreno_kernels() gates the Adreno trans-weight GEMM path behind
ne0 >= 512 && ne1 >= 512 for E031 >= 38.11 compilers (older compilers use
128). As a result, every q4_0 weight with ne1 < 512 falls back to
kernel_mul_mm_q4_0_f32_l4_lm, which on Adreno 740 runs at ~5-11 GFLOPS
while the trans-weight GEMM reaches ~300-500 GFLOPS on the same shapes —
~50x slower.

The affected weights are exactly the narrow ones: K/V projections
(ne1 = kv_dim), per-layer embedding gates/projections (ne1 = n_embd_altup),
and low-rank adapters (ne1 = rank). On gemma-4-E2B these run once per layer,
and kernel profiling shows the l4_lm kernel accounts for 79 % of the
prefill time
(-DGGML_OPENCL_PROFILING=ON: 4.4 s of 5.6 s per 256-token
batch; 92 calls x ~48 ms).

This PR lowers the threshold to 128 — the value the same function already
uses for older compilers. The shape constraints for the transposed layout
(ne0 % 32 == 0, ne1 % 4 == 0, ne2 == ne3 == 1, element count < 2^27) are
still enforced by enable_adreno_trans_weight().

Additional information

Verified on Adreno 740 (E031.41.03.64), llama-bench --device GPUOpenCL -ngl -1 -p 256 -n 64 -r 2:

model pp256 before pp256 after Δpp tg64 before tg64 after
gemma-4-E2B 45.9 205.3 +348 % 11.5 11.6
gemma-4-E4B 29.2 109.5 +275 % 7.8 7.9
gemma-3-1b 105.9 497.0 +369 % 16.7 17.1
granite-350m 137.8 1066.7 +674 % 48.8 47.9
LFM2.5-230M (control) 2006 2115 +5 % 107.6 107.3

Greedy 24-token generation is byte-identical between the two builds.

Evidence: GGML_SCHED_DEBUG=2 shows only the q4_0 embedding gathers on CPU;
the per-kernel profile (cl_profiling.csv) attributes 4.4 s of 5.6 s to
kernel_mul_mm_q4_0_f32_l4_lm at ~10.8 ms per small matmul.

Alternative fix: repair kernel_mul_mm_q4_0_f32_l4_lm itself (it is ~1000x
slower than a straightforward tiled-GEMM estimate); this PR takes the
low-risk route of using the already-fast path.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES — the analysis, fix and on-device verification were
    developed with AI assistance (opencode). I understand the change fully and
    will maintain it.

use_adreno_kernels() requires ne0 >= 512 && ne1 >= 512 on E031 >= 38.11
compilers, so q4_0 weights with ne1 < 512 (K/V projections, per-layer
embedding gates/projections, low-rank adapters) fall back to
kernel_mul_mm_q4_0_f32_l4_lm, which runs at ~5-11 GFLOPS vs ~300-500
GFLOPS for the trans-weight GEMM on the same shapes.

Lower the threshold to 128 (the value older compilers already use).

Verified on Adreno 740 (E031.41.03.64), llama-bench pp256/tg64 r2:
  gemma-4-E2B   pp 45.9 -> 205.3 t/s (+348%)
  gemma-4-E4B   pp 29.2 -> 109.5 (+275%)
  gemma-3-1b    pp 105.9 -> 497.0 (+369%)
  granite-350m  pp 137.8 -> 1066.7 (+674%)
  LFM2.5-230M   pp 2006 -> 2115 (+5%), tg neutral
Greedy output is byte-identical between builds.
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning OpenCL Issues specific to the OpenCL backend labels Oct 16, 2026
@ggml-gh-bot

ggml-gh-bot Bot commented Oct 16, 2026

Copy link
Copy Markdown

Hi @EdgeBites, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

  • AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.


Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

@EdgeBites
EdgeBites marked this pull request as ready for review October 16, 2026 05:20
@EdgeBites
EdgeBites requested a review from a team as a code owner October 16, 2026 05:20

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning OpenCL Issues specific to the OpenCL backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant