Pre-built llama.cpp containers for running LLMs with Vulkan or ROCm acceleration on AMD Ryzen AI Max “Strix Halo” integrated GPUs (gfx1151).

AI Toolbox Cockpit is the preferred way to install, launch, and update these toolboxes. It provides tested, pre-configured profiles; supports Toolbx and Distrobox; and can run supported servers directly with Podman or Docker, so Toolbx is not required.

pipx install git+https://github.com/kyuz0/ai-toolbox-cockpit.git
ai-toolbox-cockpit

The repository's refresh-toolboxes.sh remains available for manual Toolbx refreshes. The Cockpit is recommended for normal installation and updates.

For automatic Framework cooling and TuneD profile switching, see the GPU workload watcher. It detects llama.cpp, DS4, hipfire, vLLM, and the Halogen Flash API server, including containerized processes.

This is a hobby project maintained in my spare time. If you find these toolboxes and tutorials useful, you can buy me a coffee to support the work! ☕

• Supported Toolboxes
• Temporary llama.cpp ROCm inference workaround
• Manual container setup
• Legacy Benchmark Archive
• Memory Planning and VRAM Estimator
• Building Locally
• Distributed Inference
• More Documentation
• References

You can check the containers on DockerHub: kyuz0/amd-strix-halo-toolboxes.

These are stable, tested containers that are automatically rebuilt whenever the llama.cpp master branch is updated.

These use nightly or custom backend stacks. Their rebuild policy is noted below.

The Maya recipe builds the native GLM and DeepSeek engines for gfx1151. The dedicated release poller follows the latest stable upstream release. Every build resolves and verifies an immutable source commit, checks CLI startup and dependencies, and publishes a timestamped image plus the experimental rocm-10.0-maya channel. maya-version prints the source and runtime provenance. CI startup checks do not replace GPU model qualification.

In AI Toolbox Cockpit, select AMD Strix Halo → Maya. In Models, select GLM-5.3-Flash Maya-S24 (94.7 GB, MTP included) or DeepSeek-V4.1-Flash-1017 Maya-S (339.5 GB plus an 8 GB DSpark sidecar), download the pinned files, then select Prepare for Maya. Server Mode offers target-only generation or the model's native speculative mode, with one sequence and up to 68,096 context tokens. DeepSeek reads its Engram tables and nonresident experts from NVMe; the disk footprint is larger than the required physical memory.

For manual use, maya-prepare --help and maya-serve --help describe the same preparation and launch options. Mount the model directory at the same absolute path for both commands: generated packs refer to the original weights in place. Use the standard AMD /dev/kfd and /dev/dri device mounts, retained Podman groups and host IPC. The image has a normal shell entry and supports Toolbx/Distrobox installation.

toolboxes/Dockerfile.rocm-10.0-gufo builds the standalone Gufo engine from upstream main against the gfx1151 ROCm 10.0 package set. The poll-gufo workflow checks upstream every four hours and rebuilds when its commit differs from the published image revision. Recipe and build-workflow changes also trigger a rebuild. Each build uses an exact upstream commit, records it in the image and binary, and checks version/help before publication. Manual builds select rocm-10.0-gufo; the optional gufo_commit input selects an exact upstream main commit. The Dockerfile defaults to reviewed upstream commit b3facce1180e4bb44ac25c8f8a0fee348ae36da9 for local builds. The published experimental channel is docker.io/kyuz0/amd-strix-halo-toolboxes:rocm-10.0-gufo.

podman pull docker.io/kyuz0/amd-strix-halo-toolboxes:rocm-10.0-gufo
podman run --rm -it --name gufo-server \
  --userns keep-id --security-opt label=disable \
  --device /dev/kfd --device /dev/dri --group-add keep-groups --ipc=host \
  -p 127.0.0.1:18080:18080 \
  -v /absolute/path/to/Qwen3.8-Flash-Next-GGUF:/models:ro \
  docker.io/kyuz0/amd-strix-halo-toolboxes:rocm-10.0-gufo \
  gufo serve --host 0.0.0.0 --port 18080 --sessions 1 llm \
  --model /models/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  --context 133760 --max-tokens 8192 \
  --served-model-name Qwen3.8-Flash-Next-UD-Q4_K_XL

Add --log-progress to show prefill, decode, speed, and speculative-acceptance progress in the server output. Point --model at the first shard; Gufo loads the remaining numbered shards. Add --speculative mtp --mtp-model /models/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf for the tested Flash Next MTP path. DeepSeek uses its own --dspark-model sidecar flag. Keep these auxiliary GGUFs separate from the target weights.

Legacy images (rocm-6.4.2, rocm-6.4.3, rocm-7.1.1) are excluded from these lists.

The images include RDMA support for llama.cpp RPC. On Linux hosts with a RoCEv2-capable NIC, llama.cpp automatically negotiates the RDMA transport when available and otherwise falls back to TCP.

On Toolbx hosts, refresh-toolboxes.sh detects /dev/infiniband and adds the required device, rdma group, and unlimited memlock options automatically. For manual Toolbx creation, append:

--device /dev/infiniband --group-add rdma --ulimit memlock=-1

Use these commands only if you prefer to manage the container yourself. For the guided, tested path across Toolbx, Distrobox, Podman, and Docker, use AI Toolbox Cockpit.

The examples below use Toolbx. See the central Strix Halo setup guide for host preparation and other supported container engines.

Option A: Vulkan (RADV/AMDVLK) - best for compatibility

toolbox create llama-vulkan-radv \
  --image docker.io/kyuz0/amd-strix-halo-toolboxes:vulkan-radv \
  -- --device /dev/dri --group-add video --security-opt seccomp=unconfined

toolbox enter llama-vulkan-radv

Option B: ROCm (Recommended for Performance)

toolbox create llama-rocm-10.0 \
  --image docker.io/kyuz0/amd-strix-halo-toolboxes:rocm-10.0 \
  -- --device /dev/dri --device /dev/kfd --group-add video --group-add render --group-add sudo \
  --security-opt seccomp=unconfined

toolbox enter llama-rocm-10.0

Inside the toolbox:

llama-cli --list-devices

Example: Qwen3 Coder 30B (BF16) Consider: setting your Hugging Face HF_TOKEN for faster downloads

HF_XET_HIGH_PERFORMANCE=1 hf download unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF \
  BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
  --local-dir models/qwen3-coder-30B-A3B/

HF_XET_HIGH_PERFORMANCE=1 hf download unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF \
  BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00002-of-00002.gguf \
  --local-dir models/qwen3-coder-30B-A3B/

⚠️ IMPORTANT: Always use flash attention (-fa 1) and resident loading (--load-mode none) on Strix Halo to avoid crashes/slowdowns.

Server Mode (API):

llama-server -m models/qwen3-coder-30B-A3B/BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
  -c 8192 -ngl 999 -fa 1 --load-mode none

Router Mode:

Uses models.ini preset configuration for multi-model routing.

llama-server --models-preset models.ini --host 0.0.0.0 --port 8080 --models-max 1 --parallel 1

CLI Mode:

llama-cli --load-mode none -ngl 999 -fa 1 \
  -m models/qwen3-coder-30B-A3B/BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
  -p "Write a Strix Halo toolkit haiku."

AI Toolbox Cockpit is the recommended update path. If you created the Toolbx container manually, the repository script can refresh it to the latest nightly or stable build:

./refresh-toolboxes.sh all

The benchmark results in this repository are legacy and are no longer updated. For recent benchmarks, improved methodology, and interactive benchmark curves, visit local-llm-benchmarks.dev.

The previous results remain available as an archive in the interactive viewer and its toolbox comparison.

Historical benchmark logs are retained in the repository.

Strix Halo uses unified memory. To estimate VRAM requirements for models (including context overhead), use the included tool:

gguf-vram-estimator.py models/my-model.gguf --contexts 32768

See docs/vram-estimator.md for details.

You can build the containers yourself to customize packages or llama.cpp versions. Instructions: docs/building.md.

Run models across a cluster of Strix Halo machines using run_distributed_llama.py.

1. Setup SSH keys between nodes.
2. Run python3 run_distributed_llama.py on the main node.
3. Follow the TUI to launch the cluster.

• Legacy benchmark archive
• docs/vram-estimator.md
• docs/building.md
• docs/troubleshooting-firmware.md

• Strix Halo Home Lab (deseven)
• Strix Halo Testing Builds (lhl)
• AMD ROCm 7.14 installation guide