Skip to content

Enhance CI: add an opt-in @claude assistant limited to the trusted CI tools - #3739

Draft
roaffix wants to merge 14 commits into
feature/ci-reworkfrom
feature/claude-integration
Draft

roaffix wants to merge 14 commits into
feature/ci-reworkfrom
feature/claude-integration

Conversation

@roaffix

@roaffix roaffix commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Draft, stacked on #3738 : please review and merge that one first. This adds two commits on top of it, and does nothing until an admin enables it.

This adds an opt-in @claude assistant: Anthropic's Claude Code GitHub Action, running as an ArrayFire-owned GitHub App. Maintainers can ask it, in an issue or a pull request, to explain or review code, or to write a change and push it to a branch. It builds and tests only through the trusted CI scripts, and can request CI only with /cpu-tests or /gpu-tests.

Description

Type: a new, optional CI feature. No library code changes. The workflow is skipped unless the repository variable AF_CI_CLAUDE is true.

What it does:

  • claude.yml answers @claude from people with write access, in issues, pull requests and review comments.
    • It runs as the organisation's GitHub App, through anthropics/claude-code-action, pinned to a commit SHA.
    • In an issue it works on a branch, and replies with a summary and a link to open the pull request.
  • It builds and tests only with the trusted CI. It runs master's own ci.sh --docker <job> from a second checkout. That builds a copy of its working tree, without .git, in a container that gets no token.
  • It requests CI only through a wrapper: request-tests.sh can post nothing but /cpu-tests or /gpu-tests on the current pull request.
  • The request rules gain a bot:
    • its /cpu-tests run the Linux jobs;
    • its /gpu-tests must pass extra rules: the light checks and the CPU tests are green on the head, the pull request is by someone with write access (or by the bot), and it stays under 2 requests per pull request and 6 per day (configurable);
    • a maintainer still approves every GPU run.
  • The model: claude-haiku-5-5 by default. A comment can pick another from an allow-list, e.g. @claude [sonnet] ….
  • Its commits keep Claude's Co-Authored-By attribution.

What it can't do:

  • answer people without write access;
  • edit anything under .github/;
  • run builds any other way than through the trusted ci.sh;
  • read the process environment, /proc, the runner's temporary directory, git's config or remote URLs, or home-directory dotfiles (its tool rules deny them);
  • post arbitrary comments;
  • merge, approve, or change settings.

Why: it was shown to the maintainers in a test copy of the repository. It answered in issues and PRs and pushed a change. When its own change failed lint, it reproduced the failure with the trusted ci.sh, fixed it, re-checked it and pushed, in about a minute. It then requested the CPU tests itself.

Impact on hardware, software and backends: none until enabled. When enabled:

  • each @claude runs one job on a standard runner (free for public repositories), up to 90 minutes and 40 turns;
  • model tokens are billed by Anthropic: in the test copy about $0.01–0.02 per task with Haiku, and $0.002 for a short answer. Larger models cost more and can be limited with AF_CI_CLAUDE_MODELS;
  • GPU minutes only through the bot's limited, approved requests.

New functions: none. Backporting: not applicable.

Related:

Known follow-ups, to fix before this leaves draft:

  • app-id is deprecated: actions/create-github-app-token's app-id input should become client-id, once the real App exists.
  • Private organisation members can't use it yet. GitHub reports private members as CONTRIBUTOR in author_association, which the workflow's first filter rejects. The fix is to gate on the API's permission check, as the request workflow already does. Until then, maintainers can make their membership public.
  • One wording fix: the assistant's rules still mention oneAPI changes as a reason to request GPU tests; such a request is answered with "nothing to test" and costs nothing.

Changes to Users

  • Library users: nothing changes.
  • Maintainers, once it's enabled:
    • write @claude … in an issue or pull request; for example, "@claude the lint check fails on this PR, please fix it";
    • choose a model with @claude [sonnet] …, from the allowed list;
    • its pull requests go through the same light checks, /cpu-tests, /gpu-tests approval and merge queue as anyone's.
  • Admins: the setup below.

Checklist

  • Rebased on latest master: on top of #XXXX, which is on the latest master
  • Code compiles: no library change; the workflow checks (actionlint, shellcheck, the CI scripts' tests) pass
  • Tests pass: the model choice (24 cases), the comment wrapper (20) and the request rules with the bot cases
  • Functions added to unified API: not applicable
  • Functions documented: not applicable. The setup is described below and in the workflow's header
Setup (admins)
  1. A GitHub App owned by the organisation (not a person):

    • Permissions: Contents read/write, Issues read/write, Pull requests read/write, Actions read, Checks read.
    • Install it on arrayfire/arrayfire only.
  2. A model credential: an Anthropic API key in the organisation's account. Alternatively, OIDC to Amazon Bedrock or Google Vertex AI, which avoids a long-lived key and needs a small change to the action's inputs.

  3. An environment claude, with deployments from master only and the secrets ANTHROPIC_API_KEY, AF_CLAUDE_APP_ID and AF_CLAUDE_APP_PRIVATE_KEY.

  4. Repository variables:

    Variable Value
    AF_CI_CLAUDE true (the switch)
    AF_CI_CLAUDE_BOT the App's bot login, e.g. arrayfire-claude[bot]
    AF_CI_CLAUDE_BOT_ID the bot's user ID
    AF_CI_CLAUDE_MODEL optional; the default model (claude-haiku-5-5)
    AF_CI_CLAUDE_MODELS optional; the models a comment may pick
    AF_CI_CLAUDE_EFFORT optional; the reasoning effort
    AF_CI_GPU_AI_PER_PR_DAY, AF_CI_GPU_AI_PER_DAY optional; the bot's GPU limits (2 and 6)
  5. Actions permissions: if the repository allows only selected actions, allow anthropics/claude-code-action. The other actions it uses are GitHub's own.

Testing evidence
  • In a test copy of the repository:
    • Claude answered issues and pull-request comments, and pushed changes.
    • Asked to fix a failing lint check, it reproduced the failure with the trusted ci.sh, fixed it and pushed.
    • Asked to "request the CPU tests", it ran the wrapper. The wrapper posted /cpu-tests as the App's bot, and the request workflow ran only the Linux jobs, since the bot isn't a maintainer.
  • Locally: three headless runs with the workflow's exact tool rules and only fake secrets. No secret value appeared in any output.
    • the wrapper was allowed;
    • gh pr comment was denied;
    • request-tests.sh cpu; cat /proc/self/environ was denied as a whole.
  • The test suites run in the workflows check.
How to switch it off
  • Unset AF_CI_CLAUDE: the workflow is skipped from the next comment on.
  • Uninstall the App, or delete the claude environment's secrets: that removes its access.
  • Unset AF_CI_CLAUDE_BOT: the bot's /cpu-tests and /gpu-tests comments are then ignored.
Commits
  1. Add an @claude workflow with ArrayFire's own tools
  2. Let the Claude App's bot request tests within limits

The Unix and Windows workflows downloaded their own CMake, Ninja and
Doxygen, ran partly on a retired runner image (Ubuntu 20.04), repeated
their build steps in YAML and reported to a CDash dashboard instead of
the pull request. They are removed here; their
jobs come back in the following commits as calls to one script. Doing
both in one commit keeps every workflow in the tree passing actionlint.

.github/scripts/ci.sh runs one CI job: it installs the job's
dependencies, then runs the job's CMake workflow preset or check, in
the job's pinned Ubuntu container (--docker) or on the macOS and
Windows hosts, so CI and a developer run the same command; --list
shows the jobs. With --docker it chooses the build parallelism from the
container's memory, keeps ccache in a volume or in a host directory CI
can cache, and builds either this checkout or another commit's tree
(AF_CI_SOURCE_DIR) with this repository's scripts and settings, also as
a copy without .git. It must run on macOS's bash 3.2.

install-deps-ubuntu.sh, install-deps-macos.sh and
install-deps-windows.ps1 install each job's packages: the compilers and
BLAS, the CUDA, OpenCL and oneAPI toolkits by version (with
nvJitLink's development package, which the installers' library bundling
looks up by name), the runtime libraries alone for GPU test containers,
and vcpkg at a pinned release (2026.10.16), whose MSYS2 helpers still
find their packages. lint.sh checks the formatting of the changed lines
with clang-format 15. checkout-commit.sh fetches one commit
anonymously. count-vcpkg-archives.sh tells whether a run added vcpkg
archives worth saving. .gitattributes keeps the shell scripts' LF
endings, which Git Bash on Windows runners needs.

The release source workflow builds the tarball with
make-source-tarball.sh and uploads it with gh release upload instead of
the archived upload-release-asset action; only the upload step can
write, and a manual run builds the tarball as an artifact without
publishing it. It no longer fetches the SuiteSparse matrices, which the
tarball never included.
- The non-MKL presets set AF_COMPUTE_LIBRARY to "FFTW/LAPCK/BLAS",
  which matches neither supported value, so configure skipped the
  FFTW, CBLAS and LAPACK checks.
- clFFT followed a moving branch, the only way the OpenCL build could
  change without a commit here. It is pinned to the commit its
  arrayfire-release branch points to, like every other OpenCL
  dependency.
- NUM_PROC_THREADS = 0 runs Doxygen's experimental parallel input
  processing on every core. With Doxygen 1.9.7 it intermittently
  corrupted the heap and aborted: 4 of 10 runs on a 24-core machine.
  On one thread every run passed, in the same time.
- The docs build writes docs/details/examples.dox into the source tree
  on every configure, which left the tree dirty; it is ignored.
The ci-* configure, build, test and workflow presets hold each CI
job's settings, so ci.sh runs "cmake --workflow --preset ci-<job>" and
a developer can run the same:

- CPU on Linux with GCC and Clang and OpenBLAS, with icx and oneMKL,
  and on the newest Ubuntu; macOS with Apple Clang and Accelerate,
  built with spdlog's bundled fmt (Homebrew's fmt 12 breaks it) and
  without the mean/var cases that FMA contraction breaks there; Windows
  with MSVC 2022 and vcpkg, at /W3 (/Wall flooded the log with
  thousands of system-header warnings) and with MSBuild's multi-tool
  task compiling in parallel;
- compile-only CUDA, OpenCL and oneAPI builds. CUDA builds for compute
  capabilities 7.5, 8.9 and 9.0, plus Blackwell's 10.0 and 12.0 through
  nvcc flags, because CUDA_architecture_build_targets goes through
  CMake's own FindCUDA copy of cuda_select_nvcc_arch_flags on this
  tree, which rejects two-digit majors; and a Windows CUDA build;
- the documentation.

The test presets write JUnit, cap the BLAS threads at four and fail
when no test runs. The Matrix Market tests are on, with FetchContent
pointed at data that ci.sh restores into the build tree (added in a
following commit).

The Visual Studio 2019 presets become 2022 ones, since no GitHub runner
has 2019 any more, and the file moves to format 6 (CMake 3.25) for the
workflow presets.
checks.yml runs on every push to a pull request or master and in the
merge queue. It is cheap, so the CPU builds can run on a schedule of
their own:

- lint: clang-format 15 on the lines changed since the base. The tree
  has unformatted files, and library code isn't reformatted here.
- docs: the Doxygen build.
- workflows: actionlint (which also runs shellcheck on the workflows'
  run: steps), shellcheck on the CI scripts, the CI scripts' Python
  test suites, and check-bash32.sh.

check-bash32.sh runs the scripts that the macOS job runs on the host
under bash 3.2, the macOS runner's bash, in a bash:3.2 image pinned by
digest and without network: their syntax, ci.sh --list, a short SHA
through checkout-commit.sh, and a grep for bash-4-only syntax. Nothing
else caught bash 4 syntax until the macOS job failed.
vcpkg's current blas and lapack ports collide with the lapack-reference
overlay port: the overlay's lapack.pc, its blas.pc links for the
noblas feature and its wrapper in share/lapack belong to them now, and
the Windows build stopped at the file conflict. ArrayFire's own CI
pull request, the fix/restore-ci branch, fixes the overlay, and its
Windows run passes with it. Its portfile and triplet are taken as they
are, so the two branches don't conflict:

- the overlay installs lapack-reference.pc and its wrapper in
  share/lapack-reference, and no blas.pc links;
- vcpkg_cmake_config_fixup uses PACKAGE_NAME ${PORT};
- the triplet sets VCPKG_PROVIDED_FORTRAN, since vcpkg 2026.07.29 and
  later install MinGW gfortran for the port only when the triplet asks
  for it, as vcpkg's own x64-windows triplet does.
The sparse tests' matrices come from the SuiteSparse collection, which
refuses CI's downloads (HTTP 403). The fix/restore-ci branch bundles
the nine matrices with a SHA-256 manifest; the bundle and the manifest
are taken from it as they are, and its check_matrix_data.py is adapted
as check-matrix-data.py.

Before every build, ci.sh runs check-matrix-data.py, which checks the
bundle and each matrix against the manifest and the tested tree's list
of matrices, fails naming every file that differs, and restores the
matrices into the build tree. The tests download each matrix through
af_dep_check_and_populate, whose FetchContent_Populate takes
FETCHCONTENT_SOURCE_DIR_<name> as the source when it is set; the ci
preset and matrixmarket.cmake (for the GPU builds and the CMake 3.16
build) set those to the restored matrices, so nothing is downloaded and
nothing under test/ changes. The GPU build archives carry the matrices
to the test jobs.
check-cuda-release.sh configures the CUDA backend as users and the
installers get it: ArrayFire's default architecture list and
AF_INSTALL_STANDALONE=ON. It prints the architecture flags and
compiles one CUDA object with them, which catches a default list with
an architecture the toolkit dropped (with CUDA 13, this tree's list
still has compute_53). It installs the cuda_dependencies component and
checks that every library the installers bundle on Linux is there,
under the toolkit's soname, the name libafcuda loads it by (with CUDA
13, this tree bundles cuFFT, cuSOLVER and cuSPARSE under the wrong
major). It needs only a configure and one compile.

install-cuda-windows.ps1 installs a CUDA toolkit on Windows with
NVIDIA's network installer, pinned by URL and SHA-256 per version
(12.9, 13.1, 13.4), with only the subpackages the build needs. From
CUDA 13.0 on, nvcc no longer includes crt (crt/host_config.h) and nvvm
(cicc): NVIDIA's redistrib manifests list them apart and the
installers carry crt_<version> and nvvm_<version>, though the
installation guides list them only from 13.2 on. msvc-env-windows.ps1
sets up Visual Studio's x64 developer environment for the Ninja
generator.
gpu.yml tests one commit on GPU runners. The scripts and settings come
from the ref it runs on, and only the source tree from the commit.
Each leg, a CUDA version from AF_CI_CUDA_VERSIONS or OpenCL, is built
on a free standard runner into an archive of the test programs,
libraries and test data, then tested on a GPU runner (AF_CI_GPU_RUNNER)
in a container with the runtime libraries alone, which installs in
about a third of the time of the toolkit. One approve job holds the
gpu-tests environment, so one reviewer's approval covers every leg, and
it starts with the builds. A failed build fails only its own leg; a
summary job reports every leg's passed and failed tests.

The CUDA test job compares the driver with the toolkit (cuda-compat.sh)
and, when the driver is older, installs NVIDIA's forward-compatibility
package for data-center GPUs such as the T4: ArrayFire's JIT has the
driver load PTX from NVRTC, which minor-version compatibility doesn't
cover. The OpenCL test job registers NVIDIA's ICD when the container
toolkit doesn't, prints the platforms and fails clearly without an
NVIDIA one. linux-cuda-gpu.cmake and linux-opencl-gpu.cmake hold the
builds' trusted settings for any commit.

The workflow is off unless AF_CI_GPU_TESTS is true.
cpu.yml runs the CPU builds and tests on Linux (GCC and Clang, in
pinned containers), macOS and Windows, in the merge queue and on every
push to master, and for a given commit when started by hand or by
another workflow; such a commit is built on its own with the trusted
scripts and presets. Actions are pinned to commit SHAs. Only master's
own builds save the ccache and the vcpkg cache, because a build of
another commit could plant entries in them; the vcpkg cache is saved
whenever a run added archives, even if the build failed later. The
macOS and Windows jobs build outside a container, so a request runs
them only when its caller asks for them.

With AF_CI_ACCELERATOR_BUILDS set to true, pushes and requests also
compile the CUDA (once per AF_CI_CUDA_VERSIONS version, with Blackwell
targets and the release checks), OpenCL and oneAPI backends, and the
CUDA backend on Windows. The matrix job has a fixed name: GitHub never
expands a skipped matrix job's name, which showed up as a check named
"matrix.job".

cpu-placeholders.yml gives a pull request instant checks with the same
names, so that it can join a merge queue that requires them. The
nightly runs the icx and oneMKL job, the job on the minimum CMake
version and the job on the newest Ubuntu, which may fail without
failing the run.

A merge-queue entry or a push to master that only adds or changes
non-code files (code-changes.py: Markdown at the top level, docs/, the
issue and pull-request templates, GitHub's citation files, .gitignore,
.mailmap) builds nothing. The changes job lists the changed files with
the compare API and the read-only token; a merge-queue entry is
compared with the branch it merges into, since the group's parent may
be the entry ahead. None of those files is read by a CPU build, an
installer rule or the release tarball. A removed or renamed file runs
the tests in full, since CPack checks at every Linux configure that
README.md exists; so does any other file and every request. The CPU
jobs still run and skip their steps, so the required checks keep their
names (a skipped matrix job's name isn't expanded) and pass in seconds,
each saying why.
A /cpu-tests or /gpu-tests comment on a pull request asks for the
tests of its current head. test-requests.yml runs from the default
branch, so the pull request can't change it, and nothing from the pull
request is checked out or run in it. test-requests.py decides, pins
the head SHA, and cpu.yml or gpu.yml test exactly that commit; the
result goes to the cpu-tests or gpu-tests commit status and a comment.

- /cpu-tests (all adds the accelerator builds): maintainers with write
  access on any pull request; the pull request's author on their own,
  once the light checks have run on it, on Linux only.
- /gpu-tests [cuda|opencl|all] [cuda=X.Y]: maintainers with write
  access only. Without a backend the changed files choose, and a
  change that can't affect a GPU backend gets a reply and starts
  nothing; CUDA versions come from AF_CI_CUDA_VERSIONS (12.9 when
  unset). Every reply says what runs and why.
- An accepted reply says when the branch is behind its base, since CI
  fixes made there since aren't in the test.

Failed API calls become annotations of the run, readable without
logging in; a failed comment after the status is set is only a
warning. The tests run the decisions on fixtures, the commands with
failing writes, and the CUDA driver check.
The "Running CI" section says what runs when, which checks the light
checks are, how to request the CPU and GPU tests and what they cost,
and how to run any CI job locally with ci.sh.
The workflows pin every action to a commit SHA; Dependabot proposes the
updates weekly.
claude.yml answers "@claude" in issues, pull requests and review
comments as ArrayFire's own GitHub App, with only the tools it lists.
It is off unless the repository variable AF_CI_CLAUDE is true, and
reacts only to people with write access. Claude edits, commits and
pushes in the working tree; it builds and tests only with the trusted
ci.sh from a second checkout of the default branch, which builds a copy
of the working tree without .git in a container that gets no token.
Its file and command rules keep its tools away from the process
environment, /proc, the runner's temporary directory, git's config and
the dotfiles in the home directory.

claude-model.py chooses the model (a default, a list a request may
pick from, an effort) from repository variables. request-tests.sh is
the only way Claude can post: it builds a /cpu-tests or /gpu-tests
comment itself from a fixed set of arguments and posts it on the event's
pull request. Both have tests.
test-requests.py accepts requests from the App's bot
(AF_CI_CLAUDE_BOT): /cpu-tests on Linux, and /gpu-tests only when the
light checks and the workflow's own cpu-tests status are green on the
head, within AF_CI_GPU_AI_PER_PR_DAY (2) requests per pull request and
AF_CI_GPU_AI_PER_DAY (6) per day, and only on pull requests by people
with write access or by the bot itself. A request with several GPU legs
counts once. The decide job may read commit statuses and workflow runs
for these checks. CONTRIBUTING.md mentions the bot.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant