Calling af::convolve concurrently from multiple host threads on the CUDA
backend silently returns results computed with a filter submitted by a
different thread; no error is raised.

Description

For filters up to MAX_CONV2_FILTER_LEN (17), conv2 stages the filter
through the module-global __constant__ char cFilter[...] declared in
src/backend/cuda/kernel/convolve2.cuh:13. In
src/backend/cuda/kernel/convolve.hpp (conv2Helper, ~line 159) the code
obtains the kernel module from common::getKernel(...) — which is cached
per device, so all host threads share the same cFilter symbol — and then
does, with no lock making the copy + launch pair atomic:

auto constMemPtr = convolve2.getDevPtr(conv_c_name);
convolve2.copyToReadOnly(constMemPtr, reinterpret_cast<CUdeviceptr>(fptr),
                         f0 * f1 * sizeof(aT));
EnqueueArgs qArgs(p.mBlocks, p.mThreads, getActiveStream());
convolve2(qArgs, ...);

copyToReadOnly is a cuMemcpyDtoDAsync on the per-device stream
(src/backend/cuda/Kernel.cpp:24). With two host threads enqueueing into
the same stream, the order can become copy(filterA), copy(filterB), launch(A), launch(B) — thread A's convolution then reads thread B's
filter. The 1d/3d and separable paths (sFilter) follow the same pattern.

Answers to the template questions:

• ArrayFire was installed from the official installer (Ubuntu package
  arrayfire-unified3 / libaf.so.3.9.0), not self-built.
• Backend: CUDA. (CPU backend not affected as far as we can tell; the
  staging pattern above is CUDA-specific. OpenCL not tested.)
• Workaround: yes — serializing all af::convolve calls with a
  process-wide mutex makes multi-threaded results bit-identical to
  single-threaded runs. That is what we ship now.
• Reproducibility: reliable — every run of the program below produces
  wrong results on every thread (4–8% of iterations on our system).
• Expected: concurrent convolve calls return the same values as
  single-threaded calls (the multi-threading documentation presents
  ArrayFire as thread safe).
• AF_TRACE=all AF_PRINT_ERRORS=1: the bug still reproduces and the trace
  contains no error lines at all — the corruption is completely
  silent. (Trace of the run available on request; it is just ~2.5k lines
  of normal jit/mem/platform traces.)
• We hit this in production (an image pipeline where two service threads
  convolve different filter banks): ~600 of 351k output elements differed
  from serial runs, silently.
• Additional observation: a variant of the reproduction that also runs the
  verification arithmetic (abs/max/JIT subtraction) concurrently with
  the convolves segfaults instead of producing wrong results; we did not
  investigate whether that is the same underlying issue.

Reproducible Code and/or Steps

// g++ -O2 -std=c++14 convolve_race.cpp -laf -lpthread -o convolve_race
#include <arrayfire.h>
#include <cstdio>
#include <thread>
#include <vector>

int main() {
    af::info();
    const int numThreads = 4;
    const int iters = 100;
    af::setSeed(42);
    af::array image = af::randu(512, 512, 3);

    std::vector<af::array> filters, refs;
    for (int t = 0; t < numThreads; ++t) {
        filters.push_back(af::randu(15, 15) - 0.5f);
        refs.push_back(af::convolve(image, filters[t]));  // single-threaded reference
        refs[t].eval();
    }
    af::sync();

    std::vector<std::vector<af::array>> results(numThreads);
    auto worker = [&](int t) {
        af::setDevice(0);
        results[t].reserve(iters);
        for (int i = 0; i < iters; ++i) {
            af::array r = af::convolve(image, filters[t]);  // concurrent
            r.eval();
            results[t].push_back(r);
        }
    };
    std::vector<std::thread> threads;
    for (int t = 0; t < numThreads; ++t) threads.emplace_back(worker, t);
    for (auto& th : threads) th.join();
    af::sync();

    int totalWrong = 0;
    for (int t = 0; t < numThreads; ++t) {
        int wrong = 0;
        for (const af::array& r : results[t]) {
            if (af::max<float>(af::abs(r - refs[t])) > 1e-6f) ++wrong;
        }
        std::printf("thread %d: %d of %d convolve results are wrong\n", t, wrong, iters);
        totalWrong += wrong;
    }
    return totalWrong > 0 ? 1 : 0;
}

Output of three consecutive runs (expected: 0 everywhere):

thread 0: 6 of 100 convolve results are wrong
thread 1: 7 of 100 convolve results are wrong
thread 2: 6 of 100 convolve results are wrong
thread 3: 8 of 100 convolve results are wrong

thread 0: 7 of 100 | thread 1: 5 of 100 | thread 2: 4 of 100 | thread 3: 4 of 100

thread 0: 3 of 100 | thread 1: 3 of 100 | thread 2: 4 of 100 | thread 3: 6 of 100

System Information

1. ArrayFire version: v3.9.0 (official package, arrayfire-unified3)
2. Devices: 1x NVIDIA GeForce RTX 5070 Ti
3. af::info():

ArrayFire v3.9.0 (CUDA, 64-bit Linux, build b59a1ae53)
Platform: CUDA Runtime 12.2, Driver: for
[0] NVIDIA GeForce RTX 5070 Ti, 15851 MB, CUDA Compute 12.0

4. Script output:

Distributor ID: Ubuntu
Description:    Ubuntu 24.04.4 LTS
Release:        24.04
Codename:       noble
name, memory.total [MiB], driver_version
NVIDIA GeForce RTX 5070 Ti, 16303 MiB, 570.211.01
rocm-smi not found.
clinfo: Platform Name NVIDIA CUDA, Platform Version OpenCL 3.0 CUDA 12.8.99

Checklist

• Using the latest available ArrayFire release
• GPU drivers are up to date