mysqlnd: in ZTS builds every statistics update locks a process-wide mutex #24447

Cause

In ZTS builds, MYSQLND_STATS_UPDATE_VALUE takes a mutex around every counter update (mysqlnd_statistics.h L19–L32):

#define MYSQLND_STATS_UPDATE_VALUE(stats, statistic, value) \
	{ \
			MYSQLND_STATS_LOCK(stats); \
			(stats)->values[(statistic)] += (value); \
			MYSQLND_STATS_UNLOCK(_p_s); \
	}

The MYSQLND_INC_CONN_STATISTIC* macros update the process-wide mysqlnd_global_stats and the connection's own statistics (mysqlnd.h L262–L276). The per-connection mutex is uncontended (one connection per thread); the global one is shared by all threads of the process and taken on every counted packet and row. Statistics are collected by default (mysqlnd.collect_statistics defaults to 1, php_mysqlnd.c L151, and is On in php.ini-production L1207). In NTS builds the lock macros are empty. The code is the same on PHP-8.4, PHP-8.5 and master.

Measurements

AMD Ryzen 9 5950X, Ubuntu 26.04, governor performance; PHP built with the compiler flags of the official Docker image (-O2 -fstack-protector-strong -fpic -fpie), FrankenPHP 1.13.1 built against it, worker mode with 16 PHP threads, PDO MySQL with its default emulated prepares, MySQL 8.4.11 on the same host. The figure is the CPU time of the FrankenPHP process or container per request, median of alternating passes; the cost is statistics on (the default) against mysqlnd.collect_statistics=0.

A JSON API (a token lookup, then a page of 20 posts with authors, tags and comment counts), PHP 8.5.11, FrankenPHP on 4 cores / 8 threads, 3 passes (data):

The workload of the reproduction below on master (b7256ee), with FrankenPHP's 16 threads on 4 or 8 cores, on one CCD or split over both, 10–15 passes (report):

In separate runs in a KVM guest (FrankenPHP on 8 of 32 vCPUs of an AMD EPYC 9754, the JSON API, data not published), the CPU per request with statistics on was 2.2, 2.4 and 4.2 times that with statistics off at 50 clients, and 1.24 times with netem raising the round trip on loopback to 0.56 ms. PHP-FPM (NTS) with the same queries shows no such cost, since the lock macros are empty there. A profile of FrankenPHP in that VM had lock functions at the top of more than a quarter of the samples (__pv_queued_spin_lock_slowpath 18%, pthread_mutex_lock and pthread_mutex_unlock 9%).

Lock count

In the reproduction setup below (16 PHP threads, one prepared SELECT of 20 rows per request), a uprobe on pthread_mutex_lock in the FrankenPHP process (bpftrace 0.25, two passes of 10 seconds each, statistics on) counts 722.74 calls per request: 371.38 on one mutex shared by all threads (the global statistics) and 351.36 spread over 16 others, one per thread (the connection statistics). With collect_statistics=0 the process makes no pthread_mutex_lock calls at all. The address with the most futex calls is the shared mutex (34–38 calls per request); none of the 16 per-thread mutexes is among the 20 addresses with the most futex calls. Tracing slows the server to about 4,400 req/s (from about 22,000 without it), so the futex counts show where the contention is, not how much of it there is without tracing.

uprobe:/proc/PID/root/usr/lib/x86_64-linux-gnu/libc.so.6:pthread_mutex_lock /pid == PID/ { @locks = count(); @by_mutex[arg0] = count(); }
tracepoint:syscalls:sys_enter_futex /pid == PID/ { @futex = count(); @futex_by_addr[args.uaddr] = count(); }

Proposed change

Relaxed atomic additions instead of the mutex under ZTS, keeping the statistics. The mutex protects only the increments: mysqlnd_fill_stats_hash() already reads the counters without it, there is no reset, and the macros that update two or three counters lock once per counter, so readers never got a consistent snapshot and relaxed atomics do not change that.

PR #24448 does this for master: atomic_uint_least64_t counters under ZTS, updated with atomic_fetch_add_explicit(..., memory_order_relaxed), read with atomic_load_explicit() and initialized with atomic_init(); LOCK_access removed; NTS unchanged (the same machine code for the 455 mysqlnd functions). With it, statistics on cost 0.6–1.4% over statistics off in the four placements above, and a prototype of the same change on PHP 8.5.11 costs 0.8–2.8% with the JSON API. The counters stay exact under concurrency, and the ext/mysqlnd, ext/mysqli and ext/pdo_mysql tests give the same result as on master. Every pass is in the report, the background in a write-up.

Until then, mysqlnd.collect_statistics=0 removes the cost for threaded deployments that do not need mysqli_get_client_stats(), mysqli_get_connection_stats() or the mysqlnd section of phpinfo().

Reproduction

A minimal setup with the official FrankenPHP image (dunglas/frankenphp:1.13-php8.5.11, digest below; ZTS) and MySQL 8.4.11: one prepared SELECT returning 20 rows of a 1000-row table per request, wrk -t4 -c50 -d20s, CPU time of the FrankenPHP container from its cgroup cpu.stat before and after wrk. It needs Linux with cgroup v2 and Docker's systemd cgroup driver, wrk, taskset, curl, and free ports 3307 and 8080.

AMD Ryzen 9 5950X; FrankenPHP on 4 cores with their SMT siblings (CPUs 8–11 and 24–27, one CCD), wrk on CPUs 6–7 and 22–23, MySQL on 0–5 and 16–21:

With 16 threads, statistics on cost about 10% more CPU per request in every pass. With 8 threads, one per CPU, the passes fall into two states, so the cost per pass ranges from about −1% to 23%.

APP_CPUS=8-11,24-27 LOAD_CPUS=6-7,22-23 DB_CPUS=0-5,16-21 THREADS=16 ./run.sh

FROM dunglas/frankenphp:1.13-php8.5.11@sha256:2a9c6663f73ad1e401909634206bd25e13bed5b94d86eefa3b5ec64c985b4595
RUN install-php-extensions pdo_mysql
COPY index.php /app/index.php

{
	auto_https off
	frankenphp {
		num_threads THREADS
		worker {
			file /app/index.php
			num THREADS
		}
	}
}

:8080 {
	root * /app
	php_server
}

<?php
$pdo = new PDO('mysql:host=127.0.0.1;port=3307;dbname=test', 'root', 'pw');
$handler = static function () use ($pdo): void {
    $st = $pdo->prepare('SELECT id, title, body, created_at FROM t ORDER BY id LIMIT 20 OFFSET ?');
    $st->bindValue(1, random_int(0, 980), PDO::PARAM_INT);
    $st->execute();
    header('Content-Type: application/json');
    echo json_encode($st->fetchAll(PDO::FETCH_ASSOC));
};
while (frankenphp_handle_request($handler)) {
}

CREATE TABLE t (id INT PRIMARY KEY, title VARCHAR(255) NOT NULL, body TEXT NOT NULL, created_at DATETIME NOT NULL);
INSERT INTO t
WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < 1000)
SELECT i, CONCAT('Post ', i), REPEAT('Lorem ipsum dolor sit amet. ', 20), '2026-01-01 00:00:00' + INTERVAL i MINUTE FROM n;

#!/bin/bash
# CPU time of the FrankenPHP container per request, with mysqlnd statistics on (default) and off.
#   APP_CPUS=8-11,24-27 LOAD_CPUS=6-7,22-23 DB_CPUS=0-5,16-21 [THREADS=16] ./run.sh
# Needs Linux with cgroup v2 and the systemd cgroup driver of Docker, wrk, taskset, curl, free ports 3307 and 8080.
set -e
cd "$(dirname "$0")"
: "${APP_CPUS:?}" "${LOAD_CPUS:?}" "${DB_CPUS:?}"
docker rm -f repro-mysql repro-app >/dev/null 2>&1 || true
docker build -q -t mysqlnd-zts-repro . >/dev/null
sed "s/THREADS/${THREADS:-16}/g" Caddyfile > Caddyfile.run
docker run -d --name repro-mysql --network host --cpuset-cpus "$DB_CPUS" -e MYSQL_ROOT_PASSWORD=pw -e MYSQL_DATABASE=test \
	-v "$PWD/init.sql:/docker-entrypoint-initdb.d/init.sql:ro" mysql:8.4.11 --port=3307 --mysqlx=OFF >/dev/null
timeout 120 bash -c "until docker exec repro-mysql mysql -h127.0.0.1 -P3307 -uroot -ppw test -e 'SELECT COUNT(*) FROM t' >/dev/null 2>&1; do read -t 1 <> <(:) || true; done" || { echo 'MySQL did not start' >&2; exit 1; }
echo 'mysqlnd.collect_statistics=0' > off.ini
for pass in $(seq 1 ${PASSES:-3}); do
	for stats in on off; do
		extra=(); [ $stats = off ] && extra=(-v "$PWD/off.ini:/usr/local/etc/php/conf.d/zz-off.ini:ro")
		docker run -d --name repro-app --network host --cpuset-cpus "$APP_CPUS" -v "$PWD/Caddyfile.run:/etc/frankenphp/Caddyfile:ro" "${extra[@]}" mysqlnd-zts-repro >/dev/null
		timeout 60 bash -c 'until curl -fs http://127.0.0.1:8080/ >/dev/null; do read -t 1 <> <(:) || true; done' || { echo 'FrankenPHP did not answer 200' >&2; docker logs repro-app 2>&1 | tail -5 >&2; exit 1; }
		taskset -c "$LOAD_CPUS" wrk -t4 -c50 -d5s http://127.0.0.1:8080/ >/dev/null
		cg=/sys/fs/cgroup/system.slice/docker-$(docker inspect -f '{{.Id}}' repro-app).scope/cpu.stat
		u0=$(awk '/^usage_usec/{print $2}' "$cg")
		out=$(taskset -c "$LOAD_CPUS" wrk -t4 -c50 -d20s http://127.0.0.1:8080/)
		u1=$(awk '/^usage_usec/{print $2}' "$cg")
		n=$(echo "$out" | awk '/requests in/{print $1}')
		echo "threads ${THREADS:-16} pass $pass statistics $stats: $n requests, $(echo "$out" | awk '/Requests\/sec/{print $2}') req/s, $(awk -v u=$((u1 - u0)) -v n="$n" 'BEGIN{printf "%.3f", u / n / 1000}') ms CPU per request"
		docker rm -f repro-app >/dev/null
	done
done
docker rm -f repro-mysql >/dev/null

PHP Version

PHP 8.5.11 (ZTS); the code is the same on PHP-8.4, PHP-8.5 and master (b7256ee).

Operating System

Ubuntu 26.04, kernel 7.0.0-38, on bare metal and in the KVM guest; x86-64