Description

This PR makes the HikariCP leak-detection threshold and JMX MBean registration configurable per database pool via db.properties, instead of relying on HikariCP defaults that cannot be changed without a code change.

CloudStack already maps a subset of db.properties values onto HikariConfig in framework/db/src/main/java/com/cloud/utils/db/TransactionLegacy.java (e.g. maxActive, maxIdle, maxWait, minIdleConnections, connectionTimeout, keepAliveTime). This PR adds two more, following the exact same parsing/threading pattern:

Supported for all three pools that use the shared datasource factory: cloud, usage, simulator.

Behaviour:

• leakDetectionThreshold absent or 0 → leak detection disabled (unchanged default behaviour). Only applied when set to a value > 0. (HikariCP itself ignores values below 2000 ms with a warning.)
• registerMbeans absent or false → MBeans disabled (unchanged default). true → Hikari JMX MBeans registered for live pool-counter observation.
• Settings are applied only on the HikariCP path; the DBCP datasource path is unaffected.
• A debug-level log line reports the effective values per pool (no credentials or sensitive data).

Motivation / context: in production we saw the management server become unstable — and eventually crash — on clusters exercising Host-HA. Watching MySQL with SHOW PROCESSLIST during the incident showed the number of sessions owned by the cloud DB user climbing steadily over a couple of hours, all of them in the Sleep state, until the HikariCP pool (db.cloud.maxActive, default 250) was exhausted and the server could no longer borrow a connection. That signature — monotonically growing, never-reaped, all idle, all owned by the cloud user — is a classic DB connection leak in a periodic code path (suspected Host-HA host checks) that borrows a pooled connection and never returns it.

The problem is these symptoms tell you that connections leak, not where. HikariCP already has the exact tool for that — leakDetectionThreshold — but CloudStack hard-wires it off with no way to turn it on. This PR exposes it (and registerMbeans) through db.properties so an operator can enable leak detection on a live server; HikariCP then logs an Apparent connection leak detected stack trace identifying the precise code path that borrowed the connection and failed to return it, and the MBeans give live pool-counter visibility. The actual leak fix is a separate change; this PR is the diagnostic enabler.

Everything is disabled by default, so there is no behavioural change for existing deployments that don't set the new properties.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

N/A

How Has This Been Tested?

Build: compiled the affected module and its dependencies off tag 4.22.1.0:

mvn -pl framework/db -am -DskipTests compile

Result: BUILD SUCCESS (checkstyle passed). The change is confined to property parsing and threading through the existing createDataSource → createHikaricpDataSource chain, reusing the existing parseNumber(...) helper.

Unit tests: the apply-logic is factored into a package-private applyHikariDebugSettings(HikariConfig, Long, Boolean, String) and covered by 4 new TransactionLegacyTest cases — defaults-disabled, 0-keeps-disabled, leak-detection-enabled (60000), and register-MBeans-enabled:

mvn -pl framework/db test -Dtest='TransactionLegacyTest#applyHikariDebugSettings*'

Result: Tests run: 4, Failures: 0, Errors: 0 — BUILD SUCCESS.

Runtime validation plan (on a patched management server):

1. In /etc/cloudstack/management/db.properties:

   db.cloud.leakDetectionThreshold=60000
   db.cloud.registerMbeans=true
   

2. systemctl restart cloudstack-management
3. Reproduce the Host-HA scenario and watch:

   grep -iE "leak detection|Apparent connection leak|ProxyLeakTask|Hikari" \
     /var/log/cloudstack/management/management-server.log
   

   Expected: java.lang.Exception: Apparent connection leak detected with a stack trace through com.zaxxer.hikari.HikariDataSource.getConnection(...) → com.cloud.utils.db.TransactionLegacy... identifying the borrowing path.
4. With registerMbeans=true, the com.zaxxer.hikari:type=Pool (cloud) MBean is visible via JMX for live pool counters.

How did you try to break this feature and the system with this change?

Edge cases considered:

• Property absent → null → leak detection disabled, registerMbeans=false (existing behaviour preserved).
• leakDetectionThreshold=0 → not applied (disabled).
• leakDetectionThreshold between 1–1999 ms → passed to Hikari, which warns and ignores it (documented Hikari behaviour; noted in the code comment and the sample config).
• registerMbeans=false explicitly → MBeans off.
• DBCP connection-pool path → new values are not forwarded, so behaviour is unchanged.
• Default datasource fallback path (getDefaultHikaricpDataSource) → untouched.

These cases (defaults, 0, enabled threshold, enabled MBeans) are locked down by the new applyHikariDebugSettings unit tests.