Skip to content

Add deadlock event with JVMTI-captured stacktraces#470

Draft
jbachorik wants to merge 4 commits intomainfrom
jb/deadlock-stacktraces
Draft

Add deadlock event with JVMTI-captured stacktraces#470
jbachorik wants to merge 4 commits intomainfrom
jb/deadlock-stacktraces

Conversation

@jbachorik
Copy link
Copy Markdown
Collaborator

@jbachorik jbachorik commented Apr 14, 2026

What does this PR do?:

Adds a native JFR deadlock event (datadog.DeadlockedThread) that captures real
JVMTI stacktraces for both the deadlocked thread and the lock owner. This provides
proper navigable stacktraces in JFR tools — a significant improvement over the
string-based stacks in the dd-trace-java fallback implementation.

Key components:

  • New T_DEADLOCK JFR event type with fields: deadlockId, eventThread, stackTrace,
    lockName, lockOwnerThread, lockOwnerStackTrace
  • JVMTI GetStackTrace() capture for both threads in the deadlock pair, stored in
    the profiler's CallTraceStorage as real stacktrace constant pool references
  • DeadlockDetector Java class using ThreadMXBean.findDeadlockedThreads() with
    periodic polling on a daemon thread
  • JavaProfiler.startDeadlockDetector() / stopDeadlockDetector() lifecycle API

Motivation:

PROF-11534 — The existing deadlock event in dd-trace-java uses JDK JFR with string
stacktraces. This PR adds the improved native implementation in java-profiler that
captures real stacktraces via JVMTI, which dd-trace-java can use when the profiler
is loaded.

Additional Notes:

  • dd-trace-java keeps its existing DeadlockEventFactory as a fallback for
    environments where the profiler isn't loaded
  • Works on JDK 8+ (all APIs are JDK 1.5+; JVMTI is always available)
  • One event per deadlocked thread; events in the same cycle share a deadlockId
  • Lock name truncated to 160 chars to fit within single-byte JFR event size prefix
  • Supports both monitor and java.util.concurrent lock deadlocks (falls back to
    findMonitorDeadlockedThreads() when synchronizer usage is not supported)

How to test the change?:

./gradlew ddprof-test:testRelease --tests '*DeadlockDetectionTest*'

The test creates a real deadlock between two threads, starts the detector with a
100ms interval, and verifies that datadog.DeadlockedThread events appear in the
profiler's JFR output with non-empty stacktraces and matching deadlock IDs.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.
  • This PR doesn't touch any of that.
  • JIRA: PROF-11534

🤖 Generated with Claude Code via muse implement

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jbachorik jbachorik added the AI label Apr 14, 2026
Comment thread ddprof-lib/src/main/cpp/flightRecorder.cpp Outdated
Comment thread ddprof-lib/src/main/cpp/profiler.cpp
- Simplify lock_name truncation with inline min()
- Add JVMTI safepoint warning on captureStackTrace
- Move DeadlockDetector to test-only; profiler exposes only recordDeadlock API
- Remove startDeadlockDetector/stopDeadlockDetector from JavaProfiler
- Use before()/after() hooks in test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jbachorik jbachorik marked this pull request as ready for review April 14, 2026 15:28
@jbachorik jbachorik requested a review from a team as a code owner April 14, 2026 15:28
@dd-octo-sts
Copy link
Copy Markdown

dd-octo-sts bot commented Apr 14, 2026

CI Test Results

Run: #24452346678 | Commit: 3b3f1e0 | Duration: 23m 24s (longest job)

All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - - -
8-ibm - - -
8-j9 - -
8-librca - -
8-orcl - - -
11 - - -
11-j9 - -
11-librca - -
17 - -
17-graal - -
17-j9 - -
17-librca - -
21 - -
21-graal - -
21-librca - -
25 - -
25-graal - -
25-librca - -

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-04-15 12:05:20 UTC

@jbachorik jbachorik marked this pull request as draft April 14, 2026 16:52
jbachorik and others added 2 commits April 15, 2026 13:01
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant