Skip to content

Recover Azure App Service agents after overlapping restarts - #12899

Draft
Lewis-E wants to merge 8 commits into
lewis.lewis/java-named-pipe-statsd-fixesfrom
lewis.lewis/svls-9057-aas-agent-supervision
Draft

Lewis-E wants to merge 8 commits into
lewis.lewis/java-named-pipe-statsd-fixesfrom
lewis.lewis/svls-9057-aas-agent-supervision

Conversation

@Lewis-E

@Lewis-E Lewis-E commented Oct 15, 2026 •

Copy link
Copy Markdown

What Does This Do

This makes the tracer's agent supervisor in Azure App Service recover from overlapping restarts, and makes its lifecycle readable in default-level logs. It's stacked on #12898, so the diff here is the 6 commits on top of that branch. Each commit message has the details.

  1. ProcessSupervisor logs each agent start, exit (exit code and run time), and stop at info, with the tracer PID and, on Java 9+, the agent PID. It also logs when it uses an agent that another process started, when that agent disappears, and when shutdown leaves an agent running. After repeated quick failures these lines drop to debug until the agent stays up.
  2. Restarts back off from 2 s, doubling to a 60 s cap, and never stop. The fault count resets only after the agent has stayed up for 60 s.
  3. While an agent this tracer started is running, the supervisor health-checks it every 10 s and restarts it after 3 failed checks in a row. It never starts a second copy while its own is still running.
  4. A pipe whose instances are all busy counts as present.
  5. The trace agent health check sends GET /info over a pipe handle of its own, with tracing muted, and waits at most 2 s.
  6. In App Service, each JVM writes its own log file, <name>-<pid>.<ext>, which rolls over at DD_MAX_LOGFILE_SIZE (10 MB by default). At startup a background thread deletes this tracer's own per-process files older than DD_TRACE_LOGFILE_RETENTION_DAYS (32 by default, 0 turns cleanup off). These match the .NET tracer's settings.

Motivation

On Azure App Service (Windows), an overlapped IIS recycle leaves Java apps without telemetry until someone restarts them (SVLS-9057, escalation SLES-2826). The old and new worker run at the same time, and both tracers supervise datadog-trace-agent.exe and dogstatsd.exe on the same install-wide named pipes.

We reproduced the overlap in App Service. Before these changes:

  • The default logs couldn't show what happened. Starts, exits, exit codes, and use of another worker's agent were debug-only or not logged.
  • A new JVM truncated the shared tracer log file from 3,522 bytes to 0, so the old JVM's last lines, including its shutdown, were lost. This is likely why customer logs never cover an outage.
  • Each worker start runs several JVMs in a row through JavaBootstrapper, four in our runs, and each one starts and stops its own agents.
  • ProcessSupervisor reset its fault count on every start, so an agent that kept exiting restarted every 2 s forever, while a start that kept throwing stopped retrying after 5 attempts. Once it started an agent, it blocked in waitFor(), so a hung agent was never checked.

With these commits and the StatsD commits from #12898 in a repro build, both agents failed one health check while starting and then passed, with no restart. When the old worker's agents exited, a JVM in the new worker restarted both within 1 s. Traces came back about 2 min 27 s after the recycle started, and about 1 min 50 s of that was JVM restarts in the new worker. In a run with the per-process log file, the old JVM's file was still there after the recycle.

Additional Notes

Design choices and known gaps:

  • The per-process log file changes how tracer logging works in App Service. It's the last commit, so it can come out on its own if the Java tracer team wants a different approach.
  • It gets the PID through ProcessHandle by reflection, with a JMX fallback on Java 8, because PidHelper can't be used safely during logging startup. Its cleanup runs on a plain daemon thread.
  • It fails checkConfigurations: it calls EnvironmentVariables.get directly, and DD_MAX_LOGFILE_SIZE and DD_TRACE_LOGFILE_RETENTION_DAYS aren't in metadata/supported-configurations.json. If the team wants the log file, this gets fixed here. Registering the keys now would add public configuration ahead of review.
  • AasPerProcessLogFile is left out of agent-logging's instruction coverage rule. Its PID lookup has separate Java 8 and Java 9+ paths, and no single JVM runs both. The branch rule still applies to it. If reviewers prefer, the PID lookup can move into its own small class, and only that class would be excluded.
  • Busy pipes are recognized by the English text of the open error.
  • Any HTTP status from /info counts as healthy.
  • A slow /info from an agent that another worker started causes one duplicate start. The duplicate loses the pipe race and exits.
  • Pipe reads can't time out, so an /info request to a hung agent stays blocked until that agent exits. Later checks fail without sending another request.
  • The supervisor restarts only an agent it started, and never kills one it didn't. If an adopted trace agent hangs while holding the pipe, the tracer logs it once and keeps retrying on the capped backoff, but traces stay down until that agent exits.
  • On Java 8 the logs omit the child agent's PID, because Java 8 has no public API for it.

Tests: ProcessSupervisorTest covers the backoff, the fault-count reset, restarting after failed health checks, adopting an agent it didn't start, and not starting a second agent while one won't stop. AasPerProcessLogFileTest covers the file name, rollover, and retention cleanup. The pipe health checks and the busy-pipe check ran in the App Service repro on Windows and have no unit tests. Per-process pipe names, so that overlapping workers never share agents, are the planned follow-up and aren't in this PR.

Contributor Checklist

Jira ticket: SVLS-9057

In Azure App Service the tracer starts and supervises the trace agent and
DogStatsD. Overlapping worker processes during platform restarts can leave
those agents missing or shared, but starts, exits, and stops were only
logged at debug level, so default customer logs could not show what
happened.

- Log each agent start, exit (with exit code and run time), and stop at
  info, with the tracer PID and, on Java 9+, the agent PID.
- Log when an already-running agent not started by this process is used,
  and when that agent goes away.
- After repeated quick failures, move lifecycle lines to debug until the
  agent stays up, so a crash loop does not flood the log.
- Warn when shutdown leaves an agent running.
ProcessSupervisor reset its fault count on every successful start, so an
agent that kept exiting was restarted every 2 seconds forever, while a
start that kept throwing gave up for good after 5 attempts.

- Forget faults only after the process stays up for 60 seconds.
- Back off after each fault, doubling from 2 seconds to a cap of 60
  seconds, and never stop retrying.
- Move repeated start exceptions to debug after the first few, so a
  permanently failing start does not log a stack trace every minute.
Once ProcessSupervisor started an agent it blocked in waitFor() until the
agent exited, so an agent that hung but stayed alive was never checked or
replaced.

- While its own agent runs, wait in keep-alive steps of 10 seconds and
  run the health check after each one.
- After 3 failed health checks in a row, stop that agent and go back
  through the normal start path, after the fault backoff. Log the first
  failed check and the restart once.
- Keep tracking an agent that is still running after a stop request, so
  a restart never starts a second copy next to it.
- Supervisors without a health check keep only waiting for the exit.
On Windows, File.exists() can report false for a named pipe whose
instances are all busy. The supervisor then took a busy agent for a
missing one and started a competing agent, which lost the pipe race and
exited.

When File.exists() reports false on Windows, open the pipe once and close
it again. Only "cannot find the file" means the pipe is absent; "All pipe
instances are busy" and other open errors mean a process is serving it.
The trace agent health check only tested that its pipe existed, so a hung
agent that kept its pipe was trusted forever.

- After the pipe check passes, send a minimal GET /info over a pipe handle
  of the check's own, outside the tracer's HTTP clients, with tracing
  muted.
- Treat any HTTP status line as an answer. A busy pipe counts as present.
- Pipe reads cannot time out, so send the request on a daemon thread and
  give up after 2 seconds. While that request is still blocked, later
  checks fail without sending another one.
- DogStatsD never replies, so its check stays "pipe present".
Overlapping worker processes in Azure App Service run several JVMs that
share one configured log file, and each new JVM truncates it, so the old
JVM's last lines are lost. In an overlapped IIS recycle, a new JVM wiped
the old JVM's log, including its shutdown. Copy the .NET tracer:

- Write to <name>-<pid>.<ext> instead of the configured file.
- Roll over at DD_MAX_LOGFILE_SIZE bytes (default 10 MB).
- At startup, delete this tracer's own per-process files last written
  more than DD_TRACE_LOGFILE_RETENTION_DAYS ago (default 32, 0 turns
  cleanup off), stopping quietly at the first failure.

Only active when DD_AZURE_APP_SERVICES is set.
@datadog-prod-us1-5

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.90 s 13.81 s [-0.1%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent 12.88 s 12.94 s [-1.1%; +0.2%] (no difference)
startup:petclinic:appsec:Agent 17.07 s 16.93 s [-0.2%; +1.8%] (no difference)
startup:petclinic:iast:Agent 17.03 s 16.95 s [-0.4%; +1.3%] (no difference)
startup:petclinic:profiling:Agent 16.55 s 16.70 s [-2.0%; +0.2%] (no difference)
startup:petclinic:sca:Agent 17.06 s 16.88 s [+0.3%; +1.9%] (maybe worse)
startup:petclinic:tracing:Agent 16.08 s 16.11 s [-1.2%; +0.8%] (no difference)

Commit: 391843bd · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

The supervisor now health checks its own agent while it runs, restarts it
after three failed checks in a row, uses an agent another process
started, and never gives up retrying. Only the backoff arithmetic was
tested, so coverage of these paths was too low for the build to pass.

The new tests start a fake Process from a mocked ProcessBuilder. The fake
returns quickly from timed waits, so health checks that normally run ten
seconds apart run fast, and each test can choose how the process exits or
refuses to stop. They check that the supervisor:

- restarts its own process after exactly three failed health checks;
- keeps its process when a health check passes before the third failure;
- uses a process it did not start, then starts its own once that one
  goes away (this waits out the real ten second healthy interval);
- keeps tracking a process that ignores stop requests instead of
  starting a second copy;
- retries after the process fails to start;
- logs the exit code and starts again after the process exits.
AasPerProcessLogFile had no tests, so the agent-logging coverage check
failed. The new tests check that, in Azure App Service only, the tracer
writes to <name>-<pid>.<ext> next to the configured file. They also check
that it falls back to standard error when that file cannot be opened,
rolls over to numbered files at DD_MAX_LOGFILE_SIZE (0 turns rollover
off), and appends to an existing file instead of truncating it. Cleanup
is checked too: it deletes only this tracer's per-process files older
than DD_TRACE_LOGFILE_RETENTION_DAYS (0 turns it off), and it stops
quietly when a file cannot be deleted.

The class is left out of the instruction coverage rule. Its PID lookup
has one path for Java 8 and another for Java 9+, and coverage runs on
only one JVM, so one path always shows as missed. The class can't use
PidHelper, because PidHelper sets up an slf4j logger and this code runs
while the tracer's logger is starting. The branch rule still covers the
class, and both rules still cover RollingFileOutputStream.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core comp: logging Tracer internal logging type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant