Skip to content

WebSocket upgrades fail intermittently under pm2: ERR_HTTP_SOCKET_ASSIGNED (@pm2/io httpMetrics vs @fastify/websocket) #545

Description

@melvincarvalho

Symptom

On a production JSS (0.0.204, node v22.21.1, Ubuntu) running under pm2, WebSocket upgrades to /.webrtc fail in intermittent bursts. Clients see either a failed connect or — worse — a half-open socket (TLS front returns 101, but the inner upgrade has already died, so the client gets silence: no messages, no error). Fresh-socket clients (curl, Node WebSocket) almost always succeed; browsers holding keep-alive connections to the same host intermittently hit it.

Logs

{"level":40,"err":{"type":"Error","message":"ServerResponse has an already assigned socket",
 "stack":"Error [ERR_HTTP_SOCKET_ASSIGNED]: ServerResponse has an already assigned socket
    at ServerResponse.assignSocket (node:_http_server:291:11)
    at Server.onUpgrade (.../node_modules/@fastify/websocket/index.js:56:19)
    at Server.emit (node:events:531:35)
    at Server.<anonymous> (.../lib/node_modules/pm2/node_modules/@pm2/io/build/main/metrics/httpMetrics.js:153:37)
    at onParserExecuteCommon (node:_http_server:977:14)",
 "code":"ERR_HTTP_SOCKET_ASSIGNED"},"msg":"websocket upgrade failed"}

Bursts of these correlate exactly with client-observed dead signaling connections.

Analysis

pm2's @pm2/io httpMetrics instrumentation wraps the HTTP server's events and sits in the upgrade path ahead of @fastify/websocket's onUpgrade. "Already assigned socket" points at upgrades arriving on reused keep-alive sockets (a connection that previously served a normal HTTP response), where the wrapper's ServerResponse handling collides with the upgrade's assignSocket. That matches the observed pattern: fresh sockets fine, browser keep-alive connections intermittent.

Possibly aggravated in this deployment by a pm2 daemon/CLI version mismatch (in-memory 6.0.14 vs local 4.5.6), so the injected @pm2/io came from a different pm2 than the one managing the process.

Suggested directions

  • register the upgrade handler so it runs ahead of (or is resilient to) external 'request'/'upgrade' instrumentation — e.g. guard onUpgrade against an already-assigned socket and destroy/retry instead of throwing through
  • document pm2 deployment guidance for websocket-bearing instances (metrics-off configuration, or systemd as the recommended supervisor when --webrtc/--tunnel/notifications are enabled)
  • a health probe: the /.webrtc route could expose a trivial HTTP GET response (instead of 404) so operators can distinguish "flag off" from "upgrade path broken"

Environment

  • jss 0.0.204 under pm2 (fork mode), node v22.21.1
  • mitigation being applied operationally: pm2 update + restart; will report recurrence here

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions