Skip to content

usockets: decouple dns_ready_head and closed_connecting_head lists - #29099

Closed
robobun wants to merge 4 commits into
mainfrom
farm/3c97b97f/decouple-dns-close-lists
Closed

robobun wants to merge 4 commits into
mainfrom
farm/3c97b97f/decouple-dns-close-lists

Conversation

@robobun

@robobun robobun commented Apr 10, 2026

Copy link
Copy Markdown
Collaborator

What

Give us_connecting_socket_t a dedicated next_closed link for the deferred-free list (closed_connecting_head) instead of reusing the dns_ready_head next pointer, make us_connecting_socket_free idempotent via a scheduled_for_free bit, and have the drain loop skip entries that are already scheduled for free.

Why

The v1.3.11 linux-x64-baseline crash signature is:

panic(main thread): Segmentation fault at address 0x0
  us_internal_drain_pending_dns_resolve   loop.c:238
  ← return from us_internal_socket_after_resolve

Disassembly of the callee prologue:

mov 0x8(%rdi),%rsi   ; rsi = c->context
mov (%rsi),%rax      ; rax = context->loop   ← fault, rsi == 0
decl 0xb0(%rax)      ; loop->num_polls--

c->context (offset 8 in us_connecting_socket_t) reads as 0. Nothing ever writes c->context = NULL explicitly, so the block at c was freed and recycled by a subsequent us_calloc. #29067 added an extra context ref, which keeps the context alive but does nothing when c itself is the recycled block — the field is zero regardless. #29068 added Bun__addrinfo_cancel, which correctly refuses once request.result is set, so it doesn't change the guarded paths either.

Analysis of the pending_resolve_callback invariant

us_connecting_socket_free(c) is reachable from four sites and in every one c->pending_resolve_callback == 0:

  1. after_resolve(c) c->closed branch — pending cleared two lines earlier; c has just been dequeued from the drain snapshot.
  2. us_connecting_socket_close(c) non-pending branch — explicit pending == 0.
  3. us_connecting_socket_close(c) cancel-success branch — Bun__addrinfo_cancel only returns 1 while request.result == NULL, i.e. before the worker has snapshotted notify (both sides hold global_cache.lock), so us_internal_dns_callback(c) will never fire and c never reaches dns_ready_head.
  4. after_open(s) success — s was created by start_connections(c), called only from after_resolve(c) after it cleared pending.

And us_internal_dns_callback(c) fires exactly once per c (Bun__addrinfo_set either enqueues directly or appends to notify, never both; afterResult moves notify out under the same lock). So c is never simultaneously on closed_connecting_head and reachable from a future dns_ready_head snapshot — the invariant holds through every 2/3/4-socket interleaving traced.

The production workload (fetch via proxy + WebSocket + spawn + AbortSignal, ~3 req/s, one crash per several process-hours) nevertheless hits it, so either there is an interleaving outside the traced set or another writer is corrupting c->next. Separately, us_loop_run_bun_tick has no re-entrancy guard and waitForPromise / HTMLRewriter.transform / expect().resolves can nest it on the same loop from inside on_connect_error, which runs free_closed_sockets inside a drain — but even that path leaves the outer next untouched because next is captured before after_resolve(s).

The structural hazard this removes

The two lists previously shared c->next. us_connecting_socket_free(c) did:

c->next = c->context->loop->data.closed_connecting_head;  // overwrites dns_ready_head link

so the correctness of the drain loop depended entirely on the invariant above. With a dedicated next_closed link, enqueuing c for deferred free can no longer truncate or splice the drain chain, us_connecting_socket_free becomes safely idempotent (double-call early-returns instead of double-unlinking and cycling the close list), and the drain loop skips anything already scheduled for free (valid memory until free_closed_sockets runs) instead of derefing a released context/addrinfo_req.

This turns a subtle cross-function invariant into a structural guarantee: even if a future change violates pending_resolve_callback, the failure mode is a skipped drain entry, not a recycled pointer in after_resolve.

Testing

resolve-dns.test.ts 71/71, node-net/socket.test.ts/tcp-server.test.ts unchanged vs USE_SYSTEM_BUN=1 baseline (failures there are environmental — proxy cert / ECONNREFUSED to external hosts). The new dns-connecting-socket-stress.test.ts (ASAN-only, 25k concurrent net.connect("localhost") + random-phase abort, quarantine_size_mb=0) runs clean on the patched build; it also ran clean on a pre-#29067 ASAN build, consistent with the field hit rate. No deterministic fail-before — the exact interleaving is not reproduced here. The stress test is included as a regression guard for the lifecycle rather than a gate.

@robobun

robobun commented Apr 10, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 5:55 AM PT - May 1st, 2026

❌ @robobun, your commit 511372d has 3 failures in Build #49657 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 29099

That installs a local version of the PR into your bun-29099 executable, so you can run:

bun-29099 --bun

@robobun

robobun commented Apr 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: draft — defensive hardening, not a bug fix. No fail-before exists on current main.

Root cause of the field crash (1.3.11, linux-x64-baseline, ~2026-04-02): that build predates all of:

The crash disasm (c->context == 0 at after_resolve entry ⇒ c itself recycled) is consistent with a re-entrant us_loop_run_bun_tick from on_connect_error running us_internal_free_closed_sockets mid-drain (no tick_depth gate), combined with the shared ->next field. Each link in that chain is now blocked on main, and #30024 removes the c->context->loop deref entirely.

What this PR adds on top: separate next_closed link so closed_connecting_head and dns_ready_head use independent fields; us_connecting_socket_free becomes idempotent (scheduled_for_free bit); drain skips already-scheduled entries. Structural — removes the latent shared-field hazard so a future caller that violates the pending_resolve_callback invariant degrades to a skipped drain entry instead of a recycled pointer.

Conflict resolution (511372d, vs #30024): kept main's group/loop/ssl_ctx layout, dropped ssl bit, new us_connecting_socket_free(c) signature, detach() helper. Layered scheduled_for_free bit + next_closed link on top. Verified: builds clean, resolve-dns.test.ts 71/71, 500× concurrent net.connect("localhost")+abort under debug+ASAN clean.

Four asks from the investigation, answered against current main:

  1. ->next collision / free while on dns_ready_head? Collision is real but unreachable: close defers via pending_resolve_callback, cancel fails once result is set, after_open can only fire post-after_resolve, tick_depth blocks re-entrant free, and c->loop is now captured directly.
  2. after_open success path needs the pending_resolve_callback guard? No — after_open is only reachable after start_connections ← after_resolve, which already cleared it.
  3. link_connecting_socket early-return leaves c->context NULL? No — link runs immediately after us_calloc. (Field is now c->group; c->loop is set unconditionally before link.)
  4. Deterministic/ASAN repro? Could not construct one on current main. 25k connect+abort under ASAN clean on both pre-fix(usockets): hold a context ref for the lifetime of pending_resolve_callback #29067 and main.

CI #48661 (prev sha): my test passed on debian-13-x64-asan (51.5s). Failed ASAN shard was an agent loss at fetch-http2-client.test.ts (not my shard). Windows/darwin failures (test-integration-rspack, serve-stream-reject-flush-leak, websocket-server, s3-storage-class, bake/*) all appear on other branches' builds 48613–48653.

Gate: rejects because the test skips on bun-debug (harness isASAN keys off the binary name bun-asan). Even if it ran, it would pass without the src change — no fail-before on main.

Recommendation: take or leave as hardening. Happy to close if the structural change isn't wanted given the root cause is already addressed.

@github-actions

Copy link
Copy Markdown
Contributor

This PR may be a duplicate of:

  1. fix(usockets): snapshot loop before context unref in us_connecting_socket_free #29064 - Also fixes the v1.3.11 segfault in us_connecting_socket_free / us_internal_drain_pending_dns_resolve; modifies the same function in socket.c to address the same crash, but with a different approach (snapshotting the loop pointer vs. this PR's structural fix with separate next_closed link)

🤖 Generated with Claude Code

@robobun

robobun commented Apr 10, 2026

Copy link
Copy Markdown
Collaborator Author

Not a duplicate of #29064 — that one snapshots loop before the unlink's us_socket_context_unref, which with #29067 already merged is now redundant (the pending-resolve ref keeps the context alive past unlink). This PR instead separates the two linked lists so they use independent link fields, and makes us_connecting_socket_free idempotent. The two are compatible; this one is the broader structural change.

CI build failures are from #29092 (ZIG_PARALLEL_SEMA) which broke main — same failures on other PRs opened after it; #29097 is the follow-up fix. My change compiles clean locally (verified on e1069426f base, the commit immediately before #29092).

@robobun
robobun force-pushed the farm/3c97b97f/decouple-dns-close-lists branch from 1e28538 to 7e39306 Compare April 10, 2026 05:07
robobun and others added 2 commits April 21, 2026 00:54
Give us_connecting_socket_t a dedicated next_closed link for the
deferred-free list instead of reusing the dns_ready_head next pointer,
and make us_connecting_socket_free idempotent via a scheduled_for_free
bit. Drain skips entries that are already scheduled for free.

The two lists previously shared c->next, so enqueuing c for deferred
free while it was still reachable via a dns_ready_head drain snapshot
silently spliced the remaining drain entries onto the close list. The
pending_resolve_callback invariant is supposed to prevent that overlap,
but the v1.3.11 loop.c:238 SIGSEGV @ 0x0 (c->context == NULL in
after_resolve) shows a recycled us_connecting_socket_t reaching the
drain loop, and neither #29067 (context ref) nor #29068 (cancel)
addresses that fault mode. This change makes the separation structural
so a future invariant violation degrades to a skipped entry rather than
a use-after-free.
@robobun
robobun force-pushed the farm/3c97b97f/decouple-dns-close-lists branch from 7e39306 to e7d15f1 Compare April 21, 2026 01:07
@alii

alii commented Apr 28, 2026

Copy link
Copy Markdown
Member

@robobun status?

@robobun

robobun commented Apr 28, 2026

Copy link
Copy Markdown
Collaborator Author

Status: draft, parked. This is defensive hardening for the v1.3.11 loop.c:238 SIGSEGV (recycled us_connecting_socket_t in the DNS drain), not a proven root-cause fix.

Happy to mark ready if the hardening is wanted despite no fail-before, or close if not.

alii and others added 2 commits April 28, 2026 14:38
…e-dns-close-lists

# Conflicts:
#	packages/bun-usockets/src/internal/internal.h
#	packages/bun-usockets/src/socket.c
@robobun

robobun commented Jun 28, 2026

Copy link
Copy Markdown
Collaborator Author

Closing. This was defensive hardening for the v1.3.11 us_internal_drain_pending_dns_resolve SIGSEGV, and a deep read-only analysis of current main has confirmed the hazard it guards against is no longer reachable.

Why close

The field crash (1.3.11, linux-x64-baseline, built ~2026-04-02) predates every one of these, all since merged:

On top of that, #30412 rewrote the runtime in Rust. The new resolver (src/runtime/dns_jsc/dns.rs) was audited separately and preserved every locking invariant of the Zig original: one global mutex, after_result sets result and moves the notify list out under a single lock acquisition, cancel checks result under that same lock and so cannot return success after the completer has committed.

The analysis

To settle "real fix vs. unnecessary hardening," a 48-agent research pass was run against a clean checkout of a1c39ded9b, asking one question: is there any reachable interleaving that calls us_connecting_socket_free(c) while c is still an unreached element of a live dns_ready_head drain snapshot? Verdict, unanimous across independent units:

  1. us_connecting_socket_free has exactly 4 call sites. Each is individually proven safe.
  2. pending_resolve_callback == 0 on a published socket implies either its after_resolve already ran (so the drain already passed it) or Bun__addrinfo_cancel returned 1 (so it was never and will never be enqueued). There is no third state.
  3. us_internal_socket_after_open is unreachable for a connecting socket until after_resolve for it has already executed; start_connections is the only producer of those polls and only after_resolve calls it.
  4. The adversarial case from the original report, several fetch()es to one hostname sharing an AbortSignal and the error handler aborting the others mid-drain, is handled: cancel returns 0 (the result is already set, that is why they are on the snapshot), the socket is detached but not freed, and nothing on that path writes ->next.
  5. No Rust code calls us_connecting_socket_free, and every external us_connecting_socket_close runs on the loop thread.

So the ->next field sharing between dns_ready_head and closed_connecting_head is real but latent: as long as the pending_resolve_callback invariant holds (and it does, everywhere), nothing can exploit it. This PR turned that invariant into a structural guarantee, but there is no input that distinguishes the two states, which is also why no fail-before test could be produced.

One real finding came out of this

tick_depth, the re-entrancy guard #29354 added, is only incremented/decremented in eventing/epoll_kqueue.c, and that entire file is inside #if defined(LIBUS_USE_EPOLL) || defined(LIBUS_USE_KQUEUE). Windows builds with LIBUS_USE_LIBUV (libusockets.h:634), so on Windows tick_depth is permanently 0 and the gate at loop.c:357 is permanently true: us_internal_free_closed_sockets runs on every loop_post, including a nested one, which is exactly what the comment above the gate says must not happen. #29354's diffstat confirms it touched epoll_kqueue.c and not libuv.c. That is a separate, Windows-only issue and does not belong in this PR; it is written up in the investigation thread and the fix (mirroring the accounting into the libuv loop-run entry points) is a few lines.

Thanks to the original reporter for an unusually precise investigation; the disassembly pinning the fault to c->context == NULL inside after_resolve is what made the whole thing tractable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants