Reference-count ceiling is 32 767 on 32-bit targets: refcount_overflow aborts the process

On any 32-bit target — wasm32-wasip2 here — an object reaching 32 767 live references calls refcount_overflow(), which is std::process::abort(). Under wasm that is an unreachable trap, so the embedder gets no Python-level error and no message.

Reproducer

marker = object()
holder = [marker] * 32767

32767 aborts. 32760 returns normally. Bisected precisely: [marker] * 32765 returns, * 32766 traps — 32 767 total live references at the trap, counting the ones the interpreter holds itself.

Reproduced with RustPython 0.5.0 embedded in a wasm32-wasip2 component (rustpython-vm with compiler, freeze-stdlib, host_env, stdio), run under wasmtime 45, built by Rust 1.96.1 stable.

Symbolised guest backtrace, from a release build with -C strip=none:

0: abort
1: std::sys::pal::wasi::abort_internal
2: std::process::abort
3: rustpython_common::refcount::refcount_overflow
4: rustpython_vm::object::payload::PyPayload::into_ref
5: rustpython_vm::vm::context::Context::new_str
6: rustpython_vm::frame::ExecutingFrame::execute_make_function

Cause

crates/common/src/refcount.rs packs the reference count and its flags into one usize, so the counter widths follow the pointer width:

// State layout (usize):
//   [1 bit: destructed] [1 bit: published] [1 bit: leaked] [N bits: weak_count] [M bits: strong_count]
// 64-bit: N=30, M=31.  32-bit: N=14, M=15.

15 bits gives 32 767 strong references at 32-bit pointer width, against roughly 2.1 billion at 64-bit. The weak count is narrower still: 14 bits, 16 383.

main carries the same layout as the 0.5.0 tag.

Why this is reachable in ordinary use, not a synthetic limit

Interned strings and shared singletons cross 32 767 once a realistic module graph is loaded. Measured inside the component:

import urllib3 alone succeeds, and performs a real HTTP request.
import urllib3 followed by import decimal aborts. So does import http.cookiejar, and so does import charset_normalizer, in either order. Each of those imports alone succeeds.
import requests therefore aborts, which puts most of the third-party ecosystem out of reach on 32-bit targets.

Three other explanations were tested and refuted, so the ceiling is not standing in for something else:

Stack. Rebuilding with -C link-arg=-zstack-size=16777216, verified applied — the __stack_pointer global moves from 1048576 to 16777216 — changes nothing.
Recursion limit. sys.setrecursionlimit at 256, 400 and 600 abort identically. A debug build, whose limit is 256, raises RecursionError on the same script only because it stops before the refcounts climb.
Memory. A 512 MiB allocation succeeds after import urllib3.

What a fix has to preserve

The compare-and-swap runs over the whole packed word, which also carries the weak count, DESTRUCTED, LEAKED and PUBLISHED. Widening the strong count on 32-bit therefore cannot be a field-width tweak alone. Two directions, both with costs worth weighing by someone who knows the GC design:

Use a 64-bit state word on 32-bit targets, paying for a 64-bit atomic where the platform lacks one natively.
Split the flags out of the counter word, so both counts get the full width.

Related for context rather than as a duplicate: rust-lang/rust#108706 makes the same observation about Arc — an overflow that is theoretical at 64-bit is reachable at 32-bit.

Downstream tracking issue, with the full measurement log: https://gitlab.com/lx-industries/openblob/-/issues/590

I have a fix implemented and tested, and would open it as a PR if the approach looks right to you. Two findings shape it.

All references below are permalinked to 695dda4642718bf007f871b1b72a73d9655bbb98 for the current layout, and to my branch for the proposed one.

The weak half of the word has never counted anything

WEAK_COUNT has been touched by exactly one commit since it appeared — #7194, which introduced the packed layout. In that commit it already appeared only twice: its own definition, and the initial state in RefCount::new.

const WEAK_COUNT: usize = 1 << STRONG_WIDTH;
...
state: Radium::new(COUNT + WEAK_COUNT),

Nothing has ever incremented it, decremented it, or read it back — not in that commit, not since, and not anywhere else in the repository. Weak references are tracked by WeakRefList, whose count is what Py::weak_count walks.

So WEAK_WIDTH reserves half the counter space for a field that has never held a count. On a 64-bit target that costs nothing anyone notices, because the remaining 31 bits are ample. On a 32-bit target it is the whole bug: the strong count is squeezed into 15 bits to reserve 14 for the unused half.

Widening the strong count needs no type change

This is the part that makes the fix small. The state stays one usize in PyAtomic<usize>, and the compare-and-swap still runs over the whole word — so inc, dec, safe_inc and leak keep their existing atomicity, and DESTRUCTED, PUBLISHED and LEAKED keep their bit positions. Only the mask widens (proposed):

const STRONG_WIDTH: u32 = usize::BITS - FLAG_BITS;
const STRONG: usize = (1 << STRONG_WIDTH) - 1;

RefCount::new seeds the word with COUNT instead of COUNT + WEAK_COUNT. Nothing tests the word for zero, and the strong() == 0 branch in inc behaves the same either way.

Result: 29 bits on a 32-bit target and 61 on a 64-bit one, so the ceiling moves from 32 767 to 536 870 911, and from roughly 2.1 billion to roughly 2.3 quintillion. No new dependency, no 64-bit atomic on platforms that lack one, no change to the concurrency design.

If the field is reserved rather than abandoned, the same bug closes without removing it — give the strong count a larger share of the split and leave a narrower weak field in place. That still lifts the 32-bit ceiling clear of anything an import graph reaches. I took removal because nothing references the field today, but this is your call and I will switch if you prefer the field kept.

Tested on the target the bug lives on

Two unit tests: strong_count_reaches_past_a_16_bit_ceiling drives the count past 2²⁰ references, and a_new_refcount_holds_one_strong_reference_and_nothing_else asserts a fresh RefCount's raw word is exactly COUNT — read raw, because get masks the flags away and would hide a field packed back in beside the counter.

rustup target add wasm32-wasip2
CARGO_TARGET_WASM32_WASIP2_RUNNER=wasmtime \
  cargo test -p rustpython-common --target wasm32-wasip2 refcount

Layout	Target	Result
current	`wasm32-wasip2`	`wasm trap: unreachable`, exit 134
widened	`wasm32-wasip2`	3 passed
widened	`x86_64-unknown-linux-gnu`	3 passed

The first row is the abort in this issue, reproduced by a unit test rather than an interpreter session. A 64-bit host passes either way, since 31 bits already covered 2²⁰ — the tests guard 32-bit targets, and the doc comment says so.

cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher --exclude rustpython-capi passes: 999 passed, and the 8 _opcode snapshot tests that fail here fail identically on an unmodified checkout in my environment, so they are not related to this change.

One correction to this issue's scope

I opened this against wasm32-wasip2 because that is where I hit it, but the layout has no target_arch, target_pointer_width or target_os conditional anywhere — the width follows usize::BITS alone. The 32 767 ceiling is therefore identical on i686, armv7 and riscv32.

What I measured on wasm32 is that import requests crosses it. I have not run the same import on a native 32-bit target, so I will not claim it as measured there — though the references accumulate on interned strings and shared singletons, which have no platform-specific behaviour. On a native target the symptom would be a bare SIGABRT rather than a trap, which would not obviously point at reference counting.

Branch, if useful before a PR exists: JMLX42:fix-32bit-refcount-ceiling.

Assisted-by: Claude Code:claude-opus-5 — analysis, patch and tests; reviewed before posting.

Correction: widening the field alone is not enough — the accessor truncates

My earlier comment said the change is small because "only the mask widens". That was wrong, and the missing half is a use-after-free introduced by the widening. A second commit now carries it, and anyone evaluating the approach should weigh both together.

What I missed

State::strong returns u32:

fn strong(self) -> u32 {
    ((self.inner & STRONG) / COUNT) as u32
}

That is exact today, because the field is 31 bits on a 64-bit target. Once the strong count takes the bits the weak field held it becomes 61 bits, and the cast starts truncating above 4 294 967 295 references. Two call sites then read a wrapped value and act on it:

inc treats strong() == 0 as "the previous fetch_add created a permission to run decrement again" and increments a second time.
dec treats strong() == 1 as the last reference and reports the object collectable.

So a count of exactly 2³² reads as 0, and 2³²+1 reads as 1 — the object is freed while 4 billion references remain. The widening trades an abort for memory corruption unless the accessor widens with it.

The fix

fn strong(self) -> usize, returning (self.inner & STRONG) / COUNT unchanged, and the four call sites drop their now-redundant as usize casts. No behaviour changes on a 32-bit target, where 29 bits fit u32 either way.

Why it did not show up in testing

It cannot, at any reasonable cost. Reaching it needs 2³² live references to one object, roughly 34 GB of pointers on a 64-bit host. The tests I ran exercise 2²⁰ references on wasm32-wasip2, where the count is 29 bits and the old u32 return was never wrong. Both branches pass on both targets before and after this commit — the commit is reasoned from the field width, not from a failing test, and I would rather say so than imply coverage I do not have.

Branch state

fix-32bit-refcount-ceiling is now two commits:


`818d1622`	Give the strong count the bits an unused weak field held
`64877c5a`	Return the strong count as usize so a 64-bit target cannot truncate it

The alternative I offered earlier — keep a narrower weak field rather than removing it — avoids this entirely if the split leaves the strong count under 32 bits. That is a point in its favour I did not have when I raised it.

Assisted-by: Claude Code:claude-opus-5 — analysis, patch and tests; reviewed before posting.

Question: how would you want the 2³² boundary tested, if at all?

The second commit widens State::strong to usize so the widened field cannot truncate. I have no test for it, and I want to follow your conventions rather than invent one.

The boundary needs 2³² live references to a single object — roughly 34 GB of pointers on a 64-bit host. My existing tests run 2²⁰ references on wasm32-wasip2, which exercises the ceiling but never approaches the truncation point.

The closest pattern I found is your vendored test.support.bigmemtest. What appeals is that it does not skip: with no -M, it substitutes a tiny size and runs the same body, so the code path stays covered on an ordinary run and only the magnitude is gated. It also carries the cost as data — size and memuse — rather than as a comment.

Three ways I can see to land this, and I do not have a strong preference:

Nothing. Once strong returns usize the property is type-level, and the compiler enforces it. A test would assert what the signature already guarantees.
A bigmemtest-shaped pair in Rust. One parameterised body, a cheap call at 2²⁰ that always runs, and an #[ignore]d call at 2³²+1 naming the memory cost in its reason string — --run-ignored only standing in for -M.
Something you already use that I have not found. git grep '#\[ignore' over crates/ and src/ at 695dda46 returns nothing, so an ignored Rust test would be the first in the tree. I would rather not introduce a convention sideways in a bug-fix PR.

If (1) is the answer, I will say so plainly in the PR description instead of leaving the gap unmentioned. If (2), I will write it. If (3), point me at the example and I will match it.

Related: refcount_test exists for reference-counting tests, but it gates on cpython_only, so it is about observing counts rather than stressing them — not applicable here as far as I can tell.

Assisted-by: Claude Code:claude-opus-5 — reviewed before posting.

Hey thank you so much. that's very small number and serious regression for wasm

@JMLX42

@JMLX42 Thanks! I opened a PR including your patches! Sorry to be late!

Give the strong count the bits an unused weak field held (RustPython#…

Give the strong count the bits an unused weak field held (#8574)

Navigation Menu

Repository navigation

Reference-count ceiling is 32 767 on 32-bit targets: refcount_overflow aborts the process #8469

Reproducer

Cause

Why this is reachable in ordinary use, not a synthetic limit

What a fix has to preserve

The weak half of the word has never counted anything

Widening the strong count needs no type change

Tested on the target the bug lives on

One correction to this issue's scope

Correction: widening the field alone is not enough — the accessor truncates

What I missed

The fix

Why it did not show up in testing

Branch state

Question: how would you want the 2³² boundary tested, if at all?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Reference-count ceiling is 32 767 on 32-bit targets: refcount_overflow aborts the process #8469

Description

Reproducer

Cause

Why this is reachable in ordinary use, not a synthetic limit

What a fix has to preserve

Activity

JMLX42 commented on Aug 8, 2026

The weak half of the word has never counted anything

Widening the strong count needs no type change

Tested on the target the bug lives on

One correction to this issue's scope

JMLX42 commented on Aug 8, 2026

Correction: widening the field alone is not enough — the accessor truncates

What I missed

The fix

Why it did not show up in testing

Branch state

JMLX42 commented on Aug 8, 2026

Question: how would you want the 2³² boundary tested, if at all?

youknowone commented on Aug 22, 2026

youknowone commented on Aug 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions