Skip to content

py/objarray: Avoid double zero init on sized bytearrays.#18771

Closed
Gadgetoid wants to merge 2 commits into
micropython:masterfrom
pimoroni:patch-bytearray-memset
Closed

py/objarray: Avoid double zero init on sized bytearrays.#18771
Gadgetoid wants to merge 2 commits into
micropython:masterfrom
pimoroni:patch-bytearray-memset

Conversation

@Gadgetoid

@Gadgetoid Gadgetoid commented Feb 4, 2026

Copy link
Copy Markdown
Contributor

As per the implementation of m_malloc0, if MICROPY_GC_CONSERVATIVE_CLEAR is set then all RAM is guaranteed to be zero-init by gc_alloc.

py/objarray.c: Guard the explicit zero init in bytearray_make_new against being run, initialising the RAM to zero a second time, if this flag is set.

py/objstr.c: Guard the explicit zero init in bytes_make_new against being run, initialising the RAM to zero a second time, if this flag is set.

Note that MICROPY_GC_CONSERVATIVE_CLEAR is default enabled by MICROPY_ENABLE_GC, and no ports currently override this value.

We stumbled across this inconsistency when testing the timings of allocating, collecting and collecting/freeing PSRAM, with a script looking something like the following:

import gc
import time

t_start = time.ticks_us()
gc.collect()
t_taken = time.ticks_diff(time.ticks_us(), t_start)
baseline = t_taken / 1000
print(f"Baseline: {t_taken / 1000:.2f}ms")


for y in range(1, 41):
    b4 = gc.mem_free()
    t_start = time.ticks_us()
    test = bytearray(1024 * y * 10)
    alloc_time_ms = time.ticks_diff(time.ticks_us(), t_start) / 1000
    af = gc.mem_free()

    t_start = time.ticks_us()
    gc.collect()
    t_taken = time.ticks_diff(time.ticks_us(), t_start)

    ram_used = (b4 - af) / 1024
    time_ms = t_taken / 1000
    factor = (time_ms - baseline) / ram_used

    del test
    
    t_start = time.ticks_us()
    gc.collect()
    collect_time_ms =  time.ticks_diff(time.ticks_us(), t_start) / 1000
    print(f"{int(ram_used):4d} {time_ms: 7.2f} {collect_time_ms: 7.2f} {alloc_time_ms: 7.2f} {factor: 6.2f}")

We had a custom image class which did its own allocation via m_malloc internally, and found - quite unintentionally - that it was twice as fast to allocate our custom image versus an identically sized bytearray. A little sleuthing revealed that bytearray ignores the value of MICROPY_GC_CONSERVATIVE_CLEAR and will zero (using memset()) RAM that is already guaranteed to be zeroed by gc_alloc, eg:

micropython/py/objarray.c

Lines 190 to 196 in cef2538

} else if (mp_obj_is_int(args[0])) {
// 1 arg, an integer: construct a blank bytearray of that length
mp_uint_t len = mp_obj_get_int(args[0]);
mp_obj_array_t *o = array_new(BYTEARRAY_TYPECODE, len);
memset(o->items, 0, len);
return MP_OBJ_FROM_PTR(o);
} else {

when gc_alloc already does:

micropython/py/gc.c

Lines 885 to 888 in edab53c

#if MICROPY_GC_CONSERVATIVE_CLEAR
// be conservative and zero out all the newly allocated blocks
memset((byte *)ret_ptr, 0, (end_block - start_block + 1) * BYTES_PER_BLOCK);
#else

Testing

To ensure no non-zero values leaked into bytearrays I ran this very basic test with various maximum sizes:

for s in range(1024):
    if sum(bytearray(s)) > 0:
        raise RuntimeError(f"{s}: FAIL! - Non zero init bytearray found.")

And here are the bytes(n) alloc times, on PSRAM with an RP2350 clocked to 200MHz both before and after the change`:

size (kb) before (ms) after (ms)
10 2.05 1.59
20 5.25 3.75
30 9.25 6.12
40 12.87 8.36
50 16.31 10.53
60 19.69 12.67
70 23.01 14.81
80 26.33 16.92
90 29.63 19.03
100 32.92 21.14
110 36.22 23.25
120 39.50 25.36
130 42.79 27.47
140 46.09 29.58
150 49.36 31.68
160 52.64 33.80
170 55.95 35.89
180 59.23 38.02
190 62.51 40.10
200 65.80 42.22

Trade-offs and Alternatives

I am unsure if there is some subtle caveat for gc_alloc zero init that might cause problems specifically with a bytearray, but afaict this change is minimal and gives us bytearray allocations in roughly half the time.

This is less noticeable on SRAM (tested on RP2350) where a 200k allocation takes 2.6ms, but on PSRAM it brings a 200k bytearray alloc from roughly 50ms down to 25ms, and a modest 10k alloc from 2ms down to 1ms. The alloc time for sizes between these two extremes is linear.

As per the implementation of m_malloc0, if
MICROPY_GC_CONSERVATIVE_CLEAR is set then all RAM is guaranteed to be
zero-init by gc_alloc.

py/objarray.c: Guard the explicit zero init in bytearray_make_new
against being run, initialising the RAM to zero a second time, if this
flag is set.

Note that MICROPY_GC_CONSERVATIVE_CLEAR is default enabled by
MICROPY_ENABLE_GC, and no ports currently override this value.

Co-authored-by: Mike Bell <mdb036@gmail.com>
Signed-off-by: Phil Howard <github@gadgetoid.com>
@codecov

codecov Bot commented Feb 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.41%. Comparing base (cef2538) to head (65db7d7).
⚠️ Report is 25 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #18771      +/-   ##
==========================================
- Coverage   98.41%   98.41%   -0.01%     
==========================================
  Files         171      171              
  Lines       22324    22322       -2     
==========================================
- Hits        21971    21969       -2     
  Misses        353      353              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Feb 4, 2026

Copy link
Copy Markdown

Code size report:

Reference:  tests/target_wiring/README: Add README describing target_wiring specs. [cef2538]
Comparison: py/objstr: Avoid double zero init on sized bytes. [merge of 65db7d7]
  mpy-cross:   -32 -0.008% 
   bare-arm:    +0 +0.000% 
minimal x86:   -12 -0.006% 
   unix x64:   -64 -0.007% standard
      stm32:   -32 -0.008% PYBV10
      esp32:   -24 -0.001% ESP32_GENERIC
     mimxrt:   -40 -0.011% TEENSY40
        rp2:   -40 -0.004% RPI_PICO_W
       samd:   -32 -0.012% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:   -46 -0.010% VIRT_RV32

@Gadgetoid

Copy link
Copy Markdown
Contributor Author

Quick eyeball over the MicroPython codebase turns up

memset(o, 0, sizeof(*o));
as another redundant memset, though its impact would be negligible.

There's also

micropython/py/objstr.c

Lines 284 to 285 in cef2538

vstr_init_len(&vstr, len);
memset(vstr.buf, 0, len);
which is redundant since vstr_init_len calls vstr_init which does vstr->buf = m_new(char, vstr->alloc); This is what underpins bytes(n) and, as such, it has exactly the same issue as bytearray(n).

As per the implementation of m_malloc0, if
MICROPY_GC_CONSERVATIVE_CLEAR is set then all RAM is guaranteed to be
zero-init by gc_alloc.

py/objstr.c: Guard the explicit zero init in bytes_make_new
against being run, initialising the RAM to zero a second time, if this
flag is set.

Signed-off-by: Phil Howard <github@gadgetoid.com>
@dpgeorge dpgeorge added the py-core Relates to py/ directory in source label Feb 4, 2026
@dpgeorge

dpgeorge commented Feb 4, 2026

Copy link
Copy Markdown
Member

Nice find!

@dpgeorge dpgeorge left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, thanks.

(What we really should do is remove the MICROPY_GC_CONSERVATIVE_CLEAR option. It shouldn't be necessary. Someone needs to find time to investigate why that option was added in the first place, as per the comment in py/mpconfig.h.)

@dpgeorge

dpgeorge commented Feb 5, 2026

Copy link
Copy Markdown
Member

Someone needs to find time to investigate why that option was added in the first place, as per the comment in py/mpconfig.h.

See #2195.

@dpgeorge

dpgeorge commented Feb 5, 2026

Copy link
Copy Markdown
Member

Rebased and merged in 53d9004 and 1095bbb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

py-core Relates to py/ directory in source

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants