Skip to content

tools/mpy_ld.py: automatically provide memset/memmove/memcpy if they are undefined#19357

Draft
dpgeorge wants to merge 2 commits into
micropython:masterfrom
dpgeorge:tools-mpy-ld-automatic-mem-funcs
Draft

tools/mpy_ld.py: automatically provide memset/memmove/memcpy if they are undefined#19357
dpgeorge wants to merge 2 commits into
micropython:masterfrom
dpgeorge:tools-mpy-ld-automatic-mem-funcs

Conversation

@dpgeorge

Copy link
Copy Markdown
Member

Summary

Native modules sometimes need memset/memmove/memcpy. This is currently handled by explicitly providing wrapper functions for these that call into the provided native glue API, namely mp_fun_table.memset_ etc.

This PR improves the situation by automatically providing wrapper/trampoline functions when they are needed by the native code.

This is WIP. It currently works only for ARM Thumb2.

Testing

Tested on PYBD_SF6 (armv7emdp arch).

Will be tested by CI.

Trade-offs and Alternatives

TBD.

Generative AI

Not used.

@dpgeorge dpgeorge added the tools Relates to tools/ directory in source, or other tooling label Jun 19, 2026
@dpgeorge

Copy link
Copy Markdown
Member Author

@agatti I thought I'd have a go at this, and it turned out pretty easy: the native linker allows direct relocation of memset/memmove, so the trampoline functions for them are very simple.

This should work on Thumb2. Just need to extend it to all other archs. Then we no longer need any special cases to handle these libc functions.

@github-actions

github-actions Bot commented Jun 19, 2026

Copy link
Copy Markdown

Code size report:

Reference:  tools/ci.sh: Add new Unix target with terse error messages. [5770ced]
Comparison: examples/natmod: Remove memset, memmove, memcpy. [merge of 534cd8d]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
      esp32:    +0 +0.000% ESP32_GENERIC
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

@agatti

agatti commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Looks interesting! For RV32/RV64 you should get by with this:

AUIPC T6, offset >> 12          # 0x00000F97 | (offset & 0xFFFFF000)
JALR  ZERO, T6, offset & 0xFFF  # 0x000F8067 | ((offset & 0xFFF) << 20)

assuming that T6 can be trashed there (it should, but you never know). There are different opcode sequences depending on the distance between PC and the offset, but this one should always work assuming a ±2GiB range. If you get stuck on RISC-V let me know and I'll look at it.

@dpgeorge dpgeorge force-pushed the tools-mpy-ld-automatic-mem-funcs branch 2 times, most recently from 5eb8663 to 8b65eee Compare June 20, 2026 05:33
@dpgeorge

Copy link
Copy Markdown
Member Author

For RV32/RV64 you should get by with this:

Thanks for the hint, but the runtime linker (in py/persistentcode.c) can only relocate machine-word entries, nothing else. So in the end I used:

auipc a3, 0
c.lw  a3, 8(a3)
c.jr  a3
<word sized absolute pointer to memset/memmove>

(where the absolute pointer is filled in at runtime upon loading the .mpy)

CI is showing that works on RV32, but fails on RV64... not sure why.

@agatti

agatti commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

I'm on mobile right now but I seem to recall that c.lw won't load the upper 32 bits of the register.

I guess you either need to use another opcode or clear the register beforehand.

@dpgeorge

Copy link
Copy Markdown
Member Author

I'm on mobile right now but I seem to recall that c.lw won't load the upper 32 bits of the register.

Ah, you're right. LW is sign-extending to 6-bits, LWU is zero-extending. I need to use LD instead on RV64.

@dpgeorge dpgeorge force-pushed the tools-mpy-ld-automatic-mem-funcs branch 4 times, most recently from 4a261d6 to 07e6a5a Compare June 20, 2026 12:36
dpgeorge added 2 commits June 20, 2026 23:15
Using a direct pointer to `mp_fun_table.memset_`.

TODO: get working on all archs.

Signed-off-by: Damien George <damien@micropython.org>
They are now provided by `mpy_ld.py`.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge dpgeorge force-pushed the tools-mpy-ld-automatic-mem-funcs branch from 07e6a5a to 534cd8d Compare June 20, 2026 13:15
@codecov

codecov Bot commented Jun 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.51%. Comparing base (5770ced) to head (534cd8d).

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #19357   +/-   ##
=======================================
  Coverage   98.51%   98.51%           
=======================================
  Files         176      176           
  Lines       22904    22904           
=======================================
  Hits        22563    22563           
  Misses        341      341           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tools Relates to tools/ directory in source, or other tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants