natmod: Do not share build directory across architectures. by agatti · Pull Request #19339 · micropython/micropython

agatti · 2026-06-14T11:12:05Z

Summary

This PR modifies the native modules build system to use different directories for different architectures.

Previously, the natmod build system would unconditionally put all build data into a single directory (usually "$(PWD)/build" unless overridden), and the static library runtime cache into another directory ("$(PWD)/.mpy_ld_cache"). That works if building is always done for a single architecture or if a full clean is performed before switching architectures, which sometimes is not wanted (collecting the runtime cache may take quite some time depending on the module complexity, for example).

With these changes, both the build directory and the runtime cache directories are marked with the name of the architecture they target, so for example "$(PWD)/build" becomes "$(PWD)/build_x64" (the same happens to the cache directory name).

That does not come without drawbacks unless the build directory name is overridden in the makefile. MPY files as part of their preambles will contain the path name provided to mpy_ld.py as the first entry in the constant string pool, meaning the output files will be a tiny bit larger. To offset this, a new option was added to mpy_ld.py, --internal-filename which - if passed - will write the given string in the module file instead of the default path name. If said option is not provided, generated MPY files will still carry the raw command line output path, maintaining the same behaviour as before these changes.

That also lets advanced users shrink their MPY files even further (eg. by using single letter names) and/or hide potentially sensitive information provided by the path name if a custom directory is used and said directory contains things like product code names or client names.

This also closes #6585.

Testing

Besides manual testing performed both for native Unix and QEMU targets, this is should also be tested by CI.

Trade-offs and Alternatives

Users building a certain module for just one architecture should probably look into overriding the BUILD variable if a longer MPY file name is a deal-breaker, or provide their own shorter file name now that the possibility is there.

Right now mpy-tool.py does not have such a functionality for already-built MPY files. I had a quick look at it, but it may require more modifications than what I'm willing to add to this PR.

Generative AI

I did not use generative AI tools when creating this PR.

codecov · 2026-06-14T11:16:38Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.50%. Comparing base (d901e98) to head (f20c234).

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #19339      +/-   ##
==========================================
- Coverage   98.51%   98.50%   -0.01%     
==========================================
  Files         176      176              
  Lines       22903    22903              
==========================================
- Hits        22562    22561       -1     
- Misses        341      342       +1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-06-14T11:29:41Z

Code size report:

Reference:  unix/README: Update the supported targets list. [d901e98]
Comparison: tools/mpy_ld.py: Allow overriding the internal MPY file name. [merge of f20c234]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
      esp32:    +0 +0.000% ESP32_GENERIC
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

octoprobe-bot · 2026-06-14T12:47:26Z

Octoprobe PR report

Duration: 1h 3min, 57706 tests run, 66ms/test
Tested ports: esp32 esp8266 mimxrt nrf rp2 samd stm32
Summary Report , Logdirectory

Test	Tests passed	Tests skipped	Tests xfailed	Tests failed
format flash	10	8
run-tests.py	16645	2140	9
run-tests.py --via-mpy --emit native	16402	2374	16	2
run-tests.py --via-mpy	16645	2141	8
run-perfbench.py	400	8
run-natmodtests.py	592	107	7
run-tests.py --test-dirs=extmod_hardware	57	89	25	9
run-tests.py --test-dirs=extmod_hardware --emit-native	58	87	23	2
Total	50809	6954	88	13

Failures

Group: run-tests.py --test-dirs=extmod_hardware

Test	esp32 0c30- ESP32_C3_DEVKIT	esp32 5c34- ESP32_C3_DEVKIT	esp32 5d21- ESP32_DEVKIT	esp32 472b- ESP32_S3_DEVKIT	esp32 1830- LOLIN_C3_MINI	mimxrt 1133- TEENSY40	nrf 3c2a- ARDUINO_NANO_33	rp2 5334- RPI_PICO2	rp2 5334- RPI_PICO2- RISCV	rp2 552b- RPI_PICO2_W	rp2 5f2c- RPI_PICO_W	rp2 6038- RPI_PICO_W	samd 5f2a- ADA_ITSYBITSY_M0	stm32 2b35- NUCLEO_WB55	stm32 3a21- PYBV11	stm32 3a21- PYBV11- DP	stm32 3a21- PYBV11- DP_THREAD	stm32 3a21- PYBV11- THREAD
extmod_hardware/machine_pwm.py	_pass	FAIL	_pass	_pass	_pass	_pass	skip	_XFAIL _{xfail_master_478.json}	_XFAIL _{xfail_master_478.json}	_pass	FAIL	FAIL	_pass	_pass	_pass	_pass	_pass	_pass
extmod_hardware/machine_sdcard_dma_align.py	FAIL	FAIL	skip	skip	FAIL	skip	skip	skip	skip	skip	skip	skip	skip	skip	skip	skip	skip	skip
extmod_hardware/machine_uart_irq_break.py	_pass	FAIL	_pass	_pass	_pass	skip	skip	_XFAIL _{xfail_master_478.json}	_XFAIL _{xfail_master_478.json}	_pass	_pass	_pass	skip	skip	skip	skip	skip	skip
extmod_hardware/machine_uart_irq_rx.py	_pass	FAIL	_pass	_pass	_pass	skip	_XFAIL _{xfail_master_478.json}	skip	skip	skip	skip	skip	_pass	_pass	_pass	_pass	_pass	_pass
extmod_hardware/machine_uart_irq_rxidle.py	_pass	FAIL	_pass	_pass	_pass	_pass	skip	_XFAIL _{xfail_master_478.json}	_XFAIL _{xfail_master_478.json}	_pass	_pass	_pass	_pass	_pass	_pass	_pass	_pass	_pass

Group: run-tests.py --test-dirs=extmod_hardware --emit-native

Test	esp32 0c30- ESP32_C3_DEVKIT	esp32 5d21- ESP32_DEVKIT	esp32 472b- ESP32_S3_DEVKIT	esp32 1830- LOLIN_C3_MINI	mimxrt 1133- TEENSY40	nrf 3c2a- ARDUINO_NANO_33	rp2 5334- RPI_PICO2	rp2 5334- RPI_PICO2- RISCV	rp2 552b- RPI_PICO2_W	rp2 5f2c- RPI_PICO_W	rp2 6038- RPI_PICO_W	samd 5f2a- ADA_ITSYBITSY_M0	stm32 2b35- NUCLEO_WB55	stm32 3a21- PYBV11	stm32 3a21- PYBV11- DP	stm32 3a21- PYBV11- DP_THREAD	stm32 3a21- PYBV11- THREAD
extmod_hardware/machine_sdcard_dma_align.py	FAIL	skip	skip	FAIL	skip	skip	skip	skip	skip	skip	skip	skip	skip	skip	skip	skip	skip

Group: run-tests.py --via-mpy --emit native

Test	esp32 0c30- ESP32_C3_DEVKIT	esp32 5c34- ESP32_C3_DEVKIT	esp32 5d21- ESP32_DEVKIT	esp32 472b- ESP32_S3_DEVKIT	esp32 1830- LOLIN_C3_MINI	mimxrt 1133- TEENSY40	nrf 3c2a- ARDUINO_NANO_33	rp2 5334- RPI_PICO2	rp2 5334- RPI_PICO2- RISCV	rp2 552b- RPI_PICO2_W	rp2 5f2c- RPI_PICO_W	rp2 6038- RPI_PICO_W	samd 5f2a- ADA_ITSYBITSY_M0	stm32 2b35- NUCLEO_WB55	stm32 3a21- PYBV11	stm32 3a21- PYBV11- DP	stm32 3a21- PYBV11- DP_THREAD	stm32 3a21- PYBV11- THREAD
micropython/viper_ptr8_load_boundary.py	_pass	_pass	_pass	_pass	_pass	_pass	_pass	_pass	_pass	_pass	_pass	_pass	FAIL	_pass	_pass	_pass	_pass	_pass
stress/qstr_limit.py	_pass	_pass	_pass	_pass	_pass	_pass	_pass	_pass	_pass	_pass	_pass	_pass	FAIL	_pass	_pass	_pass	_pass	_pass

This commit modifies the native modules build system to use different directories for different architectures. Previously, the natmod build system would unconditionally put all build data into a single directory (usually "$(PWD)/build" unless overridden), and the static library runtime cache into another directory ("$(PWD)/.mpy_ld_cache"). That works if building is always done for a single architecture or if a full clean is performed before switching architectures, which sometimes is not wanted (collecting the runtime cache may take quite some time depending on the module complexity, for example). With these changes, both the build directory and the runtime cache directories are marked with the name of the architecture they target, so for example "$(PWD)/build" becomes "$(PWD)/build_x64" (the same happens to the cache directory name). Signed-off-by: Alessandro Gatti <a.gatti@frob.it>

This commit makes a minor change to the native module build scripts, to make the intermediate MPY file keep the same file name it will have once the final pre-processing step takes place. A MPY file will retain its file name as the first QSTR entry in the file, and before these changes the name being written would be the one used for the intermediate MPY output file (ie. "$BUILDDIR/$MODNAME.native.mpy"). The `mpy-tool.py` preprocessing pass would keep that string intact but rename the file, so things wouldn't match in error tracebacks. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>

This commit adds an option to the MPY linker to brand generated file with a custom internal file name rather than the raw output path provided in the command line. MPY files, as part of their preambles, will contain a path name as the first entry in the constant string pool. However that may contain unneeded data (ie. sometimes the plain file name is enough) or it may have sensitive data in there (if an alternate build path is provided it may contain product code names or client names which should probably not be disclosed). Until now there was no provision to let users use a different internal path name. A new option was added to `mpy_ld.py`: "--source-name" which, if passed, will write the given string in the module file. If said option is not provided, generated MPY files will still carry the raw command line output path but stripped of all leading path segments. The example natmod makefiles have been updated to always provide that option whenever the final file name on the filesystem wouldn't match the internal MPY file name. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>

agatti added the tools Relates to tools/ directory in source, or other tooling label Jun 14, 2026

dpgeorge reviewed Jun 16, 2026

View reviewed changes

Comment thread py/dynruntime.mk Outdated

Comment thread tools/mpy_ld.py Outdated

dpgeorge reviewed Jun 17, 2026

View reviewed changes

Comment thread tools/mpy_ld.py Outdated

agatti added 3 commits June 17, 2026 11:11

agatti force-pushed the natmod-build-multiarch branch from a59aebf to f20c234 Compare June 17, 2026 16:06

dpgeorge approved these changes Jun 19, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

natmod: Do not share build directory across architectures.#19339

natmod: Do not share build directory across architectures.#19339
agatti wants to merge 3 commits into
micropython:masterfrom
agatti:natmod-build-multiarch

agatti commented Jun 14, 2026

Uh oh!

codecov Bot commented Jun 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 14, 2026 •

edited

Loading

Uh oh!

octoprobe-bot commented Jun 14, 2026 •

edited

Loading

Group: run-tests.py --test-dirs=extmod_hardware

Group: run-tests.py --test-dirs=extmod_hardware --emit-native

Group: run-tests.py --via-mpy --emit native

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

agatti commented Jun 14, 2026

Summary

Testing

Trade-offs and Alternatives

Generative AI

Uh oh!

codecov Bot commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

octoprobe-bot commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Octoprobe PR report

Group: run-tests.py --test-dirs=extmod_hardware

Group: run-tests.py --test-dirs=extmod_hardware --emit-native

Group: run-tests.py --via-mpy --emit native

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Jun 14, 2026 •

edited

Loading

github-actions Bot commented Jun 14, 2026 •

edited

Loading

octoprobe-bot commented Jun 14, 2026 •

edited

Loading