Skip to content

extmod, ports: add new vfs_rom_ioctl code to support partial prepare/erase, and implement on alif, samd, stm32#19348

Open
dpgeorge wants to merge 6 commits into
micropython:masterfrom
dpgeorge:extmod-vfs-rom-ioctl-partial-erase
Open

extmod, ports: add new vfs_rom_ioctl code to support partial prepare/erase, and implement on alif, samd, stm32#19348
dpgeorge wants to merge 6 commits into
micropython:masterfrom
dpgeorge:extmod-vfs-rom-ioctl-partial-erase

Conversation

@dpgeorge

Copy link
Copy Markdown
Member

Summary

At the moment, ROMFS on some ports (eg alif, samd, stm32) supports only a single large partition that must be prepared/erase in one go. This PR enhances these ports so that their ROMFS partition can be prepared/erased in blocks (down to the erase size of the underlying flash). This has two advantages:

  1. It no longer has a timeout in mpremote when deploying very large (>8MB) ROM filesystems. This timeout would occur if the prepare stage took > 10 seconds. But now it's split up into many smaller prepare calls, each taking less than 1 second, so does not timeout anymore.
  2. It opens the possibility to do partial updates of the ROMFS partition, eg support multiple sub-partitions and update them one at a time, or update the start of the partition last to protect against power loss during the update. Both of those can be important for OTA updates that go into ROMFS.

Changes in this PR:

  • add new MP_VFS_ROM_IOCTL_GET_MIN_PREPARE ioctl
  • add new 4-arg version of existing MP_VFS_ROM_IOCTL_WRITE_PREPARE ioctl
  • implement the above two on alif, samd and stm32 ports
  • update mpremote to use the new ioctls when they are supported by a target

The change is fully backwards compatible with boards that don't support the newer ioctls.

Testing

Tested on ALIF_ENSEMBLE, ADAFRUIT_ITSYBITSY_M0_EXPRESS, RPI_PICO2 and PYBD_SF6:

  • using new mpremote with old firmware (before these changes)
  • romfs query and romfs deploy
  • multiple ROMFS partitions on ALIF_ENSEMBLE
  • using old mpremote with new firmware

Trade-offs and Alternatives

This adds a small amount of extra code to ports that use the new features. But ports that don't have the space do not have to implement it.

The impact to mpremote is minimal (some code was refactored there).

Generative AI

Not used.

dpgeorge added 5 commits June 16, 2026 16:41
This allows devices to support partial update of the ROM segments.

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge dpgeorge added port-stm32 port-samd tools Relates to tools/ directory in source, or other tooling port-alif labels Jun 16, 2026
@dpgeorge

Copy link
Copy Markdown
Member Author

@kwagyeman FYI this might be useful to you.

@codecov

codecov Bot commented Jun 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.51%. Comparing base (d901e98) to head (720f797).

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #19348   +/-   ##
=======================================
  Coverage   98.51%   98.51%           
=======================================
  Files         176      176           
  Lines       22903    22903           
=======================================
  Hits        22562    22562           
  Misses        341      341           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jun 16, 2026

Copy link
Copy Markdown

Code size report:

Reference:  unix/README: Update the supported targets list. [d901e98]
Comparison: tools/mpremote: Implement incremental preparation for romfs deploy. [merge of 720f797]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
      esp32:    +0 +0.000% ESP32_GENERIC
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:   +28 +0.010% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

This uses the new GET_MIN_PREPARE ioctl to support incremental preparation
of the ROM partition.  On very large partitions the preparation can take a
long time (eg due to the erase time of many blocks) and this can lead to a
timeout of mpremote.  Using incremental preparation prevents the timeout
and gives better feedback to the user as to the progress.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge dpgeorge force-pushed the extmod-vfs-rom-ioctl-partial-erase branch from b07275b to 720f797 Compare June 16, 2026 07:41
@kwagyeman

Copy link
Copy Markdown
Contributor

@dpgeorge Can you add the mimxrt too?

@dpgeorge

Copy link
Copy Markdown
Member Author

Can you add the mimxrt too?

The mimxrt port (and others like rp2, esp32) do not need this addition, because they return a block device from the vfs.rom_ioctl(2, 0) call. And that block device allows you to read/write/erase blocks in arbitrary order.

The changes in this PR are only needed for ports that do not expose a block device for the ROMFS.

@kwagyeman

Copy link
Copy Markdown
Contributor

Mmm, would be nice to have a unified api here.

@dpgeorge

Copy link
Copy Markdown
Member Author

would be nice to have a unified api here.

That's not easy to implement. Eg on STM32 parts with ROMFS in internal flash, the page erase size is sometimes very large (eg 256kb on F767) and it's not feasible to expose that as a block device object (because you can't write 256kb at once so it'd need to support partial writes and that makes it quite complicated). So the way it's designed is:

  • in cases where the ROMFS partition is exactly within an existing block device (eg rp2, esp32 external memory-mapped flash used for code and data and filesystem) then vfs.rom_ioctl() can just return that existing block device object (instead of implemented a whole different API to access that flash)
  • in cases where the ROMFS partition is not backed by a traditional block device (eg STM32F767 internal flash) then it implements vfs.rom_ioctl() ioctl codes to erase/write the data
  • ROMFS is a feature that you probably still want on devices with limited resources, and in that case the port should be able to choose the most minimal way to implement ROMFS partition access (either a block device or rom_ioctl() codes)
  • the complexity to use the API is then pushed up to the user of vfs.rom_ioctl(), eg mpremote has logic to handle all the different cases (and this PR introduces a new scenario that needs to be handled: when partial erase may or may not be supported by the target)

In the end it's not that hard to support the different ways that it works, especially writing it in Python is pretty straightforward. I can add some docs for that (see also #19127).

If we really need to have a unified API for this... then it requires a redesign of the whole interface to come up with something that will suit all the ports and all the different ways of having writable ROM.

(Note that the situation in master already has this non-unified API. This PR does not change that but rather extends the vfs.rom_ioctl() interface to support partial erase on those targets that don't use the block protocol.)

@kwagyeman

Copy link
Copy Markdown
Contributor

Well, I can just use exceptions on returned types and name errors to have one piece of code handle everything, so, I can do that to handle everything.

@dpgeorge

Copy link
Copy Markdown
Member Author

I did some more thinking about this, and maybe it does need a bit of an overhaul to unify it, and also provide information about non-memory-mappable flash partitions that are used for filesystem only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

port-alif port-samd port-stm32 tools Relates to tools/ directory in source, or other tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants