mpremote mip install: Better support for natmods, substitute arch/version specific info - #19478

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.51%. Comparing base (7b6130a) to head (b51154f).

@@            Coverage Diff             @@
##           master   #19478      +/-   ##
==========================================
- Coverage   98.55%   98.51%   -0.04%     
==========================================
  Files         179      179              
  Lines       23243    23243              
==========================================
- Hits        22907    22899       -8     
- Misses        336      344       +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Code size report:

Reference:  nrf: Enable machine.mem_backup via POWER GPREGRET registers. [7b6130a]
Comparison: docs: Cross-link documentation about natmods. [merge of b51154f]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
      esp32:    +0 +0.000% ESP32_GENERIC
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

I believe you forgot about https://github.com/micropython/micropython/blob/master/docs/reference/mpyfiles.rst#architecture-specific-flags.

@agatti good point. How do I configure those flags for a natmod build? And how are/should they be serialized in a path?

That's been introduced as part of the natmod toolchain in b87d73f - tl;dr: set ARCH_FLAGS in the makefile to the value you'd pass to mpy-cross's -march-flags (eg. ARCH_FLAGS=zcmp).

Right now this is only available for RV32/RV64, but it is supposed to also cover Xtensa once a PR containing that gets through (see 1c51263).

Regarding how to serialise and encode that, it depends whether you want to get things explicit or not. Loading a natmod for the right architecture kind but with the wrong architecture flags will fail the same way as if you'd load, say, an x86 natmod on an ESP32. If you want to get things explicit then you'd have to keep track of which strings mean what value, which requires active maintenance.

Right now all flags are exposed as bitfields/variable-length integers so the quick way out of this is to encode things as either /{MPY_ARCH}_{MPY_VERSION}/ or /{MPY_ARCH}_{MPY_VERSION}_{hex(MPY_ARCH_FLAGS_VALUE)}/? You can query the architecture flags value of the target using this snippet [1], and the architecture flags encoding inside the MPY file is described here: https://github.com/micropython/micropython/blob/master/docs/reference/mpyfiles.rst#the-header. The payload's format is, however, arch-dependent and we haven't reached enough maturity and/or edge cases to warrant a change of how the values are encoded. Until MPYv7 shows up it is safe to assume things will stay as they are right now.

[1]

Thanks for the info! Hex serialization of the flags is pragmatic. We could have the marker be MPY_ARCH_FLAGS_HEX - that way if we pick another more explicit/readable serialization later, it would be easy to support without breaking compatibility?

Is the mapping between the flags reported by sys.implementation.mpy_, and the appropriate .mpy file to load 1-to-1? Or can the loader actually load multiple different modules (built with different flags)?

You may still want to handle cases where the arch flags aren't there, otherwise the vast majority of modules you'll ever see will end up having a 0 in there. For the load/reject rules, I'm afraid you'll have to write a per-architecture checker.

RV32/RV64 a module is loaded successfully if all requested extensions are supported by the interpreter (ie. (natmod_flags & runtime_flags) == natmod_flags, no capabilities means that (0 & runtime_flags) == 0 and hence it will load).

Xtensa/Xtensawin instead - whilst this is not set in stone yet - will need to have an interpreter CPU core version equal or higher than what is requested by the module, with no flags indicating the bare minimum core version (LX3 - the one in the ESP8266). In other words, max(natmod_flags, LX3) <= runtime_flags. See cafe858 (#18809).

@jonnor

My proposal would be the possibility to add to the single .json metadata file all per-arch or/and per-mpy version souces, which can be resolved by mpy, I do not like approach I need to vendor a bunch of copy-pasted json files for each arch with just a single change of url. The mpy already uses .json's and native modules often is be not a single mpy file, so .json metadata already neccessary

@agatti @jonnor

I have some doubts about this approach and propose a more broadly applicable concept that would extend to a larger list of indices and would not break the existing mip concept, although it would require major changes to the mip module. Please see my comments here #19479