Skip to content

esp32: support IDF v5.5.4 and make it the recommended version#19149

Open
dpgeorge wants to merge 4 commits into
micropython:masterfrom
dpgeorge:esp32-update-idf-5-5-4
Open

esp32: support IDF v5.5.4 and make it the recommended version#19149
dpgeorge wants to merge 4 commits into
micropython:masterfrom
dpgeorge:esp32-update-idf-5-5-4

Conversation

@dpgeorge
Copy link
Copy Markdown
Member

Summary

This PR updates the esp32 port to support IDF v5.5.4, tests that version in CI, and makes it the recommended version.

Also includes a minor related fix to extmod/nimble/modbluetooth_nimble.c.

Building all generic boards against v5.5.1 and v5.5.4 using ports/esp32/tools/metrics_esp32.py gives:

BOARD BOARD_VARIANT IDF Version Binary Size Static IRAM Size Static DRAM Size
ESP32_GENERIC v5.5.1 1699227 120215 54512
v5.5.4 1716987 (+17760/1.0) 115807 (-4408/-3.7) 54836 (+324/0.6)
ESP32_GENERIC D2WD v5.5.1 1396911 112279 53684
v5.5.4 1408791 (+11880/0.9) 108639 (-3640/-3.2) 54016 (+332/0.6)
ESP32_GENERIC SPIRAM v5.5.1 1545911 117639 54168
v5.5.4 1541203 (-4708/-0.3) 113715 (-3924/-3.3) 54516 (+348/0.6)
ESP32_GENERIC_C2 v5.5.1 1508902 42180 38344
v5.5.4 1505818 (-3084/-0.2) 40620 (-1560/-3.7) 38740 (+396/1.0)
ESP32_GENERIC_C3 v5.5.1 1660062 93452 41888
v5.5.4 1673134 (+13072/0.8) 89524 (-3928/-4.2) 41552 (-336/-0.8)
ESP32_GENERIC_C5 v5.5.1 1839392 64866 49060
v5.5.4 1867752 (+28360/1.5) 62358 (-2508/-3.9) 49556 (+496/1.0)
ESP32_GENERIC_C6 v5.5.1 1867280 91542 46252
v5.5.4 1890976 (+23696/1.3) 88316 (-3226/-3.5) 46152 (-100/-0.2)
ESP32_GENERIC_P4 v5.5.1 1559176 97222 27360
v5.5.4 1573448 (+14272/0.9) 101672 (+4450/4.6) 28680 (+1320/4.8)
ESP32_GENERIC_P4 C6_WIFI v5.5.1 1733912 97620 36144
v5.5.4 1765140 (+31228/1.8) 101972 (+4352/4.5) 31680 (-4464/-12.4)
ESP32_GENERIC_S3 v5.5.1 1689663 112575 49492
v5.5.4 1707031 (+17368/1.0) 105999 (-6576/-5.8) 49152 (-340/-0.7)
ESP32_GENERIC_S3 SPIRAM_OCT v5.5.1 1693119 115215 50312
v5.5.4 1710423 (+17304/1.0) 108643 (-6572/-5.7) 49928 (-384/-0.8)

Firmware size increases on almost all SoCs by quite a bit, around 20k. But it's very nice to see IRAM usage decrease by about 4k on everything except ESP32-P4.

This metrics_esp32.py script is also updated with changes I needed to make to get it to generate the above table.

Testing

Tested with ESP32_GENERIC firmware, running as many tests as possible from the test suite. The only regressions are:

  • tests/multi_espnow/70_channel.py seems to always fail (between two ESP32's at least) with data sent to a wrong channel being received
  • tests/multi_bluetooth/ble_gap_device_name.py fails with a timeout near the end of the test when run against PYDF_SF6 (but ESP32 <-> ESP32 passes this test)

TODO: test some other SoCs.

Trade-offs and Alternatives

This increases code size, but there's not much we can do about that, we need to keep supporting the latest IDF.

I needed to set CONFIG_BT_NIMBLE_STATIC_TO_DYNAMIC=n because otherwise BLE crashes upon start up. Maybe in the future we can fix this and enable this option to save some RAM.

I needed to add a small hack to cmake to manually include a missing compile definition to the qstr extraction phase: list(APPEND MICROPY_CPP_DEF_EXTRA "ESP_PLATFORM"). I'm not sure why this (and a few other definitions) are missing from qstr extraction, but without this one the NimBLE code fails to compile because it picks up the wrong configuration header (the default one instead of the ESP specific config header).

Generative AI

I did not use generative AI tools when creating this PR.

They are provided in a private header that is already included.

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge dpgeorge added this to the release-1.29.0 milestone Apr 24, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 24, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.46%. Comparing base (8a56be6) to head (00839cd).

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #19149   +/-   ##
=======================================
  Coverage   98.46%   98.46%           
=======================================
  Files         176      176           
  Lines       22811    22811           
=======================================
  Hits        22460    22460           
  Misses        351      351           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@robert-hh
Copy link
Copy Markdown
Contributor

Wit esp_idf 5.5.4 it should be possible to drop the call to i2c_master_probe() in machine_i2c.c. As far as I followed the change notes, the required change to i2c_master_transmit_receive() was made already in v5.5.2. I'll test.

Signed-off-by: Damien George <damien@micropython.org>
@robert-hh
Copy link
Copy Markdown
Contributor

No pure joy. A zero length I2C write still causes an error message to be printed. But at least a NAK does not cause an error any more. So I can change the code to use i2c_master_probe() only for i2c.scan() (Zero length transactions). Then the additional probe for read/write of length > 0 are gone.

@robert-hh
Copy link
Copy Markdown
Contributor

The code can now use i2c_master_execute_defined_operations() for 0-length writes, used by i2c.scan(). For other read/writes i2c_master_transmit() and i2c_master_receive() can be used. i2c_master_execute_defined_operations() behaves odd for receive with len=0. For now a receive with len=0 is silently discarded.

@dpgeorge
Copy link
Copy Markdown
Member Author

OK, thanks for testing with I2C. I'll see what I can do there.

@robert-hh
Copy link
Copy Markdown
Contributor

I prepared a change already, but wanted to wait until this PR is merged.

@dpgeorge
Copy link
Copy Markdown
Member Author

I prepared a change already, but wanted to wait until this PR is merged.

Ah, very good, thank you! I'll leave it to you then.

@DvdGiessen
Copy link
Copy Markdown
Contributor

I needed to set CONFIG_BT_NIMBLE_STATIC_TO_DYNAMIC=n because otherwise BLE crashes upon start up. Maybe in the future we can fix this and enable this option to save some RAM.

I've already looked into this a bit the last few weeks. I think it's possible and had it working, although I've currently disabled it in my tree again after I ran into other Bluetooth bugs and wanted to make sure it wasn't caused by this new feature. I'll see if I can polish that into a PR in the coming weeks.

@DvdGiessen
Copy link
Copy Markdown
Contributor

DvdGiessen commented Apr 29, 2026

Note: I've been running against v5.5.4 for a while, but I've been having a few stability problems with it. On my S3 boards I get random MMU entry fault errors which I don't get with older IDF versions (v5.5.2), as well as some issues with Bluetooth bonding. I'm actively investigating these; noting it here because that might be good to know.

@dpgeorge
Copy link
Copy Markdown
Member Author

Note: I've been running against v5.5.4 for a while, but I've been having a few stability problems with it

OK, thanks, good to know. In that case we might need to move to a lower release, eg v5.5.2. We can still support v5.5.4, but officially build with v5.5.2 (for example).

@DvdGiessen
Copy link
Copy Markdown
Contributor

In that case I'll note to avoid v5.5.3 as well, as there Bluetooth is broken (see espressif/esp-idf#18323).

@StephenDone
Copy link
Copy Markdown

From v5.5.3 onwards, the following commits have been added:

commit 32895539f2f961673094de4c92812fe0571fb142
Author: wanckl <wanlei@espressif.com>
Date:   Mon Aug 18 22:12:43 2025 +0800

    fix(driver_spi): support un-aligned dma transaction and psram transaction

commit 3924afbf28ee0283a369984ec363d0662ab06dd5
Author: Wan Lei <wanlei@espressif.com>
Date:   Tue Jul 8 19:22:54 2025 +0800

    feat(spi_master): support dma transfer with psram directly

It will be nice to be on something newer than v5.5.2 in the future, in order to take advantage of this feature, as it is quite a significant improvement for large framebuffers, as they will no longer use a bounce buffer located in internal RAM for the DMA. This is effectively a large reduction in transient internal RAM usage for projects that try to use PSRAM framebuffers, probably due to insufficient internal RAM in the first place.

@dpgeorge
Copy link
Copy Markdown
Member Author

@DvdGiessen did you make any progress with stability of 5.5.4? If not then we will move only to v5.5.2 for now (but still support v5.5.4 for users wanting to build firmware with that IDF version).

@DvdGiessen
Copy link
Copy Markdown
Contributor

Not yet. Unfortunately I don't have a small and fast reproduction, and there are more things competing for my time. But no need to block this MR; having the option to build against the newer IDF for users that don't touch or hit the bug I'm seeing is useful to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants