esp32: Fix USB Serial/JTAG TX stall due to full EP buffer - #19656

Summary

Closes #19651.

I wasn't able to reproduce using standalone reproducer from this issue, but I was able to reproduce by installing micropython-stubber==1.28.3 and running stubber mcu --format py -s /dev/ttyACM0 as per issue description.

This failure is also visible by running ./tests/serial_test.py on the Espressif USB serial/jtag port. The echo test fails when the length is 64 bytes (a clue!) and is out of sync from then on:

❯ ./serial_test.py -t /dev/ttyACM0
REPL and data on /dev/ttyACM0
DATA ECHO: FAIL     
  sent 1 b'z'
  echo match
<snip matching iterations>
  sent 62 b'tgytyJXgyyXJHtHygJXHXyXyayMttgXJtyaXtagMtJXtatyygaggJygMJJMJyt'
  echo match
  sent 63 b'LeLcFFdL9dcFdeKed9K9FFddLFceeFK9KFcF9LeFeKF9dcccF9cdFFcecKLdeeL'
  echo match
  sent 64 b'yEuEbEyBb8byw8ubEBEywbyw8bbuw8ybu8uyEyB8uE8EyE8Bb8BBbub8EybBEu88'
  echo 64 b''
  sent 65 b'lMMWtklrkrl33l3kMWrrr3ll3rrrk3r3MrlltkWOOrWrkrrWMrkk3ttlOt33kO3rM'
  echo 65 b'yEuEbEyBb8byw8ubEBEywbyw8bbuw8ybu8uyEyB8uE8EyE8Bb8BBbub8EybBEu88l'
  sent 66 b'u1uw11w1FFsVVVQuIwsuFI1uuVsFwuuIVuQsIw1wwIFQQwIssVuFwuIFQFuuuuwwsw'
  echo 66 b'MMWtklrkrl33l3kMWrrr3ll3rrrk3r3MrlltkWOOrWrkrrWMrkk3ttlOt33kO3rMu1'
<snip failing iterations>

The Espressif USB Serial/JTAG peripheral (USJ) manages a 64 byte TX FIFO of bytes to send to the host (via the IN endpoint of the USB device). This TXFIFO automatically flushes whenever it is full, but if exactly 64 bytes (the endpoint length) is queued then it expects a manually triggered txfifo flush after it empties, in order to send a ZLP (zero length packet) to the host to signal that the transfer is done. Otherwise the host will assume there is more data coming and wait.

The bug is caused by this sequence of evens:

1. USJ has no data to send to host (TXFIFO/IN EP), so the SERIAL_IN_EMPTY interrupt bit is set.
2. MicroPython sends exactly 64 bytes to host, queued into the TXFIFO. The hardware will start flushing to the host.
3. The SERIAL_IN_EMPTY interrupt is enabled, but because the bit was already set it triggers.
4. ISR calls txfifo_flush() but this is a no-op as the TXFIFO is already being flushed.
5. ISR is now disabled until the next time MicroPython tries to send data.
6. The hardware sends the 64 bytes to the USB host.

Because there is exactly 64 bytes in the buffer (i.e. EP size), the USB host will assume there is more data to come and will wait for either a ZLP or more data. Because the interrupt is disabled this won't happen, and that data won't be received on the host until the next time MicroPython tries to TX serial data to the host.

Fix is to clear the SERIAL_IN_EMPTY interrupt status to avoid stale interrupts, and to be explicit about when we flush immediately on send versus when we need to wait for the TXFIFO to empty.

(Debugging tools used to figure this out were the Linux usbmon kernel module with Wireshark, and the "grabserial" tool to compare timings on the traditional UART - via grabserial -d /dev/ttyUSB0 -b 115200 --hex-ascii -T.)

As per the issue report, this is not a v1.29 regression (double checked by reproducing on ESP32_GENERIC_C3, MicroPython v1.28, and IDF v5.5.1.)

Test fix included

Second commit is a fix for the serial_test.py program when testing the USB Serial/JTAG Peripheral, as the DATA IN test would time out waiting for a soft reset when starting the next test in the sequence. This is because MicroPython flushes the plain UART peripheral on soft reset, and this could take more than the 0.1 seconds that was allowed for (for the 16384 byte test it could take a couple of seconds).

Testing

• Ran the serial_test.py program, the "stubber mcu" test on ESP32_GENERIC_C3, ESP32_GENERIC_C5 and ESP32_GENERIC_P4 boards via the USB Serial/JTAG port. All three boards failed both tests without this fix, pass with this fix.
• Also ran tools/mpremote/tests/run-mpremote-tests.py on all three boards. Pass before and after this fix.

Trade-offs and Alternatives

• It'd probably be more convenient to use the ESP-IDF driver for the USB Serial/JTAG peripheral, however this driver is synchronous only (there's no way to get a callback when data is received from the host). Also it has its own buffer layer, so we'd end up with double buffering on stdin (the ESP-IDF driver buffer and then MicroPython's stdin ringbuffer).

Generative AI

I did not use generative AI tools when creating this PR.