Skip to content

Improve mpremote socket and rfc2217 connections#19062

Draft
Josverl wants to merge 23 commits intomicropython:masterfrom
Josverl:mpr/fix_socket_rfc2217
Draft

Improve mpremote socket and rfc2217 connections#19062
Josverl wants to merge 23 commits intomicropython:masterfrom
Josverl:mpr/fix_socket_rfc2217

Conversation

@Josverl
Copy link
Copy Markdown
Contributor

@Josverl Josverl commented Apr 8, 2026

Summary

This PR fixes multiple issues with mpremote when connecting to socket and RFC2217 targets, and adds comprehensive pytest-based testing infrastructure.

possibly this PR should be split into multiple PRs:

  • mpremote fixes
  • introduction of pytest framework for mpremote + related tests
  • integration in to CI in combination with mpbridge
  • code coverage reporting for python tooling, starting with mpremote

Key improvements:

  1. Fixed RFC2217 REPL race conditions: pyserial's RFC2217 uses a background thread to process telnet data. The REPL loop's select() would signal ready when socket data arrived, but inWaiting() returned 0 because the background thread hadn't processed the data yet, causing delayed or missed input.

  2. Fixed socket connection support: Added missing in_waiting property to SerialIntercept which was causing errors when using mpremote mount over socket:// connections.

  3. Improved REPL newline handling: Fixed staircase effect when connecting to Unix port by properly configuring terminal output flags (OPOST | ONLCR) for NL → CR-NL translation.

  4. Enhanced waitchar() for network connections: Improved handling of file descriptors for socket-based connections and RFC2217 (which doesn't implement fileno()).

  5. Better error handling on disconnect: Device disconnection messages now properly display for RFC2217 and socket connections, even when command-line parameters are provided.

  6. Optimized network performance: Improved SerialTransport performance for network connections.

Testing infrastructure:

  • Added comprehensive pytest-based test suite with ~3,100 lines of tests
  • Tests for interactive REPL, Unicode handling, filesystem operations, and platform-specific scenarios
  • Support for running tests against multiple connection types (serial, socket, RFC2217)
  • Coverage collection support for both bash and pytest tests
  • Tests for edge cases including:
    • Unicode filenames and output
    • Windows console UTF-8 handling and legacy console errors
    • Filenames containing equals signs
    • REPL newline handling across platforms

Testing

Tested on the following configurations:

  • Serial connections: /dev/ttyACM0, /dev/ttyUSB0
  • Socket connections: socket://localhost:2218
  • RFC2217 connections: rfc2217://localhost:2217
  • Platforms: Linux and Windows (via WSL2 and Git Bash)

In order to enable testing on Windows - and for other tests that require some level of mocking to be created I have addedd tests using the pytest framework.

The new pytest suite includes 548+ test assertions across multiple test files covering:

  • Unicode handling test_unicode.py, test_unicode_filenames.py
  • Filesystem operations with edge cases test_equals_sign.py
  • Windows-specific console behavior test_windows_console_utf8.py
  • Test utilities validation test_utils.py
  • Interactive REPL behavior test_repl_newlines.py

To run all tests:

uv venv --python 3.14
uv pip install -e tools/mpremote[test] 

cd tools/mpremote/tests
./run-mpremote-tests.sh -t <device>

$env:MPREMOTE_DEVICE = "<device>"; pytest  -s -v

Trade-offs and Alternatives

I have chosen to use the pytest framework over stdlib's unittest as:

  • I have used pytest while developing and validating other mpremote PRs
  • pytest tests are shorter and simpler to maintain compared to unittest
  • pytest provides built-in parametrization support, allows for for testing multiple connection types (serial, socket, RFC2217) with the same test logic
  • pytest's fixture system (via conftest.py) elegantly handles complex setup/teardown of device connections and test scenarios
  • pytest can also run unittest-based tests, ensuring compatibility with any existing unittest code

I have not ported the existing bash based tests to pytest, but it should be possible to have the pytest test runner discover and run the existing "test_*.sh" files as part of as seperate PR

Generative AI

I used generative AI tools when creating this PR, but a human has checked the
code and is responsible for the code and the description above.

Josverl and others added 3 commits January 11, 2026 22:01
closes: micropython#18657

(cherry picked from commit a953024)

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
fixes: micropython#18658

(cherry picked from commit dd6e0f7)

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Adds a Modern Windows console implementation that can write
raw UTF-8 bytes directly (like POSIX)

Improve the Legacy consol to use an incremental decoder to handle
split UTF-8 sequences within writes.

fixes micropython#15228

Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 8, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.46%. Comparing base (26c1696) to head (ffbec0f).
⚠️ Report is 286 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #19062      +/-   ##
==========================================
+ Coverage   98.38%   98.46%   +0.08%     
==========================================
  Files         171      176       +5     
  Lines       22298    22802     +504     
==========================================
+ Hits        21937    22453     +516     
+ Misses        361      349      -12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 8, 2026

Code size report:

Reference:  github: Ignore codecov upload failure on forks with no token set. [b0d2072]
Comparison: mpremote/tests: Do not enable code coverage by default. [merge of ffbec0f]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
      esp32:    +0 +0.000% ESP32_GENERIC
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

Josverl and others added 19 commits April 8, 2026 23:50
RemoteCommand.wr_bytes(): Now correctly encodes strings to UTF-8 bytes
before calculating length.
Based on the suggestion by @felixdoer

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Reduces code duplication while allowing different configurations.

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Add regressions tests for utf-8 and unicode.

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
allows for skipping of some potentially destructive tests.

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
The MicroPython Unix port sends only `\n`.
With attr[1] = 0, the terminal doesn't translate `\n` to `\r\n`,
causing a staircase effect.

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
…ercept class.

This fixes an error with mounting over socket:// connections.

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
pyserial's RFC2217 uses a background thread to process telnet
data from the socket into an internal buffer. When mpremote's
REPL loop used select() on the socket, it signaled "ready" when
data arrived, but inWaiting() returned 0 because the background
thread hadn't processed the data yet.

This caused a race condition:

1. select() returns because socket has data
2. inWaiting() returns 0 (background thread hasn't processed yet)
3. REPL loop skips reading
4. Next iteration: select() returns, now inWaiting() > 0
5. Finally reads the data

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
(cherry picked from commit 1ab8ba3)
fixes micropython#15228

Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
(cherry picked from commit f104330)
Needs to be split out in more detail

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
- Update .gitignore to include coverage data files.
- Modify pyproject.toml to append coverage data during pytest runs.
- Enhance README.md with instructions for running tests with coverage.
- Update run-mpremote-tests.sh to support coverage collection.
- Add test_basic.sh and test_basic.sh.exp for device connection tests.
- Extend test_mount.sh with additional remote actions and comments.
- Improve get_devices function in test_repl_newlines.py to support MPREMOTE_DEVICE environment variable.

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
…tils module

Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
…les.

Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
@Josverl Josverl force-pushed the mpr/fix_socket_rfc2217 branch from fe618bf to 9d9637a Compare April 8, 2026 21:55
@Josverl Josverl added the tools Relates to tools/ directory in source, or other tooling label Apr 8, 2026
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
@dpgeorge
Copy link
Copy Markdown
Member

dpgeorge commented Apr 9, 2026

This is great, but it's hard to review because it does so many things at once

possibly this PR should be split into multiple PRs:

Yes! I think the first thing would be to split out the testing changes, a PR that only updates the existing tests and doesn't add any new features or new tests.

@Josverl
Copy link
Copy Markdown
Contributor Author

Josverl commented Apr 9, 2026

Absolutely. This was intended to provide insights into a possible approach to testing Mpremote in CI, and across more platforms as asked by @projectgus

  • I'll work to find smaller logical chunks, to limit the scope of change for each

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tools Relates to tools/ directory in source, or other tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants