Skip to content

extmod/vfs_posix: Add size field to ilistdir() entries.#19275

Open
finnmglas wants to merge 1 commit into
micropython:masterfrom
finnmglas:finn/fix-18661-vfs-posix-ilistdir-size
Open

extmod/vfs_posix: Add size field to ilistdir() entries.#19275
finnmglas wants to merge 1 commit into
micropython:masterfrom
finnmglas:finn/fix-18661-vfs-posix-ilistdir-size

Conversation

@finnmglas
Copy link
Copy Markdown

Summary

Previously the POSIX ilistdir() iterator returned 3-tuples (name, type, inode), while the MCU VFS implementations (FAT, LFS, ROM) return 4-tuples that also include the entry size. This caused mpremote's ls/tree commands to display zero for all file sizes when talking to a MicroPython Unix port.

This change makes vfs_posix return 4-tuples (name, type, inode, size), matching the other VFS implementations. The size is retrieved via a stat() call on each entry, with the directory path cached in the iterator state to avoid recomputation. If stat() fails the size is reported as 0, consistent with how MCU ports treat missing size info.

Fixes #18661.

Testing

  • Standard unix port builds cleanly.
  • tests/extmod/vfs_basic.py, vfs_posix_paths.py, vfs_posix_enoent.py,
    vfs_posix_ilistdir_filter.py, and vfs_posix_ilistdir_del.py all
    pass.
  • The full tests/extmod/ suite is unaffected by this change (the only
    failure, select_poll_fd.py, also fails on master and is unrelated).
  • Verified manually that sizes are correct for files, that bytes-mode
    paths still return bytes names, that mounted vfs.VfsPosix works,
    and that a nonexistent directory still raises OSError.

Generative AI

I used generative AI tools when creating this PR, but a human has checked the
code and is responsible for the code and the description above.

Previously the POSIX ilistdir() iterator returned 3-tuples
(name, type, inode), while the MCU VFS implementations (FAT, LFS,
ROM) return 4-tuples that also include the entry size. This caused
mpremote's ls/tree commands to display zero for all file sizes when
talking to a MicroPython Unix port.

This change makes vfs_posix return 4-tuples (name, type, inode, size),
matching the other VFS implementations. The size is retrieved via a
stat() call on each entry, with the directory path cached in the
iterator state to avoid recomputation. If stat() fails the size is
reported as 0, consistent with how MCU ports treat missing size info.

Fixes micropython#18661.

Signed-off-by: Finn Glas <finn@finnmglas.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented May 28, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.47%. Comparing base (8c57717) to head (76e5560).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #19275   +/-   ##
=======================================
  Coverage   98.47%   98.47%           
=======================================
  Files         176      176           
  Lines       22845    22862   +17     
=======================================
+ Hits        22497    22514   +17     
  Misses        348      348           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions
Copy link
Copy Markdown

Code size report:

Reference:  esp32: Make a generic sdkconfig.flash_qio_80m config snippet. [4fd7295]
Comparison: extmod/vfs_posix: Add size field to ilistdir() entries. [merge of 76e5560]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:  +240 +0.028% standard
      stm32:    +0 +0.000% PYBV10
      esp32:    +0 +0.000% ESP32_GENERIC
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

@dpgeorge dpgeorge added the extmod Relates to extmod/ directory in source label May 28, 2026
@dpgeorge
Copy link
Copy Markdown
Member

Thanks for the contribution.

The main downside of this change is the performance impact: each iteration through ilistdir() result now needs to (1) allocate a long string for the full path; (2) call stat which on Linux goes through to the kernel. This penalty is paid for everyone using ilistdir on POSIX, regardless of whether they need the size or not.

Might be worth measuring the actual performance impact of this change on a large directory listing, to see how significant it is.

@finnmglas
Copy link
Copy Markdown
Author

finnmglas commented May 28, 2026

Good point, I quickly benchmarked it on x86-linux: its visible

  • At 100,000 entries on ext4, the patch adds ~64 ms of wall time per listing (39.7 ms → 103.5 ms).
  • Per-entry cost is roughly flat at ~0.4-0.8 μs across all N, which confirms the expected behaviour: one extra stat() syscall + one vstr append per entry.
  • Throughput drops from ~2,500 entries/ms to ~970 entries/ms on ext4.

data:

┌─────────┬─────────┬──────────┬─────────┬───────────┬──────────┐
│ entries │ master  │ patched  │  added  │ per-entry │ slowdown │
├─────────┼─────────┼──────────┼─────────┼───────────┼──────────┤
│     100 │   51 μs │    88 μs │   37 μs │   0.37 μs │    1.73x │
├─────────┼─────────┼──────────┼─────────┼───────────┼──────────┤
│   1,000 │  457 μs │   878 μs │  421 μs │   0.42 μs │    1.92x │
├─────────┼─────────┼──────────┼─────────┼───────────┼──────────┤
│  10,000 │ 4.25 ms │  12.6 ms │  8.3 ms │   0.83 μs │    2.96x │
├─────────┼─────────┼──────────┼─────────┼───────────┼──────────┤
│  50,000 │ 20.5 ms │  48.7 ms │ 28.2 ms │   0.56 μs │    2.37x │
├─────────┼─────────┼──────────┼─────────┼───────────┼──────────┤
│ 100,000 │ 39.7 ms │ 103.5 ms │ 63.8 ms │   0.64 μs │    2.61x │
└─────────┴─────────┴──────────┴─────────┴───────────┴──────────┘

Plots (tempfs and ext4):

  • tmpfs and ext4 land within noise of each other at these scales because the inode cache is hot; the syscall overhead dominates, not the disk read.
image

@finnmglas
Copy link
Copy Markdown
Author

Options to mitigate that perf hit

A. fstatat(dirfd(self->dir), fn, ...) instead of stat(full_path, ...) - maybe a bit faster
B. Config gate (MICROPY_VFS_POSIX_ILISTDIR_SIZE, default off) - opt-in to 4 tuple
C. Return -1 for size (per docs: "size of the file or -1 if unknown") - feels a bit off tho
D. Move the fix to mpremote only -> Cost paid only by consumers who actually need size.
E. Keep status quo

Not sure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extmod Relates to extmod/ directory in source

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unix port should return 4-tuples from os.ilistdir()

2 participants