Skip to content

gh-151378: Fix unbounded memory growth when recording binary and the thread is sleeping#152089

Open
maurycy wants to merge 5 commits into
python:mainfrom
maurycy:rle-overlow-fixed
Open

gh-151378: Fix unbounded memory growth when recording binary and the thread is sleeping#152089
maurycy wants to merge 5 commits into
python:mainfrom
maurycy:rle-overlow-fixed

Conversation

@maurycy

@maurycy maurycy commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Fixes #151378

I've described the root cause in #151378.

Instead of adding some LIMIT and checking it, I'm doing two tricks here that spare us one struct and some complexity: using a fixed buffer (lazy alloc, so threads do not pay any price if there's no repeats), instead of a dynamic array, and eager encoding, instead of having PendingRLESample.

Truth be told, it's not easy to write a test. We currently have no test for any OverflowError

(No) reproduction

The same as in #151378

2026-06-24T15:42:07.243116000+0200 maurycy@gimel /Users/maurycy/work/cpython (rle-overlow-fixed ee78d43+) % ./python.exe -c "
import time
time.sleep(3600)
" & TARGET=$!

sudo ./python.exe -m profiling.sampling attach --binary -r 1000khz -d 300 -o /tmp/rle.bin $TARGET &
sleep 2; PROF=$(pgrep -fn "profiling.sampling attach")
for i in $(seq 5); do printf "t=%2dmin  RSS=%d MB\n" $i $(($(ps -o rss= -p $PROF|tr -d ' ')/1024)); sleep 60; done
[1] 65625
[2] 65626
t= 1min  RSS=32 MB
t= 2min  RSS=33 MB
t= 3min  RSS=33 MB
t= 4min  RSS=33 MB
t= 5min  RSS=33 MB
Captured 99,316,092 samples in 300.00 seconds
Sample rate: 331,053.64 samples/sec
Error rate: 0.00
  Binary Encoding:
    Records:          12,131
      RLE repeat:     12,130 (100.0%) [99,308,159 samples]
      Full stack:     1 (0.0%)
      Suffix match:   0 (0.0%)
      Pop-push:       0 (0.0%)
  Frame Efficiency:
    Frames written:   1
    Frames saved:     99,308,159 (100.0%)
    Bytes (pre-zstd): 189.6 MB
Warning: missed 200683909 samples from the expected total of 300000001 (66.89%)
Binary profile written to /tmp/rle.bin (99316092 samples)

[2]  + done       sudo ./python.exe -m profiling.sampling attach --binary -r 1000khz -d 300 -o 
2026-06-24T15:47:52.516465000+0200 maurycy@gimel /Users/maurycy/work/cpython (rle-overlow-fixed ee78d43+) % ./python.exe -VV
Python 3.16.0a0 (heads/main-dirty:ee78d4323c1, Jun 24 2026, 12:32:26) [Clang 21.0.0 (clang-2100.1.1.101)]

@maurycy maurycy changed the title gh-151378: Stop unbounded memory growth when recording binary and the thread is sleeping gh-151378: Fix unbounded memory growth when recording binary and the thread is sleeping Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

_remote_debugging: unbound memory growth when recording binary and the thread is sleeping

1 participant