Skip to content

gh-150114: Use get_process_memory_usage() in memory watchdog#150402

Open
vstinner wants to merge 2 commits into
python:mainfrom
vstinner:mem_watchdog
Open

gh-150114: Use get_process_memory_usage() in memory watchdog#150402
vstinner wants to merge 2 commits into
python:mainfrom
vstinner:mem_watchdog

Conversation

@vstinner
Copy link
Copy Markdown
Member

@vstinner vstinner commented May 25, 2026

In practice, this change adds Windows and FreeBSD support to the memory watchdog.

In practice, this change adds Windows and FreeBSD support to the
memory watchdog.
@vstinner
Copy link
Copy Markdown
Member Author

Example on Linux:

$ ./python -m test test_bigmem -u all -M 20G -v
== CPython 3.16.0a0 (tags/v3.15.0b1-20-g57ef2199503:57ef2199503, May 25 2026, 15:08:14) [GCC 16.1.1 20260515 (Red Hat 16.1.1-2)]
== Linux-7.0.9-204.fc44.x86_64-x86_64-with-glibc2.43 little-endian
== Python build: debug TraceRefs
== cwd: /home/vstinner/python/main/build/test_python_worker_308892æ
== CPU count: 12
== encodings: locale=UTF-8 FS=utf-8
== resources: all

Using random seed: 1234531347
0:00:00 load avg: 1.57 mem: 28.3 MiB Run 1 test sequentially in a single process
0:00:00 load avg: 1.57 mem: 28.3 MiB [1/1] test_bigmem
test_capitalize (test.test_bigmem.BytearrayTest.test_capitalize) ... 
 ... expected peak memory use: 4.0 GiB
 ... process data size: 0.5 GiB
 ... process data size: 2.4 GiB
 ... process data size: 4.0 GiB
 ... process data size: 4.0 GiB
 ... process data size: 4.0 GiB
(...)

cc @cmaloney

@vstinner
Copy link
Copy Markdown
Member Author

In practice, this change adds Windows and FreeBSD support to the memory watchdog.

macOS support is coming: see PR gh-150396.

@vstinner
Copy link
Copy Markdown
Member Author

"Tests / Windows (free-threading) / Build and test (x64, switch-case)" failed with:

FAIL: test_generator_not_under_consumer_arithmetic (test.test_profiling.test_sampling_profiler.test_blocking.TestBlockingModeStackAccuracy.test_generator_not_under_consumer_arithmetic)
Test that fibonacci_generator doesn't appear when consume_generator does arithmetic.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\test_profiling\test_sampling_profiler\test_blocking.py", line 137, in test_generator_not_under_consumer_arithmetic
    self.assertEqual(invalid_stacks, 0,
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
        f"Found {invalid_stacks}/{arithmetic_samples} invalid stacks where "
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        f"fibonacci_generator appears in the stack when consume_generator "
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        f"is on an arithmetic line. This indicates torn/inconsistent stack "
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        f"traces are being captured.")
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 639 != 0 : Found 639/4508 invalid stacks where fibonacci_generator appears in the stack when consume_generator is on an arithmetic line. This indicates torn/inconsistent stack traces are being captured.


======================================================================
ERROR: test_getwche (test.test_msvcrt.TestConsoleIO.test_getwche)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\test_msvcrt.py", line 102, in test_getwche
    self.check_getwch('getwche')
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "D:\a\cpython\cpython\Lib\test\test_msvcrt.py", line 92, in check_getwch
    self.run_in_separated_process(code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "D:\a\cpython\cpython\Lib\test\test_msvcrt.py", line 70, in run_in_separated_process
    subprocess.run(cmd, check=True, capture_output=True,
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                   creationflags=subprocess.CREATE_NEW_CONSOLE)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\cpython\cpython\Lib\subprocess.py", line 692, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['D:\\a\\cpython\\cpython\\PCbuild\\amd64t\\python_d.exe', '-c', '\nimport msvcrt\nfrom _testconsole import write_input\nwith open("CONIN$", "rb", buffering=0) as stdin:\n    write_input(stdin, b\'W[\')\n    assert msvcrt.getwche() == "å­—"\n']' returned non-zero exit status 3221225794.

======================================================================
FAIL: test_python_legacy_windows_stdio (test.test_cmd_line.CmdLineTest.test_python_legacy_windows_stdio)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\test_cmd_line.py", line 1047, in test_python_legacy_windows_stdio
    self.assertEqual(p.returncode, 0)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
AssertionError: 3221225794 != 0

3221225794 is 0xc0000142 (STATUS_DLL_INIT_FAILED) error code.

I restarted the job.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant