gh-132372: Speed up logging.config existing logger handling by esadomer · Pull Request #150242 · python/cpython

esadomer · 2026-05-22T12:02:54Z

This speeds up logging.config.fileConfig() and logging.config.dictConfig() when they need to handle many existing loggers while disable_existing_loggers is active.

The old code kept the existing logger names in a sorted list, but still used repeated list membership, index(), tail scans, and remove() calls for every configured logger. This keeps the sorted list for child logger ordering, uses a set for membership/removal tracking, and uses bisect_left() to jump directly to the dotted child logger range.

Validation:

PCbuild\amd64\python_d.exe -m test test_logging -j0
PCbuild\amd64\python_d.exe Tools\patchcheck\patchcheck.py
git diff --check

Benchmark:

Loaded upstream/main:Lib/logging/config.py and the patched Lib/logging/config.py into the same Python 3.12.8 release interpreter, then timed only the dictConfig() call while resetting identical existing logger state outside the timer.

Interpreter details:

3.12.8 (tags/v3.12.8:2dc476b, Dec  3 2024, 19:30:04) [MSC v.1942 64 bit (AMD64)]
executable: C:\Users\user\AppData\Local\Programs\Python\Python312\python.exe
Py_DEBUG: None
has gettotalrefcount: False

Small n results are noisy and effectively neutral:

old n=1    mean=    24.11 us median=    20.70 us stdev=   42.46 us
new n=1    mean=    21.29 us median=    20.30 us stdev=    5.53 us
speedup n=1: 1.13x

old n=2    mean=    31.05 us median=    29.30 us stdev=    9.12 us
new n=2    mean=    32.84 us median=    29.80 us stdev=   10.27 us
speedup n=2: 0.95x

old n=3    mean=    42.36 us median=    38.70 us stdev=   11.44 us
new n=3    mean=    41.17 us median=    38.20 us stdev=   10.27 us
speedup n=3: 1.03x

Larger logger sets benefit more clearly:

old n=100  mean=  5784.59 us median=  5724.35 us stdev=  264.00 us
new n=100  mean=  4001.09 us median=  3936.50 us stdev=  255.22 us
speedup n=100: 1.45x

old n=500  mean=134692.43 us median=133170.10 us stdev= 9087.73 us
new n=500  mean= 84598.86 us median= 84082.45 us stdev= 2047.04 us
speedup n=500: 1.59x

old n=1000 mean=541349.97 us median=537541.85 us stdev=13930.13 us
new n=1000 mean=338080.61 us median=333050.55 us stdev=14305.36 us
speedup n=1000: 1.60x

eendebakpt · 2026-05-22T19:21:59Z

The performance increase is moderate, but the PR reduces code duplication so I am +1.

@esadomer On your benchmark I could get a much larger speedud with this small commit 2b9ce22 (on top of your work). I have not fully tested this, but the speedup is > 5x. If you are interested you could test it further and pull into your branch.

Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>

picnixz · 2026-05-23T17:13:21Z

I would appreciate the numbers for n=1,2,3. It's not realistic to assume that there are 500 loggers. I have honestly never seen that in production. But the bisect solution is something I would have used as well so I'm also fine with it.

picnixz · 2026-05-23T17:15:53Z

+            if not child.startswith(prefixed):
+                break
+            if child in existing_set:
+                child_loggers.add(child)


Please use a dict here instaed of a set to retain the insertion time.

picnixz · 2026-05-23T17:18:43Z

+        prefixed = name + "."
+        i = bisect_left(existing, prefixed)
+        num_existing = len(existing)
+        while i < num_existing:


An alternative here would be to use islice(existing, i) which would avoid using i+=1 but this adds an import to itertools.

picnixz · 2026-05-23T17:20:56Z

+
+def _discard_existing_logger(name, existing, existing_set, child_loggers):
+    """Discard a configured logger and record its existing children."""
+    if name in existing_set:


The function is named discard_existing_logger but we have a check "if name in existing_set", which does not make sense semantically.

It also updates child_loggers so it does 2 things. I don't have a better name though but I would prefer that you remove the "if name in existing_set" check and make it the responsibility of the caller instead. It will slightly reduce the diff and the indentation of the helper (and you save 1 function call when name is not existing)

esadomer · 2026-05-23T18:28:08Z

Thanks, I updated the branch to address the inline comments:

child_loggers now uses a dict so insertion order is retained.
The membership check moved to the callers, so _discard_existing_logger() now only handles existing logger names.
I left the islice() alternative out for now since it would add another import for a cosmetic loop change.

I also added the small n numbers to the PR description. They look effectively neutral/noisy for n=1..3:

n=1: 0.98x
n=2: 1.07x
n=3: 1.00x

Validation after the change:

PCbuild\amd64\python_d.exe -m test test_logging -j0
PCbuild\amd64\python_d.exe Tools\patchcheck\patchcheck.py
git diff --check

picnixz · 2026-05-23T18:32:37Z

Can you try a non-debug build as well? it looks like you're using a DEBUG build for the benchmarks. Having a release (PGO+LTO) build may help in removing noise in general (and show clear improvements)

picnixz · 2026-05-23T18:39:27Z

            if not isinstance(logger, logging.PlaceHolder):
-                logger.setLevel(logging.NOTSET)
+                # Equivalent to setLevel(NOTSET), but clear the cache once.
+                logger.level = logging.NOTSET


Is this change really necessary? while setLevel would clear the cache multiple times, I would prefer that we pass through logger.setLevel public API instead of changing .level directly, just in case someone has a logger subclass that does something else with setLevel.

picnixz · 2026-05-23T18:43:33Z

+    if cache_clear_needed:
+        root.manager._clear_cache()
+
+def _discard_existing_logger(name, existing, existing_set, child_loggers):


Suggested change

def _discard_existing_logger(name, existing, existing_set, child_loggers):

def _forget_existing_logger(name, existing, existing_set, child_loggers):

Maybe something like that instead (since we also update child_loggers)

esadomer · 2026-05-23T18:52:26Z

Thanks, good catch.

I pushed e8e15d6 to keep preserved child loggers on the public Logger.setLevel(NOTSET) path instead of assigning .level directly, and renamed the helper to _forget_existing_logger().

I also updated the PR body with fresh numbers after that change. To clarify the benchmark setup: the python_d.exe commands were validation only; the benchmark was run with a non-debug Python 3.12.8 release interpreter (Py_DEBUG: None, no sys.gettotalrefcount). Small n is still noisy/neutral, while the larger cases are now around 1.45x to 1.60x.

I tried to produce a local CPython Windows Release/PGO build too, but this machine fails before creating PCbuild\amd64\python.exe with an unrelated MSVC C1001 in zlib-ng's arch\x86\chunkset_avx2.c. So I don't want to claim these are PGO+LTO numbers from this box.

esadomer · 2026-05-23T19:20:04Z

CI note: two jobs failed, both look unrelated to this logging.config change.

Emscripten / build and test failed during checkout before tests ran: fatal: could not read Username for 'https://github.com': terminal prompts disabled.
Sanitizers / TSan (free-threading) failed in test_ssl with a ThreadSanitizer report inside OpenSSL/libcrypto (ASN1_STRING_cmp), not in logging code.

I tried to rerun the failed jobs, but I do not have Actions rerun permission on python/cpython (admin rights / Resource not accessible by integration).

picnixz · 2026-05-23T19:33:07Z

Ok, release build is still enough. Someone else can do the benchmarks on Linux PGO+LTO later

picnixz · 2026-05-23T19:33:28Z

As for GH failures, it's known (I think GH is unstable currently so don't worry)

bedevere-app Bot mentioned this pull request May 22, 2026

Improve performance in DictConfigurator by optimizing logger lookup #132372

Open

esadomer marked this pull request as ready for review May 22, 2026 12:31

esadomer requested a review from vsajip as a code owner May 22, 2026 12:31

bedevere-app Bot added the awaiting review label May 22, 2026

pythongh-132372: Speed up logging.config existing logger handling

b7a1a9b

esadomer force-pushed the codex/find-stronger-cpython-speedup branch from d599b2e to c1a3122 Compare May 23, 2026 06:21

pythongh-132372: Batch logging cache clearing

02cf98f

Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>

esadomer force-pushed the codex/find-stronger-cpython-speedup branch from c1a3122 to 02cf98f Compare May 23, 2026 06:24

picnixz reviewed May 23, 2026

View reviewed changes

pythongh-132372: Address logging config review comments

c327ca4

picnixz reviewed May 23, 2026

View reviewed changes

pythongh-132372: Use Logger.setLevel for preserved children

e8e15d6

	def _discard_existing_logger(name, existing, existing_set, child_loggers):
	def _forget_existing_logger(name, existing, existing_set, child_loggers):

Uh oh!

Conversation

esadomer commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eendebakpt commented May 22, 2026

Uh oh!

picnixz commented May 23, 2026

Uh oh!

picnixz May 23, 2026

Choose a reason for hiding this comment

Uh oh!

picnixz May 23, 2026

Choose a reason for hiding this comment

Uh oh!

picnixz May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

esadomer commented May 23, 2026

Uh oh!

picnixz commented May 23, 2026

Uh oh!

picnixz May 23, 2026

Choose a reason for hiding this comment

Uh oh!

picnixz May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

esadomer commented May 23, 2026

Uh oh!

esadomer commented May 23, 2026

Uh oh!

picnixz commented May 23, 2026

Uh oh!

picnixz commented May 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

esadomer commented May 22, 2026 •

edited

Loading

picnixz May 23, 2026 •

edited

Loading

picnixz May 23, 2026 •

edited

Loading