Skip to content

Improve performance in DictConfigurator by optimizing logger lookup #132372

Closed
Closed
Improve performance in DictConfigurator by optimizing logger lookup#132372
Labels
performancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@allrob23
allrob23
opened on Apr 10, 2025

Feature or enhancement

Proposal:

In logging.config.DictConfigurator.configure() (in Lib/logging/config.py), the current implementation uses a list called existing to store logger names:

existing = list(root.manager.loggerDict.keys())
existing.sort()

This list is then used in both an if name in existing check and an existing.index(name) call:

if name in existing:
    i = existing.index(name)

Both operations are O(n) on a list. While this likely isn’t a major performance bottleneck for a small number of loggers, it could be avoided — and the change would be a net gain in efficiency.

One idea would be to build a set_existing = set(existing) for O(1) membership checks, and possibly a dict_existing = {name: idx} to avoid linear-time indexing, while keeping the original sorted list for the required prefix-based logic:

existing = list(root.manager.loggerDict.keys())
existing.sort()
existing_set = set(existing)
index_map = {name: idx for idx, name in enumerate(existing)}

Then the logic becomes:

if name in existing_set:
    i = index_map[name]

This avoids repeated linear scans without changing existing behavior.

Would this kind of change be reasonable here, or is there a better approach you’d recommend for balancing performance and clarity?

Happy to draft a PR if the direction makes sense.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

Activity

  1. allrob23
    added
    type-featureA feature request or enhancement
    on Apr 10, 2025
  2. eendebakpt
    added
    performancePerformance or resource usage
    on Apr 10, 2025

  3. eendebakpt commented on Apr 10, 2025

    @eendebakpt
    eendebakpt
    on Apr 10, 2025
    Contributor

    For a large number of loggers the implementation indeed looks not optimal. But how much is this a problem in applications? Unless there are benchmarks showing this part of the current implementation is a bottleneck I suspect a PR will have a low probability of being accepted.

  4. sobolevn commented on Apr 10, 2025

    @sobolevn
    sobolevn
    on Apr 10, 2025
    Member

    Please, provide a benchmark :)
    Only numbers can decide here.

  5. allrob23 commented on Apr 11, 2025

    @allrob23
    allrob23
    on Apr 11, 2025
    Author

    Thanks! I ran a benchmark to evaluate the proposed optimization. The results show a measurable improvement in configuration time as the number of loggers increases.

    Benchmark Results (500 runs each)

    Logger Count Before Optimization (avg) After Optimization (avg)
    10 3.074 ms 2.639 ms
    50 61.889 ms 52.208 ms
    100 503.351 ms 308.545 ms
    Total Time 284.38 s 181.92 s

    Tested on GitHub Codespaces with the following specs:

    • CPU: AMD EPYC 7763 (2 vCPUs)
    • RAM: 8 GB
    • OS: Fedora 40 (container)
    • Architecture: x86_64

    Please let me know what you think about this benchmark. Happy to open a PR if this looks good!

    Benchmark script

    import logging
import logging.config
import time
import random
import string

ENABLE_OPTIMIZED_CONFIG = True

# copied from logging/config.py as is
def _clearExistingHandlers():
    """Clear and close existing handlers"""
    logging._handlers.clear()
    logging.shutdown(logging._handlerList[:])
    del logging._handlerList[:]

# copied from logging/config.py as is
def _handle_existing_loggers(existing, child_loggers, disable_existing):
    """
    When (re)configuring logging, handle loggers which were in the previous
    configuration but are not in the new configuration. There's no point
    deleting them as other threads may continue to hold references to them;
    and by disabling them, you stop them doing any logging.

    However, don't disable children of named loggers, as that's probably not
    what was intended by the user. Also, allow existing loggers to NOT be
    disabled if disable_existing is false.
    """
    root = logging.root
    for log in existing:
        logger = root.manager.loggerDict[log]
        if log in child_loggers:
            if not isinstance(logger, logging.PlaceHolder):
                logger.setLevel(logging.NOTSET)
                logger.handlers = []
                logger.propagate = True
        else:
            logger.disabled = disable_existing

# copied from logging/config.py as is
def original_configure(self):
    """Do the configuration."""

    config = self.config
    if 'version' not in config:
        raise ValueError("dictionary doesn't specify a version")
    if config['version'] != 1:
        raise ValueError("Unsupported version: %s" % config['version'])
    incremental = config.pop('incremental', False)
    EMPTY_DICT = {}
    with logging._lock:
        if incremental:
            handlers = config.get('handlers', EMPTY_DICT)
            for name in handlers:
                if name not in logging._handlers:
                    raise ValueError('No handler found with '
                                        'name %r'  % name)
                else:
                    try:
                        handler = logging._handlers[name]
                        handler_config = handlers[name]
                        level = handler_config.get('level', None)
                        if level:
                            handler.setLevel(logging._checkLevel(level))
                    except Exception as e:
                        raise ValueError('Unable to configure handler '
                                            '%r' % name) from e
            loggers = config.get('loggers', EMPTY_DICT)
            for name in loggers:
                try:
                    self.configure_logger(name, loggers[name], True)
                except Exception as e:
                    raise ValueError('Unable to configure logger '
                                        '%r' % name) from e
            root = config.get('root', None)
            if root:
                try:
                    self.configure_root(root, True)
                except Exception as e:
                    raise ValueError('Unable to configure root '
                                        'logger') from e
        else:
            disable_existing = config.pop('disable_existing_loggers', True)

            _clearExistingHandlers()

            # Do formatters first - they don't refer to anything else
            formatters = config.get('formatters', EMPTY_DICT)
            for name in formatters:
                try:
                    formatters[name] = self.configure_formatter(
                                                        formatters[name])
                except Exception as e:
                    raise ValueError('Unable to configure '
                                        'formatter %r' % name) from e
            # Next, do filters - they don't refer to anything else, either
            filters = config.get('filters', EMPTY_DICT)
            for name in filters:
                try:
                    filters[name] = self.configure_filter(filters[name])
                except Exception as e:
                    raise ValueError('Unable to configure '
                                        'filter %r' % name) from e

            # Next, do handlers - they refer to formatters and filters
            # As handlers can refer to other handlers, sort the keys
            # to allow a deterministic order of configuration
            handlers = config.get('handlers', EMPTY_DICT)
            deferred = []
            for name in sorted(handlers):
                try:
                    handler = self.configure_handler(handlers[name])
                    handler.name = name
                    handlers[name] = handler
                except Exception as e:
                    if ' not configured yet' in str(e.__cause__):
                        deferred.append(name)
                    else:
                        raise ValueError('Unable to configure handler '
                                            '%r' % name) from e

            # Now do any that were deferred
            for name in deferred:
                try:
                    handler = self.configure_handler(handlers[name])
                    handler.name = name
                    handlers[name] = handler
                except Exception as e:
                    raise ValueError('Unable to configure handler '
                                        '%r' % name) from e

            # Next, do loggers - they refer to handlers and filters

            #we don't want to lose the existing loggers,
            #since other threads may have pointers to them.
            #existing is set to contain all existing loggers,
            #and as we go through the new configuration we
            #remove any which are configured. At the end,
            #what's left in existing is the set of loggers
            #which were in the previous configuration but
            #which are not in the new configuration.
            root = logging.root
            existing = list(root.manager.loggerDict.keys())
            #The list needs to be sorted so that we can
            #avoid disabling child loggers of explicitly
            #named loggers. With a sorted list it is easier
            #to find the child loggers.
            existing.sort()
            #We'll keep the list of existing loggers
            #which are children of named loggers here...
            child_loggers = []
            #now set up the new ones...
            loggers = config.get('loggers', EMPTY_DICT)
            for name in loggers:
                if name in existing:
                    i = existing.index(name) + 1 # look after name
                    prefixed = name + "."
                    pflen = len(prefixed)
                    num_existing = len(existing)
                    while i < num_existing:
                        if existing[i][:pflen] == prefixed:
                            child_loggers.append(existing[i])
                        i += 1
                    existing.remove(name)
                try:
                    self.configure_logger(name, loggers[name])
                except Exception as e:
                    raise ValueError('Unable to configure logger '
                                        '%r' % name) from e

            #Disable any old loggers. There's no point deleting
            #them as other threads may continue to hold references
            #and by disabling them, you stop them doing any logging.
            #However, don't disable children of named loggers, as that's
            #probably not what was intended by the user.
            #for log in existing:
            #    logger = root.manager.loggerDict[log]
            #    if log in child_loggers:
            #        logger.level = logging.NOTSET
            #        logger.handlers = []
            #        logger.propagate = True
            #    elif disable_existing:
            #        logger.disabled = True
            _handle_existing_loggers(existing, child_loggers,
                                        disable_existing)

            # And finally, do the root logger
            root = config.get('root', None)
            if root:
                try:
                    self.configure_root(root)
                except Exception as e:
                    raise ValueError('Unable to configure root '
                                        'logger') from e

# copied from logging/config.py with some modifications
def optimized_configure(self):
    """Do the configuration."""

    config = self.config
    if 'version' not in config:
        raise ValueError("dictionary doesn't specify a version")
    if config['version'] != 1:
        raise ValueError("Unsupported version: %s" % config['version'])
    incremental = config.pop('incremental', False)
    EMPTY_DICT = {}
    with logging._lock:
        if incremental:
            handlers = config.get('handlers', EMPTY_DICT)
            for name in handlers:
                if name not in logging._handlers:
                    raise ValueError('No handler found with '
                                        'name %r'  % name)
                else:
                    try:
                        handler = logging._handlers[name]
                        handler_config = handlers[name]
                        level = handler_config.get('level', None)
                        if level:
                            handler.setLevel(logging._checkLevel(level))
                    except Exception as e:
                        raise ValueError('Unable to configure handler '
                                            '%r' % name) from e
            loggers = config.get('loggers', EMPTY_DICT)
            for name in loggers:
                try:
                    self.configure_logger(name, loggers[name], True)
                except Exception as e:
                    raise ValueError('Unable to configure logger '
                                        '%r' % name) from e
            root = config.get('root', None)
            if root:
                try:
                    self.configure_root(root, True)
                except Exception as e:
                    raise ValueError('Unable to configure root '
                                        'logger') from e
        else:
            disable_existing = config.pop('disable_existing_loggers', True)

            _clearExistingHandlers()

            # Do formatters first - they don't refer to anything else
            formatters = config.get('formatters', EMPTY_DICT)
            for name in formatters:
                try:
                    formatters[name] = self.configure_formatter(
                                                        formatters[name])
                except Exception as e:
                    raise ValueError('Unable to configure '
                                        'formatter %r' % name) from e
            # Next, do filters - they don't refer to anything else, either
            filters = config.get('filters', EMPTY_DICT)
            for name in filters:
                try:
                    filters[name] = self.configure_filter(filters[name])
                except Exception as e:
                    raise ValueError('Unable to configure '
                                        'filter %r' % name) from e

            # Next, do handlers - they refer to formatters and filters
            # As handlers can refer to other handlers, sort the keys
            # to allow a deterministic order of configuration
            handlers = config.get('handlers', EMPTY_DICT)
            deferred = []
            for name in sorted(handlers):
                try:
                    handler = self.configure_handler(handlers[name])
                    handler.name = name
                    handlers[name] = handler
                except Exception as e:
                    if ' not configured yet' in str(e.__cause__):
                        deferred.append(name)
                    else:
                        raise ValueError('Unable to configure handler '
                                            '%r' % name) from e

            # Now do any that were deferred
            for name in deferred:
                try:
                    handler = self.configure_handler(handlers[name])
                    handler.name = name
                    handlers[name] = handler
                except Exception as e:
                    raise ValueError('Unable to configure handler '
                                        '%r' % name) from e

            # Next, do loggers - they refer to handlers and filters

            #we don't want to lose the existing loggers,
            #since other threads may have pointers to them.
            #existing is set to contain all existing loggers,
            #and as we go through the new configuration we
            #remove any which are configured. At the end,
            #what's left in existing is the set of loggers
            #which were in the previous configuration but
            #which are not in the new configuration.
            root = logging.root
            existing = list(root.manager.loggerDict.keys())
            existing_set = set(existing)
            index_map = {name: idx for idx, name in enumerate(existing)}
            #The list needs to be sorted so that we can
            #avoid disabling child loggers of explicitly
            #named loggers. With a sorted list it is easier
            #to find the child loggers.
            existing.sort()
            #We'll keep the list of existing loggers
            #which are children of named loggers here...
            child_loggers = []
            #now set up the new ones...
            loggers = config.get('loggers', EMPTY_DICT)
            for name in loggers:
                if name in existing_set:
                    i = index_map[name] + 1 # look after name
                    prefixed = name + "."
                    pflen = len(prefixed)
                    num_existing = len(existing)
                    while i < num_existing:
                        if existing[i][:pflen] == prefixed:
                            child_loggers.append(existing[i])
                        i += 1
                    existing.remove(name)
                try:
                    self.configure_logger(name, loggers[name])
                except Exception as e:
                    raise ValueError('Unable to configure logger '
                                        '%r' % name) from e

            #Disable any old loggers. There's no point deleting
            #them as other threads may continue to hold references
            #and by disabling them, you stop them doing any logging.
            #However, don't disable children of named loggers, as that's
            #probably not what was intended by the user.
            #for log in existing:
            #    logger = root.manager.loggerDict[log]
            #    if log in child_loggers:
            #        logger.level = logging.NOTSET
            #        logger.handlers = []
            #        logger.propagate = True
            #    elif disable_existing:
            #        logger.disabled = True
            _handle_existing_loggers(existing, child_loggers,
                                        disable_existing)

            # And finally, do the root logger
            root = config.get('root', None)
            if root:
                try:
                    self.configure_root(root)
                except Exception as e:
                    raise ValueError('Unable to configure root '
                                        'logger') from e

if ENABLE_OPTIMIZED_CONFIG:
    logging.config.DictConfigurator.configure = optimized_configure if ENABLE_OPTIMIZED_CONFIG else original_configure

def generate_log_config(num_loggers: int):
    """Generate a logging config dict with `num_loggers` named loggers."""
    loggers = {}
    for i in range(num_loggers):
        logger_name = f"app.module_{i}_" + ''.join(random.choices(string.ascii_lowercase, k=6))
        loggers[logger_name] = {
            "level": "DEBUG",
            "handlers": ["console"],
            "propagate": False,
        }

    config = {
        "version": 1,
        "disable_existing_loggers": False,
        "formatters": {
            "simple": {
                "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
            }
        },
        "handlers": {
            "console": {
                "class": "logging.StreamHandler",
                "formatter": "simple",
                "stream": "ext://sys.stdout"
            }
        },
        "loggers": loggers
    }

    return config


def benchmark_dict_config(num_loggers: int, runs: int = 500):
    durations = []
    for _ in range(runs):
        config = generate_log_config(num_loggers)
        start = time.perf_counter()
        logging.config.dictConfig(config)
        end = time.perf_counter()
        durations.append((end - start)*1000)

    avg_duration = sum(durations) / runs
    print(f"{num_loggers:>6} loggers → avg: {avg_duration:.6f} ms over {runs} runs")
    return avg_duration


if __name__ == "__main__":
    overall_start = time.perf_counter()
    print(f"Benchmarking logging.config.dictConfig() [{'After Optimization' if ENABLE_OPTIMIZED_CONFIG else 'Before Optimization' }]...\n")
    for count in [10, 50, 100]:
        benchmark_dict_config(count)
    overall_end = time.perf_counter()
    print(f'Total time: {overall_end - overall_start} seconds')
  6. picnixz
    added
    stdlibStandard Library Python modules in the Lib/ directory
    on Apr 11, 2025

  7. sobolevn commented on Apr 11, 2025

    @sobolevn
    sobolevn
    on Apr 11, 2025
    Member

    Please, inclide 1 and 3 as loggers counts, because I think that these are very realistic numbers for a lot of use-cases.

  8. allrob23 commented on Apr 11, 2025

    @allrob23
    allrob23
    on Apr 11, 2025
    Author

    I noticed the previous optimization didn’t perform as well for small logger counts (e.g. 1–3). To address this, I simplified the implementation by:
    • Using a set instead of a list for existing logger names.
    • Removing the sort and index logic.
    • Iterating over the set directly to find child loggers.

    This improves performance across all logger counts and simplifies the logic. Benchmark results (500 runs each):

    Logger Count Before Optimization (avg) After Optimization (avg)
    1 0.113 ms 0.097 ms
    3 0.595 ms 0.432 ms
    10 4.523 ms 3.222 ms
    50 67.603 ms 50.915 ms
    100 455.350 ms 293.010 ms
    Total Time 264.31 s 174.06 s

    Diff

    diff --git a/Lib/logging/config.py b/Lib/logging/config.py
index 6a6a7f726f7..12679e64e14 100644
--- a/Lib/logging/config.py
+++ b/Lib/logging/config.py
@@ -632,12 +632,8 @@ def configure(self):
                 #which were in the previous configuration but
                 #which are not in the new configuration.
                 root = logging.root
-                existing = list(root.manager.loggerDict.keys())
-                #The list needs to be sorted so that we can
-                #avoid disabling child loggers of explicitly
-                #named loggers. With a sorted list it is easier
-                #to find the child loggers.
-                existing.sort()
+                root = logging.root
+                existing = set(root.manager.loggerDict.keys())
                 #We'll keep the list of existing loggers
                 #which are children of named loggers here...
                 child_loggers = []
@@ -645,14 +641,11 @@ def configure(self):
                 loggers = config.get('loggers', EMPTY_DICT)
                 for name in loggers:
                     if name in existing:
-                        i = existing.index(name) + 1 # look after name
                         prefixed = name + "."
                         pflen = len(prefixed)
-                        num_existing = len(existing)
-                        while i < num_existing:
-                            if existing[i][:pflen] == prefixed:
-                                child_loggers.append(existing[i])
-                            i += 1
+                        for existing_logger in existing:
+                            if existing_logger[:pflen] == prefixed:
+                                child_loggers.append(existing_logger)
                         existing.remove(name)
                     try:
                         self.configure_logger(name, loggers[name])

    Please let me know what you think about the changes.

    here is the updated benchmark code:

    import logging
import logging.config
import time
import random
import string

ENABLE_OPTIMIZED_CONFIG = False

# copied from logging/config.py as is
def _clearExistingHandlers():
    """Clear and close existing handlers"""
    logging._handlers.clear()
    logging.shutdown(logging._handlerList[:])
    del logging._handlerList[:]

# copied from logging/config.py as is
def _handle_existing_loggers(existing, child_loggers, disable_existing):
    """
    When (re)configuring logging, handle loggers which were in the previous
    configuration but are not in the new configuration. There's no point
    deleting them as other threads may continue to hold references to them;
    and by disabling them, you stop them doing any logging.

    However, don't disable children of named loggers, as that's probably not
    what was intended by the user. Also, allow existing loggers to NOT be
    disabled if disable_existing is false.
    """
    root = logging.root
    for log in existing:
        logger = root.manager.loggerDict[log]
        if log in child_loggers:
            if not isinstance(logger, logging.PlaceHolder):
                logger.setLevel(logging.NOTSET)
                logger.handlers = []
                logger.propagate = True
        else:
            logger.disabled = disable_existing

# copied from logging/config.py as is
def original_configure(self):
    """Do the configuration."""

    config = self.config
    if 'version' not in config:
        raise ValueError("dictionary doesn't specify a version")
    if config['version'] != 1:
        raise ValueError("Unsupported version: %s" % config['version'])
    incremental = config.pop('incremental', False)
    EMPTY_DICT = {}
    with logging._lock:
        if incremental:
            handlers = config.get('handlers', EMPTY_DICT)
            for name in handlers:
                if name not in logging._handlers:
                    raise ValueError('No handler found with '
                                        'name %r'  % name)
                else:
                    try:
                        handler = logging._handlers[name]
                        handler_config = handlers[name]
                        level = handler_config.get('level', None)
                        if level:
                            handler.setLevel(logging._checkLevel(level))
                    except Exception as e:
                        raise ValueError('Unable to configure handler '
                                            '%r' % name) from e
            loggers = config.get('loggers', EMPTY_DICT)
            for name in loggers:
                try:
                    self.configure_logger(name, loggers[name], True)
                except Exception as e:
                    raise ValueError('Unable to configure logger '
                                        '%r' % name) from e
            root = config.get('root', None)
            if root:
                try:
                    self.configure_root(root, True)
                except Exception as e:
                    raise ValueError('Unable to configure root '
                                        'logger') from e
        else:
            disable_existing = config.pop('disable_existing_loggers', True)

            _clearExistingHandlers()

            # Do formatters first - they don't refer to anything else
            formatters = config.get('formatters', EMPTY_DICT)
            for name in formatters:
                try:
                    formatters[name] = self.configure_formatter(
                                                        formatters[name])
                except Exception as e:
                    raise ValueError('Unable to configure '
                                        'formatter %r' % name) from e
            # Next, do filters - they don't refer to anything else, either
            filters = config.get('filters', EMPTY_DICT)
            for name in filters:
                try:
                    filters[name] = self.configure_filter(filters[name])
                except Exception as e:
                    raise ValueError('Unable to configure '
                                        'filter %r' % name) from e

            # Next, do handlers - they refer to formatters and filters
            # As handlers can refer to other handlers, sort the keys
            # to allow a deterministic order of configuration
            handlers = config.get('handlers', EMPTY_DICT)
            deferred = []
            for name in sorted(handlers):
                try:
                    handler = self.configure_handler(handlers[name])
                    handler.name = name
                    handlers[name] = handler
                except Exception as e:
                    if ' not configured yet' in str(e.__cause__):
                        deferred.append(name)
                    else:
                        raise ValueError('Unable to configure handler '
                                            '%r' % name) from e

            # Now do any that were deferred
            for name in deferred:
                try:
                    handler = self.configure_handler(handlers[name])
                    handler.name = name
                    handlers[name] = handler
                except Exception as e:
                    raise ValueError('Unable to configure handler '
                                        '%r' % name) from e

            # Next, do loggers - they refer to handlers and filters

            #we don't want to lose the existing loggers,
            #since other threads may have pointers to them.
            #existing is set to contain all existing loggers,
            #and as we go through the new configuration we
            #remove any which are configured. At the end,
            #what's left in existing is the set of loggers
            #which were in the previous configuration but
            #which are not in the new configuration.
            root = logging.root
            existing = list(root.manager.loggerDict.keys())
            #The list needs to be sorted so that we can
            #avoid disabling child loggers of explicitly
            #named loggers. With a sorted list it is easier
            #to find the child loggers.
            existing.sort()
            #We'll keep the list of existing loggers
            #which are children of named loggers here...
            child_loggers = []
            #now set up the new ones...
            loggers = config.get('loggers', EMPTY_DICT)
            for name in loggers:
                if name in existing:
                    i = existing.index(name) + 1 # look after name
                    prefixed = name + "."
                    pflen = len(prefixed)
                    num_existing = len(existing)
                    while i < num_existing:
                        if existing[i][:pflen] == prefixed:
                            child_loggers.append(existing[i])
                        i += 1
                    existing.remove(name)
                try:
                    self.configure_logger(name, loggers[name])
                except Exception as e:
                    raise ValueError('Unable to configure logger '
                                        '%r' % name) from e

            #Disable any old loggers. There's no point deleting
            #them as other threads may continue to hold references
            #and by disabling them, you stop them doing any logging.
            #However, don't disable children of named loggers, as that's
            #probably not what was intended by the user.
            #for log in existing:
            #    logger = root.manager.loggerDict[log]
            #    if log in child_loggers:
            #        logger.level = logging.NOTSET
            #        logger.handlers = []
            #        logger.propagate = True
            #    elif disable_existing:
            #        logger.disabled = True
            _handle_existing_loggers(existing, child_loggers,
                                        disable_existing)

            # And finally, do the root logger
            root = config.get('root', None)
            if root:
                try:
                    self.configure_root(root)
                except Exception as e:
                    raise ValueError('Unable to configure root '
                                        'logger') from e

# copied from logging/config.py with some modifications
def optimized_configure(self):
    """Do the configuration."""

    config = self.config
    if 'version' not in config:
        raise ValueError("dictionary doesn't specify a version")
    if config['version'] != 1:
        raise ValueError("Unsupported version: %s" % config['version'])
    incremental = config.pop('incremental', False)
    EMPTY_DICT = {}
    with logging._lock:
        if incremental:
            handlers = config.get('handlers', EMPTY_DICT)
            for name in handlers:
                if name not in logging._handlers:
                    raise ValueError('No handler found with '
                                        'name %r'  % name)
                else:
                    try:
                        handler = logging._handlers[name]
                        handler_config = handlers[name]
                        level = handler_config.get('level', None)
                        if level:
                            handler.setLevel(logging._checkLevel(level))
                    except Exception as e:
                        raise ValueError('Unable to configure handler '
                                            '%r' % name) from e
            loggers = config.get('loggers', EMPTY_DICT)
            for name in loggers:
                try:
                    self.configure_logger(name, loggers[name], True)
                except Exception as e:
                    raise ValueError('Unable to configure logger '
                                        '%r' % name) from e
            root = config.get('root', None)
            if root:
                try:
                    self.configure_root(root, True)
                except Exception as e:
                    raise ValueError('Unable to configure root '
                                        'logger') from e
        else:
            disable_existing = config.pop('disable_existing_loggers', True)

            _clearExistingHandlers()

            # Do formatters first - they don't refer to anything else
            formatters = config.get('formatters', EMPTY_DICT)
            for name in formatters:
                try:
                    formatters[name] = self.configure_formatter(
                                                        formatters[name])
                except Exception as e:
                    raise ValueError('Unable to configure '
                                        'formatter %r' % name) from e
            # Next, do filters - they don't refer to anything else, either
            filters = config.get('filters', EMPTY_DICT)
            for name in filters:
                try:
                    filters[name] = self.configure_filter(filters[name])
                except Exception as e:
                    raise ValueError('Unable to configure '
                                        'filter %r' % name) from e

            # Next, do handlers - they refer to formatters and filters
            # As handlers can refer to other handlers, sort the keys
            # to allow a deterministic order of configuration
            handlers = config.get('handlers', EMPTY_DICT)
            deferred = []
            for name in sorted(handlers):
                try:
                    handler = self.configure_handler(handlers[name])
                    handler.name = name
                    handlers[name] = handler
                except Exception as e:
                    if ' not configured yet' in str(e.__cause__):
                        deferred.append(name)
                    else:
                        raise ValueError('Unable to configure handler '
                                            '%r' % name) from e

            # Now do any that were deferred
            for name in deferred:
                try:
                    handler = self.configure_handler(handlers[name])
                    handler.name = name
                    handlers[name] = handler
                except Exception as e:
                    raise ValueError('Unable to configure handler '
                                        '%r' % name) from e

            # Next, do loggers - they refer to handlers and filters

            #we don't want to lose the existing loggers,
            #since other threads may have pointers to them.
            #existing is set to contain all existing loggers,
            #and as we go through the new configuration we
            #remove any which are configured. At the end,
            #what's left in existing is the set of loggers
            #which were in the previous configuration but
            #which are not in the new configuration.
            root = logging.root
            existing = set(root.manager.loggerDict.keys())

            #We'll keep the list of existing loggers
            #which are children of named loggers here...
            child_loggers = []
            #now set up the new ones...
            loggers = config.get('loggers', EMPTY_DICT)
            for name in loggers:
                if name in existing:
                    prefixed = name + "."
                    pflen = len(prefixed)
                    for existing_logger in existing:
                        if existing_logger[:pflen] == prefixed:
                            child_loggers.append(existing_logger)
                    existing.remove(name)
                try:
                    self.configure_logger(name, loggers[name])
                except Exception as e:
                    raise ValueError('Unable to configure logger '
                                        '%r' % name) from e

            #Disable any old loggers. There's no point deleting
            #them as other threads may continue to hold references
            #and by disabling them, you stop them doing any logging.
            #However, don't disable children of named loggers, as that's
            #probably not what was intended by the user.
            #for log in existing:
            #    logger = root.manager.loggerDict[log]
            #    if log in child_loggers:
            #        logger.level = logging.NOTSET
            #        logger.handlers = []
            #        logger.propagate = True
            #    elif disable_existing:
            #        logger.disabled = True
            _handle_existing_loggers(existing, child_loggers,
                                        disable_existing)

            # And finally, do the root logger
            root = config.get('root', None)
            if root:
                try:
                    self.configure_root(root)
                except Exception as e:
                    raise ValueError('Unable to configure root '
                                        'logger') from e

if ENABLE_OPTIMIZED_CONFIG:
    logging.config.DictConfigurator.configure = optimized_configure if ENABLE_OPTIMIZED_CONFIG else original_configure

def generate_log_config(num_loggers: int):
    """Generate a logging config dict with `num_loggers` named loggers."""
    loggers = {}
    for i in range(num_loggers):
        logger_name = f"app.module_{i}_" + ''.join(random.choices(string.ascii_lowercase, k=6))
        loggers[logger_name] = {
            "level": "DEBUG",
            "handlers": ["console"],
            "propagate": False,
        }

    config = {
        "version": 1,
        "disable_existing_loggers": False,
        "formatters": {
            "simple": {
                "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
            }
        },
        "handlers": {
            "console": {
                "class": "logging.StreamHandler",
                "formatter": "simple",
                "stream": "ext://sys.stdout"
            }
        },
        "loggers": loggers
    }

    return config


def benchmark_dict_config(num_loggers: int, runs: int = 500):
    durations = []
    for _ in range(runs):
        config = generate_log_config(num_loggers)
        start = time.perf_counter()
        logging.config.dictConfig(config)
        end = time.perf_counter()
        durations.append((end - start)*1000)

    avg_duration = sum(durations) / runs
    print(f"{num_loggers:>6} loggers → avg: {avg_duration:.6f} ms over {runs} runs")
    return avg_duration


if __name__ == "__main__":
    overall_start = time.perf_counter()
    print(f"Benchmarking logging.config.dictConfig() [{'After Optimization' if ENABLE_OPTIMIZED_CONFIG else 'Before Optimization' }]...\n")
    for count in [1, 3, 10, 50, 100]:
        benchmark_dict_config(count)
    overall_end = time.perf_counter()
    print(f'Total time: {overall_end - overall_start} seconds')
  9. esadomer
    added 9 commits that reference this issue on May 23, 2026

    pythongh-132372: Speed up logging.config existing logger handling

    b7a1a9b

    pythongh-132372: Batch logging cache clearing

    c1a3122

    pythongh-132372: Batch logging cache clearing

    02cf98f

    pythongh-132372: Address logging config review comments

    c327ca4

    pythongh-132372: Use Logger.setLevel for preserved children

    e8e15d6

    pythongh-132372: Speed up logging.config existing logger handling

    55a64e2

    pythongh-132372: Batch logging cache clearing

    4040d60

    pythongh-132372: Address logging config review comments

    d076b07

    pythongh-132372: Use Logger.setLevel for preserved children

    843bd2e
  10. vsajip
    added a commit that references this issue on May 29, 2026

    gh-132372: Speed up logging.config existing logger handling (GH-150242)

    26696a6
  11. vsajip
    closed this as completedon May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions