gh-92810: Reduce memory usage by ABCMeta.__subclasscheck__ (alternative) by dolfinus · Pull Request #141171 · python/cpython

dolfinus · 2025-11-06T21:57:24Z

Issue: _abc._abc_subclasscheck has very poor performance and (I think) a memory leak #92810

Alternative implementation of #131914 which doesn't modify class attributes on every __subclasscheck__ call. Instead it checks in __new__ does current class have def __subclasses__(cls) overriden, and if so, enables case for scls in cls.__subclasses__(), which is disabled by default.
To handle cases like:

class Number(ABC): ...

class Real(Number): ...

class Integral(Real): ...

Integral.register(int)

assert issubclass(int, Number) is True

which previously were implemented via __subclasses__, method cls.register(subclass) is calling super(cls).register(subclass) recursively ("bubble-up" registration to all the parents).

For benchmark from #131914:

sudo ./python -m pyperf system tune
taskset -c 0 ./python benchmark.py --metaclass abc.ABCMeta --rounds 3 --classes 5000
taskset -c 0 ./python benchmark.py --metaclass _py_abc.ABCMeta --rounds 3 --classes 5000

Impl	Max memory before, MB	Max memory after, MB
_abc	6331	49
_py_abc	4422	59

Impl	Total time before	Total time after
_abc	6m 16s	1m 28s
_py_abc	8m 42s	3m 50s

Check	Impl	before	after	Impl	before	after
`isinstance(child, Parent)`	_abc	0.115us 4MiB...15MiB	0.120us 4MiB...15MiB	_py_abc	0.212us 11MiB...24MiB	0.221us 11MiB...24MiB
`issubclass(Child, Parent)`	_abc	0.105us 0MiB...1MiB	0.112us 0MiB...1MiB	_py_abc	0.203us 6MiB...8MiB	0.212us 6MiB...8MiB
`isinstance(child, Grandparent)`	_abc	0.112us 0MiB...2MiB	0.117us 0MiB...2MiB	_py_abc	0.210us 4MiB...7MiB	0.218us 4MiB...7MiB
`issubclass(Child, Grandparent)`	_abc	0.103us 0MiB	0.110us 0MiB	_py_abc	0.201us 0MiB...1MiB	0.210us 0MiB...1MiB
`not isinstance(child, Sibling)`	_abc	0.113us 4MiB...14MiB	0.115us 4MiB...15MiB	_py_abc	0.348us 13MiB...23MiB	0.362us 13MiB...23MiB
`not issubclass(Child, Sibling)`	_abc	0.105us 1MiB	0.110us 1MiB	_py_abc	0.328us 8MiB...10MiB	0.337us 9MiB...11MiB
`not isinstance(child, Cousin)`	_abc	0.115us 1MiB...2MiB	0.116us 1MiB...2MiB	_py_abc	0.350us 7MiB...9MiB	0.362us 7MiB...9MiB
`not issubclass(Child, Cousin)`	_abc	0.104us 0MiB	0.109us 0MiB	_py_abc	0.329us 4MiB	0.337us 4MiB...5MiB
`not isinstance(child, Uncle)`	_abc	7.268us 6174MiB...6333MiB	0.121us 0MiB...1MiB	_py_abc	9.957us 4382MiB...4422MiB	0.365us 6MiB
`not issubclass(Child, Uncle)`	_abc	7.099us 6171MiB	0.117us 0MiB	_py_abc	9.936us 4380MiB	0.337us 4MiB

Memory increment is measured during isinstance() / issubclass() calls, not during preparation, like class creation or registration where actual registry allocation is performed. So memory usage in tables below is almost always 0.

Timing drop in _py_abc implementation for 2 first rows is due to if subclass in cls._abc_registry: check added to match _abc.c implementation.

Check	Impl	before	after	Impl	before	after
`isinstance(child, Parent.register)`	_abc	0.273us 0MiB	0.273us 0MiB	_py_abc	0.440us 0MiB	0.270us 0MiB
`issubclass(Child, Parent.register)`	_abc	0.154us 0MiB	0.159us 0MiB	_py_abc	0.427us 0MiB	0.258us 0MiB
`isinstance(child, Grandparent.register)`	_abc	0.114us 0MiB	0.117us 0MiB	_py_abc	0.253us 0MiB	0.256us 0MiB
`issubclass(Child, Grandparent.register)`	_abc	0.103us 0MiB	0.111us 0MiB	_py_abc	0.240us 0MiB	0.245us 0MiB
`not isinstance(child, Sibling.register)`	_abc	0.027us 0MiB	0.028us 1MiB	_py_abc	0.028us 0MiB	0.028us 2MiB
`not issubclass(Child, Sibling.register)`	_abc	0.018us 0MiB	0.018us 1MiB	_py_abc	0.018us 0MiB	0.018us 2MiB
`not isinstance(child, Cousin.register)`	_abc	0.028us 0MiB	0.028us 2MiB	_py_abc	0.028us 0MiB	0.029us 3MiB
`not issubclass(Child, Cousin.register)`	_abc	0.018us 0MiB	0.018us 2MiB	_py_abc	0.019us 0MiB	0.019us 3MiB
`not isinstance(child, Uncle.register)`	_abc	0.249us 0MiB	0.234us 2MiB	_py_abc	0.843us 0MiB	0.907us 4MiB
`not issubclass(Child, Uncle.register)`	_abc	0.238us 0MiB	0.230us 2MiB	_py_abc	0.815us 0MiB	0.854us 4MiB

Check	Impl	before	after	Impl	before	after
`isinstance(child, Parent.__subclasses__)`	_abc	0.128us 0MiB	0.132us 0MiB	_py_abc	0.305us 0MiB	0.308us 0MiB
`issubclass(Child, Parent.__subclasses__)`	_abc	0.118us 0MiB	0.125us 0MiB	_py_abc	0.293us 0MiB	0.292us 0MiB
`isinstance(child, Grandparent.__subclasses__)`	_abc	0.126us 0MiB	0.130us 0MiB	_py_abc	0.304us 0MiB	0.306us 0MiB
`issubclass(Child, Grandparent.__subclasses__)`	_abc	0.116us 0MiB	0.122us 0MiB	_py_abc	0.293us 0MiB	0.292us 0MiB
`not isinstance(child, Sibling.__subclasses__)`	_abc	0.028us 0MiB	0.028us 0MiB	_py_abc	0.028us 0MiB	0.028us 1MiB
`not issubclass(Child, Sibling.__subclasses__)`	_abc	0.018us 0MiB	0.018us 0MiB	_py_abc	0.018us 0MiB	0.018us 1MiB
`not isinstance(child, Cousin.__subclasses__)`	_abc	0.028us 0MiB	0.028us 0MiB	_py_abc	0.028us 0MiB	0.028us 1MiB
`not issubclass(Child, Cousin.__subclasses__)`	_abc	0.018us 0MiB	0.018us 0MiB	_py_abc	0.018us 0MiB	0.020us 1MiB
`not isinstance(child, Uncle.__subclasses__)`	_abc	0.145us 0MiB	0.150us 1MiB	_py_abc	0.553us 0MiB	0.574us 2MiB
`not issubclass(Child, Uncle.__subclasses__)`	_abc	0.134us 0MiB	0.144us 1MiB	_py_abc	0.526us 0MiB	0.542us 2MiB

Just to check that nothing is broken:

Check	Impl	before	after	Impl	before	after
`not isinstance(child, Unrelated)`	_abc	0.028us 0MiB	0.028us 0MiB	_py_abc	0.028us 0MiB	0.028us 0MiB
`not issubclass(Child, Unrelated)`	_abc	0.018us 0MiB	0.018us 0MiB	_py_abc	0.018us 0MiB	0.018us 0MiB
`not isinstance(child, UnrelatedABC)`	_abc	0.118us 0MiB	0.118us 0MiB	_py_abc	0.477us 0MiB	0.479us 0MiB
`not issubclass(Child, UnrelatedABC)`	_abc	0.110us 0MiB	0.112us 0MiB	_py_abc	0.450us 0MiB	0.451us 0MiB

Flamegraphs for _py_abc impl and test issubclass_uncle (the most time and memory consuming case on main):
main_vs_pr141171.tar.gz

Alternative: #150540 drops recursive subclass check completely, and does not mess around with class attributes.

bedevere-app · 2025-11-06T21:57:28Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

bedevere-app · 2025-11-06T22:02:11Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

bedevere-app · 2025-11-06T22:04:09Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

bedevere-app · 2025-11-06T22:17:15Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

dolfinus · 2025-12-06T11:07:03Z

pyperformance results doesn't show a much difference:

sudo ./venv/bin/python -m pyperf system tune
./venv/bin/python -m pyperformance run --rigorous --affinity 0,1,2,3,4,5

main_cd4d0ae75c_rigorous_affinity.json
pr141171_rigorous_affinity.json

Details

Benchmarks with tag 'apps':

Benchmark	main_cd4d0ae75c_rigorous_affinity	pr141171_rigorous_affinity
chameleon	18.4 ms	18.6 ms: 1.01x slower
sphinx	1.13 sec	1.14 sec: 1.00x slower
Geometric mean	(ref)	1.00x slower

Benchmark hidden because not significant (4): 2to3, docutils, html5lib, tornado_http

Benchmarks with tag 'asyncio':

Benchmark	main_cd4d0ae75c_rigorous_affinity	pr141171_rigorous_affinity
async_tree_eager_memoization	225 ms	216 ms: 1.04x faster
async_tree_eager_io_tg	627 ms	617 ms: 1.02x faster
async_tree_eager_tg	263 ms	259 ms: 1.02x faster
asyncio_tcp	376 ms	378 ms: 1.01x slower
async_tree_cpu_io_mixed	667 ms	673 ms: 1.01x slower
async_tree_eager_cpu_io_mixed	536 ms	540 ms: 1.01x slower
async_tree_eager_cpu_io_mixed_tg	615 ms	626 ms: 1.02x slower
async_generators	501 ms	512 ms: 1.02x slower
coroutines	26.7 ms	27.9 ms: 1.05x slower
async_tree_eager_io	629 ms	659 ms: 1.05x slower
Geometric mean	(ref)	1.01x slower

Benchmark hidden because not significant (11): async_tree_none_tg, async_tree_eager, async_tree_memoization, async_tree_eager_memoization_tg, asyncio_websockets, async_tree_io_tg, asyncio_tcp_ssl, async_tree_memoization_tg, async_tree_io, async_tree_none, async_tree_cpu_io_mixed_tg

Benchmarks with tag 'math':

Benchmark	main_cd4d0ae75c_rigorous_affinity	pr141171_rigorous_affinity
nbody	109 ms	106 ms: 1.03x faster
pidigits	287 ms	288 ms: 1.00x slower
Geometric mean	(ref)	1.01x faster

Benchmark hidden because not significant (1): float

Benchmarks with tag 'regex':

Benchmark	main_cd4d0ae75c_rigorous_affinity	pr141171_rigorous_affinity
regex_v8	23.9 ms	22.4 ms: 1.07x faster
regex_effbot	2.88 ms	2.81 ms: 1.02x faster
regex_dna	199 ms	194 ms: 1.02x faster
Geometric mean	(ref)	1.03x faster

Benchmark hidden because not significant (1): regex_compile

Benchmarks with tag 'serialize':

Benchmark	main_cd4d0ae75c_rigorous_affinity	pr141171_rigorous_affinity
unpickle_list	5.37 us	5.20 us: 1.03x faster
pickle_list	5.04 us	4.95 us: 1.02x faster
xml_etree_process	77.6 ms	76.4 ms: 1.02x faster
xml_etree_iterparse	106 ms	105 ms: 1.01x faster
unpickle_pure_python	254 us	252 us: 1.01x faster
xml_etree_generate	111 ms	110 ms: 1.01x faster
xml_etree_parse	166 ms	167 ms: 1.00x slower
json_dumps	10.9 ms	11.0 ms: 1.01x slower
tomli_loads	2.25 sec	2.28 sec: 1.01x slower
json_loads	27.4 us	28.1 us: 1.02x slower
Geometric mean	(ref)	1.00x faster

Benchmark hidden because not significant (4): pickle_pure_python, pickle_dict, pickle, unpickle

Benchmarks with tag 'startup':

Benchmark	main_cd4d0ae75c_rigorous_affinity	pr141171_rigorous_affinity
python_startup_no_site	7.99 ms	8.04 ms: 1.01x slower
python_startup	13.8 ms	14.0 ms: 1.01x slower
Geometric mean	(ref)	1.01x slower

Benchmarks with tag 'template':

Benchmark	main_cd4d0ae75c_rigorous_affinity	pr141171_rigorous_affinity
genshi_xml	59.0 ms	58.4 ms: 1.01x faster
mako	12.6 ms	12.7 ms: 1.01x slower
genshi_text	25.6 ms	25.9 ms: 1.01x slower
Geometric mean	(ref)	1.00x slower

All benchmarks:

Benchmark	main_cd4d0ae75c_rigorous_affinity	pr141171_rigorous_affinity
regex_v8	23.9 ms	22.4 ms: 1.07x faster
async_tree_eager_memoization	225 ms	216 ms: 1.04x faster
unpickle_list	5.37 us	5.20 us: 1.03x faster
unpack_sequence	47.9 ns	46.5 ns: 1.03x faster
fannkuch	460 ms	447 ms: 1.03x faster
nbody	109 ms	106 ms: 1.03x faster
nqueens	103 ms	100 ms: 1.02x faster
regex_effbot	2.88 ms	2.81 ms: 1.02x faster
regex_dna	199 ms	194 ms: 1.02x faster
hexiom	6.63 ms	6.49 ms: 1.02x faster
pickle_list	5.04 us	4.95 us: 1.02x faster
deepcopy_memo	29.3 us	28.8 us: 1.02x faster
async_tree_eager_io_tg	627 ms	617 ms: 1.02x faster
xml_etree_process	77.6 ms	76.4 ms: 1.02x faster
async_tree_eager_tg	263 ms	259 ms: 1.02x faster
deepcopy	296 us	292 us: 1.02x faster
typing_runtime_protocols	198 us	196 us: 1.01x faster
logging_format	7.41 us	7.34 us: 1.01x faster
genshi_xml	59.0 ms	58.4 ms: 1.01x faster
scimark_lu	129 ms	128 ms: 1.01x faster
xml_etree_iterparse	106 ms	105 ms: 1.01x faster
unpickle_pure_python	254 us	252 us: 1.01x faster
dulwich_log	72.3 ms	71.7 ms: 1.01x faster
xml_etree_generate	111 ms	110 ms: 1.01x faster
xdsl_constant_fold	52.4 ms	52.0 ms: 1.01x faster
deepcopy_reduce	3.36 us	3.34 us: 1.01x faster
bpe_tokeniser	5.31 sec	5.29 sec: 1.00x faster
mdp	1.48 sec	1.48 sec: 1.00x slower
pidigits	287 ms	288 ms: 1.00x slower
sphinx	1.13 sec	1.14 sec: 1.00x slower
sqlalchemy_declarative	143 ms	144 ms: 1.00x slower
xml_etree_parse	166 ms	167 ms: 1.00x slower
gc_traversal	4.99 ms	5.02 ms: 1.01x slower
python_startup_no_site	7.99 ms	8.04 ms: 1.01x slower
create_gc_cycles	2.40 ms	2.42 ms: 1.01x slower
asyncio_tcp	376 ms	378 ms: 1.01x slower
bench_thread_pool	1.06 ms	1.07 ms: 1.01x slower
sqlglot_v2_parse	1.41 ms	1.42 ms: 1.01x slower
mako	12.6 ms	12.7 ms: 1.01x slower
async_tree_cpu_io_mixed	667 ms	673 ms: 1.01x slower
async_tree_eager_cpu_io_mixed	536 ms	540 ms: 1.01x slower
subparsers	46.5 ms	46.9 ms: 1.01x slower
coverage	125 ms	127 ms: 1.01x slower
chameleon	18.4 ms	18.6 ms: 1.01x slower
comprehensions	18.6 us	18.8 us: 1.01x slower
python_startup	13.8 ms	14.0 ms: 1.01x slower
json_dumps	10.9 ms	11.0 ms: 1.01x slower
genshi_text	25.6 ms	25.9 ms: 1.01x slower
tomli_loads	2.25 sec	2.28 sec: 1.01x slower
logging_simple	6.51 us	6.58 us: 1.01x slower
deltablue	3.54 ms	3.59 ms: 1.01x slower
chaos	66.1 ms	67.0 ms: 1.01x slower
raytrace	325 ms	330 ms: 1.01x slower
scimark_fft	309 ms	313 ms: 1.02x slower
richards_super	57.5 ms	58.5 ms: 1.02x slower
telco	8.59 ms	8.74 ms: 1.02x slower
scimark_monte_carlo	73.9 ms	75.2 ms: 1.02x slower
generators	34.3 ms	34.9 ms: 1.02x slower
async_tree_eager_cpu_io_mixed_tg	615 ms	626 ms: 1.02x slower
crypto_pyaes	80.9 ms	82.4 ms: 1.02x slower
async_generators	501 ms	512 ms: 1.02x slower
json_loads	27.4 us	28.1 us: 1.02x slower
pathlib	15.2 ms	15.6 ms: 1.02x slower
richards	49.4 ms	51.1 ms: 1.03x slower
logging_silent	114 ns	118 ns: 1.04x slower
spectral_norm	106 ms	110 ms: 1.04x slower
coroutines	26.7 ms	27.9 ms: 1.05x slower
async_tree_eager_io	629 ms	659 ms: 1.05x slower
Geometric mean	(ref)	1.00x slower

Benchmark hidden because not significant (36): bench_mp_pool, html5lib, meteor_contest, scimark_sor, docutils, pprint_pformat, regex_compile, pickle_pure_python, async_tree_none_tg, pickle_dict, async_tree_eager, tornado_http, dask, pyflate, go, scimark_sparse_mat_mult, async_tree_memoization, pickle, async_tree_eager_memoization_tg, asyncio_websockets, many_optionals, sqlglot_v2_normalize, pprint_safe_repr, 2to3, float, unpickle, async_tree_io_tg, asyncio_tcp_ssl, async_tree_memoization_tg, async_tree_io, sqlite_synth, sqlalchemy_imperative, sqlglot_v2_optimize, sqlglot_v2_transpile, async_tree_none, async_tree_cpu_io_mixed_tg

dolfinus · 2025-12-14T10:03:40Z

If for scls in cls.__subclasses__() was added only to handle nested ABC classes registration, like this:

class Number(ABC): ...

class Real(Number): ...

Real.register(int)
Real.register(float)

assert issubclass(int, Number)
assert issubclass(float, Number)

then it is not required anymore. In this case _abc_should_check_subclasses check and new tests with __subclasses__ overrides can be dropped.

I haven't found any uses of the such overrides with ABC/ABCMeta, other than implementing custom non-ABC metaclasses or using this as typeshed annotations in .pyi files:
https://github.com/search?q=%22def+__subclasses__%22+language%3APython&type=code&ref=advsearch&p=1
https://github.com/search?q=%22__subclasses__+%3D%22+language%3APython&type=code&ref=advsearch&p=1

UPD: implementation from #150540 drops recursive subclass check completely, and does not mess around with class attributes.

github-actions · 2026-05-01T18:22:31Z

This PR is stale because it has been open for 30 days with no activity.

read-the-docs-community · 2026-05-27T21:13:22Z

Documentation build overview

📚 cpython-previews | 🛠️ Build #32876410 | 📁 Comparing 71c4864 against main (24c6bbc)

🔍 Preview build

2 files changed

± whatsnew/3.15.html
± whatsnew/changelog.html

dolfinus · 2026-05-28T06:26:48Z

Rebased to main branch and rerun benchmarks. Added test case with unrelated class tree.

dolfinus changed the title ~~Improvement/abc meta subclasscheck v2~~ gh-92810: Reduce memory usage by ABCMeta.__subclasscheck__ (v2) Nov 6, 2025

bedevere-app Bot mentioned this pull request Nov 6, 2025

_abc._abc_subclasscheck has very poor performance and (I think) a memory leak #92810

Open

dolfinus changed the title ~~gh-92810: Reduce memory usage by ABCMeta.__subclasscheck__ (v2)~~ gh-92810: Reduce memory usage by ABCMeta.__subclasscheck__ Nov 6, 2025

dolfinus added 2 commits November 30, 2025 19:57

pythongh-92810: Add atomic FT wrappers for uint64_t

0cf31cb

pythongh-92810: Add more tests for ABC inheritance & registration

b1a574f

dolfinus force-pushed the improvement/ABCMeta_subclasscheck_v2 branch 10 times, most recently from b3bbd15 to a7ef5b0 Compare December 1, 2025 06:20

dolfinus added 2 commits December 4, 2025 09:13

pythongh-92810: Reduce memory usage by ABCMeta.__subclasscheck__

295f64d

pythongh-92810: Add What's New entry

6869658

dolfinus force-pushed the improvement/ABCMeta_subclasscheck_v2 branch from a7ef5b0 to 6869658 Compare December 4, 2025 06:14

dolfinus marked this pull request as ready for review December 4, 2025 20:38

dolfinus requested a review from AA-Turner as a code owner December 4, 2025 20:38

bedevere-app Bot added the awaiting review label Dec 4, 2025

dolfinus mentioned this pull request Dec 4, 2025

gh-92810: Reduce memory usage by ABCMeta.__subclasscheck__ #131914

Open

Merge branch 'main' into improvement/ABCMeta_subclasscheck_v2

602f031

picnixz changed the title ~~gh-92810: Reduce memory usage by ABCMeta.__subclasscheck__~~ gh-92810: Reduce memory usage by ABCMeta.__subclasscheck__ (alternative) Dec 14, 2025

pythongh-92810: Address review fixes

a2fc562

dolfinus mentioned this pull request Feb 18, 2026

gh-92810: Cover ABCMeta instancecheck/subclasscheck with more tests #144941

Open

github-actions Bot added the stale Stale PR or inactive for long period of time. label May 1, 2026

dolfinus added 2 commits May 28, 2026 00:07

Merge branch 'main' into improvement/ABCMeta_subclasscheck_v2

5579209

pythongh-92810: Fix merge conflicts

71c4864

Uh oh!

Conversation

dolfinus commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app Bot commented Nov 6, 2025

Uh oh!

bedevere-app Bot commented Nov 6, 2025

Uh oh!

bedevere-app Bot commented Nov 6, 2025

Uh oh!

bedevere-app Bot commented Nov 6, 2025

Uh oh!

dolfinus commented Dec 6, 2025

Benchmarks with tag 'apps':

Benchmarks with tag 'asyncio':

Benchmarks with tag 'math':

Benchmarks with tag 'regex':

Benchmarks with tag 'serialize':

Benchmarks with tag 'startup':

Benchmarks with tag 'template':

All benchmarks:

Uh oh!

dolfinus commented Dec 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 1, 2026

Uh oh!

read-the-docs-community Bot commented May 27, 2026

Documentation build overview

Uh oh!

dolfinus commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dolfinus commented Nov 6, 2025 •

edited

Loading

dolfinus commented Dec 14, 2025 •

edited

Loading

dolfinus commented May 28, 2026 •

edited

Loading