I'm seeing a slow down in Python 3.12.0rc1, networkx 3.1, of digraph add_edges_from and add_nodes_from methods.
I've submitted the following to the cpython project as my initial profiling would indicate this is a Python 3.12 performance regression versus Python 3.11: python/cpython#109049
The initial feedback from the cpython team is that my root cause assessment (performance of python dict methods) may be an artifact of the high number of calls together with changes to cProfile. However, all my performance tests show that for my application Python 3.12.0rc1 has a performance regression that brings its performance back to Python 3.8 (or worse). I'm reposting the profiling results here for convenience. I see that networkx uses python dict extensively, so I am wondering if you have done any performance testing with Python 3.12. I appreciate without a test case it is not straightforward to isolate any problem.
Current Behavior
Python 3.12.0rc1
Wed Sep 6 18:55:23 2023 profile.binfile
27271926 function calls (26072328 primitive calls) in 20.862 seconds
Ordered by: internal time
List reduced from 520 to 50 due to restriction <50>
ncalls tottime percall cumtime percall filename:lineno(function)
7045 4.000 0.001 5.031 0.001 digraph.py:469(add_nodes_from)
7045 3.003 0.000 5.198 0.001 digraph.py:713(add_edges_from)
7056013 1.952 0.000 2.093 0.000 {method 'update' of 'dict' objects}
1765 1.304 0.001 1.351 0.001 core.py:7090(concatenate)
1232480 0.752 0.000 1.121 0.000 reportviews.py:788(<genexpr>)
91026 0.699 0.000 1.069 0.000 resolver.py:92(Define)
203489 0.645 0.000 1.167 0.000 core.py:2952(_update_from)
41601 0.568 0.000 0.581 0.000 device.py:157(Default)
41601 0.479 0.000 0.917 0.000 mnaindexer.py:238(SetIndexing)
2 0.405 0.203 0.469 0.234 mnaloader.py:95(SetLinearTerms)
1495228 0.326 0.000 0.326 0.000 {method 'copy' of 'dict' objects}
83202 0.320 0.000 0.558 0.000 device.py:40(SetSocket)
1496872 0.311 0.000 0.311 0.000 {method 'items' of 'dict' objects}
1626268 0.308 0.000 0.308 0.000 {method 'get' of 'dict' objects}
Expected Behavior
Python 3.11.4
Wed Sep 6 18:54:04 2023 profile.binfile
27569104 function calls (26369506 primitive calls) in 16.836 seconds
Ordered by: internal time
List reduced from 541 to 50 due to restriction <50>
ncalls tottime percall cumtime percall filename:lineno(function)
7045 4.409 0.001 4.802 0.001 digraph.py:469(add_nodes_from)
7045 1.576 0.000 2.972 0.000 digraph.py:713(add_edges_from)
1765 1.337 0.001 1.380 0.001 core.py:7090(concatenate)
1232480 0.767 0.000 0.961 0.000 reportviews.py:788(<genexpr>)
7056013 0.765 0.000 0.881 0.000 {method 'update' of 'dict' objects}
91026 0.520 0.000 0.787 0.000 resolver.py:92(Define)
203489 0.486 0.000 0.721 0.000 core.py:2952(_update_from)
41601 0.464 0.000 0.464 0.000 device.py:37(<dictcomp>)
41601 0.369 0.000 0.704 0.000 mnaindexer.py:238(SetIndexing)
2 0.331 0.165 0.413 0.206 mnaloader.py:95(SetLinearTerms)
83202 0.312 0.000 0.402 0.000 device.py:40(SetSocket)
93800 0.270 0.000 1.559 0.000 core.py:3217(__getitem__)
43368 0.206 0.000 1.698 0.000 resolver.py:233(Connect)
1332528/332816 0.186 0.000 0.187 0.000 cell.py:20(GetDevices)
267312 0.182 0.000 0.182 0.000 {built-in method numpy.array}
104392 0.178 0.000 0.722 0.000 core.py:2978(__array_finalize__)
1 0.148 0.148 0.921 0.921 mnamethod.py:276(SetLinearDeviceRulesIndexing)
351240 0.129 0.000 0.839 0.000 {function MaskedArray.view at 0x7fc262658540}
1227200 0.112 0.000 0.112 0.000 reportviews.py:774(<lambda>)
1765/1 0.110 0.000 12.163 12.163 resolver.py:350(Build)
1663235 0.109 0.000 0.109 0.000 {built-in method builtins.getattr}
41601 0.105 0.000 0.162 0.000 mnaindexer.py:268(SetMatrixConstructionIndexing)
1496872 0.101 0.000 0.101 0.000 {method 'items' of 'dict' objects}
1643874 0.099 0.000 0.099 0.000 {method 'get' of 'dict' objects}
5292/12 0.098 0.000 0.743 0.062 cell.py:275(SetParameters)
1495228 0.098 0.000 0.098 0.000 {method 'copy' of 'dict' objects
Steps to Reproduce
I can try to create a test case but it seems like this is an issue that should show up in project performance tests.
Environment
Python version: 3.12.0rc1
NetworkX version: 3.1
Linux: Ubuntu 20.04 (wsl2, Windows 11)
Additional context
I'm seeing a slow down in Python 3.12.0rc1, networkx 3.1, of digraph add_edges_from and add_nodes_from methods.
I've submitted the following to the cpython project as my initial profiling would indicate this is a Python 3.12 performance regression versus Python 3.11: python/cpython#109049
The initial feedback from the cpython team is that my root cause assessment (performance of python dict methods) may be an artifact of the high number of calls together with changes to cProfile. However, all my performance tests show that for my application Python 3.12.0rc1 has a performance regression that brings its performance back to Python 3.8 (or worse). I'm reposting the profiling results here for convenience. I see that networkx uses python dict extensively, so I am wondering if you have done any performance testing with Python 3.12. I appreciate without a test case it is not straightforward to isolate any problem.
Current Behavior
Python 3.12.0rc1
Expected Behavior
Python 3.11.4
Steps to Reproduce
I can try to create a test case but it seems like this is an issue that should show up in project performance tests.
Environment
Python version: 3.12.0rc1
NetworkX version: 3.1
Linux: Ubuntu 20.04 (wsl2, Windows 11)
Additional context