Repository navigation
Instance method performance issue with free threading (3.13rc1) #123619
Description
Bug report
Bug description:
from threading import Thread
import time
class Cache:
def get(self, i):
return i
def get(i):
return i
client = Cache()
def bench_run(runner):
s = time.monotonic_ns()
for i in range(500000):
runner(i)
d = time.monotonic_ns() - s
print(f"{d:,}")
def bench_run_parallel(count, runner):
tl = []
for i in range(count):
t = Thread(target=bench_run, args=[runner])
tl.append(t)
t.start()
for t in tl:
t.join()
if __name__ == '__main__':
print("no threading class")
bench_run(client.get)
print("\nthreading class")
bench_run_parallel(6, client.get)
print("\nno threading function")
bench_run(get)
print("\nthreading function")
bench_run_parallel(6, get)Processor: 2.6 GHz 6-Core Intel Core i7
PYTHON_CONFIGURE_OPTS='--disable-gil' pyenv install 3.13.0rc1
PYTHON_GIL=0 python benchmarks/run.py
Python 3.13.0rc1 experimental free-threading build (main, Sep 3 2024, 09:59:18) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
no threading class
34,626,126
threading class
313,298,115
318,659,929
319,484,384
321,183,006
320,650,898
321,589,085
no threading function
29,890,615
threading function
31,336,844
31,927,592
32,087,732
33,612,093
32,611,520
33,897,321
When using 6 threads, the instance method becomes approximately 10 times slower, while the function method shows comparable performance.
Python 3.12.5 (main, Sep 2 2024, 15:11:13) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
no threading class
27,588,484
threading class
79,383,431
113,182,075
111,067,075
107,679,440
63,825,808
92,679,231
no threading function
26,200,075
threading function
76,489,523
105,636,550
121,436,736
106,408,462
104,572,564
70,686,101
CPython versions tested on:
3.13
Operating systems tested on:
macOS
Linked PRs
- gh-123619: Add an unstable C API function for enabling deferred reference counting #123635
- gh-123619: Fix PyUnstable_Object_EnableDeferredRefcount() #151260
- [3.15] gh-123619: Fix PyUnstable_Object_EnableDeferredRefcount() (GH-151260) #151326
- [3.14] gh-123619: Fix PyUnstable_Object_EnableDeferredRefcount() (GH-151260) #151327
Reacted by JustinLi
Activity
ZeroIntensity commentedon Sep 3, 2024
The problem here is reference count contention. Each thread has to acquire the lock for
client.getto manage its reference count, which is causing a slowdown. The reason this is only apparent for methods and not functions is that by default, Python will enable deferred reference counting (moving reference counts to the garbage collection cycle) for function objects, but not method objects.Unfortunately, we can't enable it for that many objects, because it extends the lifetime of objects. Many rely on things like
__del__to be called right after something goes out of scope (even though they shouldn't). However, adding an API to explicitly do this might be worthwhile (cc @colesbury)I've spun up a toy build with an API to enable deferred reference counting inside
gc, and that significantly helps the performance. On my end, here's what the benchmark is without deferred reference counts:And here it is with it enabled for
clientandclient.get:Fidget-Spinner commentedon Sep 3, 2024
I don't think we should expose a Python-level API for deferred rc. It makes other implementations worse off and exposes yet another implementation detail to the user. Exposing an unstable C-level API still makes sense.
ZeroIntensity commentedon Sep 3, 2024
Yeah, which is why it would make sense to give that tradeoff to the user.
Anyways, something like
PyUnstable_Object_SetDeferredRefcountsounds reasonable. Though, we do need to account for the assertions inside_PyObject_SetDeferredRefcountand raise an exception if they aren't met instead.ZeroIntensity commentedon Sep 3, 2024
I've created #123635 to add an API for this. I'm leaving it as a draft for now, though.
Yiling-J commentedon Sep 4, 2024
For developers unfamiliar with the C API (me), a Python-level API would be much more accessible, perhaps something like a class attribute
__unsafe_deferred_rc__ = True. While some code might require__del__to be called immediately, I think most code does not. BTWmethodis also mentioned in PEP 703 Deferred Reference CountingZeroIntensity commentedon Sep 4, 2024
I don't think we have a way to denote unstable Python APIs, do we?
Oh! That's good to hear. Adding deferred RC to
client.getprobably did nothing then, but it should have helpedclient.corona10 commentedon Sep 5, 2024
In my opinion, we should avoid providing implementation details to the C API as much as possible.
Deferred reference counting is one of the implementation details of CPython itself, not Python.
cc @vstinner
ZeroIntensity commentedon Sep 5, 2024
That's why my PR puts it in
PyUnstable. Deferred RC is handy for speeding up free-threading, as shown in this case -- it would be nice if we let users take advantage of it.Yiling-J commentedon Sep 6, 2024
Here are the results from 32 cores and 32 threads:
The cost of contention is extremely high. PEP 703 mentions, "A few types of objects, such as top-level functions, code objects, modules, and methods, tend to be frequently accessed by many threads concurrently." However, it seems that while functions have Deferred Reference Counting, methods do not. Python users might not realize this when using free-threading Python, and if they don't read the C API documentation, they may not know that methods can become a bottleneck at scale and they need to enable it manually using C API.
PEP 703 also mentions that lists will have a fast path. However, based on my tests, lists are slow, while tuples perform very quickly.
A benchmark on global list access:
Use tuple as global list(
gl = (1,)):ZeroIntensity commentedon Sep 6, 2024
Ah, this might be a bug: per PEP 703, deferred reference counting is enabled for the actual
functionobjects stored on a type, but not for the bound methods returned by the descriptors -- I'm not sure that was intentional.If we really want to expose a Python API, let's add something unspecific to deferred RC -- something like
sys.thread_hot()could work. That could enable deferred RC as an implementation detail, but would be allowed to change internally in the future; the only key factor that can't change is that it marks an object as "hot" across threads (I think that's pretty universal, and not exposing any implementation details).Yiling-J commentedon Sep 6, 2024
@ZeroIntensity I might have found another bug. In my global list benchmark, if I use
gl = tuple([1]), still a tuple but performance slows down again.Yiling-J commentedon Sep 18, 2024
Both are slow on main branch (main:d8c0fe1). I'm not sure if this is intentional, but if the contention case is expected, then it might not be a bug.
Python 3.14.0a0 experimental free-threading build (heads/main:d8c0fe1, Sep 18 2024, 09:23:28) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
5 remaining items
ZeroIntensity commentedon Nov 13, 2024
Now that we have an official API for DRC, I'm going to close this. Thanks, @Yiling-J!
Yiling-J commentedon Nov 13, 2024
Looks like I'll have to learn some C and CPython to make my cache library scalable. Honestly I still think the best approach would be for CPython to optimize these by default, scalability is also important for free-threading Python. It would be even better if there were an official document telling developers what is scalable for free-threading CPython. @ZeroIntensity
ZeroIntensity commentedon Nov 13, 2024
You shouldn't need to learn much C, you can use the function pretty easily with ctypes:
Yiling-J commentedon Nov 17, 2024
@ZeroIntensity I ran another round of benchmark using
enable_drc, still with the same benchmark code provided in the issue:getfunction:enable_drc(get)enable_drc(get)Seems it's getiing slower when enable drc.
client.getmethod:enable_drc(client)enable_drc(client)a little faster, not much.
Is this the expected behavior of DRC? Ideally, the throughput of the multi-threaded version should align with the single-threaded version, as seen in the 3.13 Immortal behavior. However, it seems the Immortal API is not public yet?
ZeroIntensity commentedon Nov 17, 2024
I'm not sure what the state of DRC is on main, it might still be disabled/suboptimal.
Note that you're not measuring throughput here, you're measuring time taken.
Yiling-J commentedon Nov 17, 2024
That's true but throughput can easily be derived from the time taken, and if the throughput aligns, the time should as well. My main point is that the issue still persists—the performance doesn't match my expectations.
ZeroIntensity commentedon Nov 17, 2024
Sort of, but there's not much actual computing happening here--the interpreter is just trying to pass an object around and mess with its reference count, which will basically always be faster on the main thread because of biased reference counting. So, as a result, the bottleneck is the reference count operation, which isn't really a fair benchmark. This isn't a problem limited to Python; in basically any language with multithreading, you need some sort of atomic operation to use data concurrently, and if you only measure the cost of that against single-threaded code, then of course it will seem slower.
pythongh-123619: Add an unstable C API function for enabling deferred…
pythongh-123619: Add an unstable C API function for enabling deferred…
pythongh-123619: Fix PyUnstable_Object_EnableDeferredRefcount()
gh-123619: Fix PyUnstable_Object_EnableDeferredRefcount() (#151260)
[3.14] gh-123619: Fix PyUnstable_Object_EnableDeferredRefcount() (GH-…
[3.15] gh-123619: Fix PyUnstable_Object_EnableDeferredRefcount() (GH-…
pythongh-123619: Fix PyUnstable_Object_EnableDeferredRefcount() (pyth…