Skip to content

Commit fda8211

Browse files
greateggsgregfilmor
greateggsgreg
authored and
filmor
committed
Add user-facing threading guide covering GIL, free-threading, and common pitfalls
1 parent 64ea342 commit fda8211

2 files changed

Lines changed: 215 additions & 0 deletions

File tree

‎doc/source/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ page. Use the `Python.NET issue tracker`_ to report issues.
4545
python
4646
dotnet
4747
codecs
48+
threading
4849
pyreference
4950
reference
5051

‎doc/source/threading.rst‎

Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
Threading
2+
=========
3+
4+
This page explains how Python.NET interacts with the Python Global Interpreter
5+
Lock (GIL) and with managed threads, and what guarantees the runtime makes
6+
when your code is multi-threaded. It covers both classic CPython builds and
7+
the free-threaded build introduced in CPython 3.13 (``Py_GIL_DISABLED``).
8+
9+
The model in one paragraph
10+
--------------------------
11+
12+
Python.NET embeds CPython, so every interaction with a Python object —
13+
including reading a ``PyObject``'s attributes, calling a Python callable,
14+
constructing a Python value, or letting a ``PyObject`` go out of scope — must
15+
happen while the calling thread is *attached* to the interpreter. On a
16+
classic (GIL-enabled) CPython build "attached" means "holds the GIL"; on a
17+
free-threaded build it means "has an active thread state". In both cases the
18+
attachment API is the same: ``Py.GIL()`` on the C# side and
19+
``threading.Thread`` / ``_thread`` on the Python side. Forgetting to attach
20+
will crash the process or corrupt memory.
21+
22+
Acquiring the GIL from C#
23+
-------------------------
24+
25+
When .NET code calls into Python it must hold the GIL. Use the ``Py.GIL()``
26+
disposable to acquire and release it::
27+
28+
using (Py.GIL())
29+
{
30+
dynamic np = Py.Import("numpy");
31+
var arr = np.array(new[] { 1, 2, 3 });
32+
// ... interact with arr ...
33+
}
34+
35+
``Py.GIL()`` is re-entrant: nesting calls on the same thread is harmless and
36+
cheap. Always pair acquisition with disposal — the ``using`` form does this
37+
automatically, and you must release the GIL on the same thread that acquired
38+
it.
39+
40+
If you need a Python object to outlive the ``using`` block, copy what you
41+
need (e.g. ``.As<int[]>()`` or ``new PyObject(value)``) before releasing the
42+
GIL.
43+
44+
Releasing the GIL for long-running .NET work
45+
--------------------------------------------
46+
47+
If a managed call holds the GIL but then does long-running work that does not
48+
touch Python (heavy CPU, blocking I/O, native interop), release the GIL so
49+
other Python threads can run::
50+
51+
IntPtr threadState = PythonEngine.BeginAllowThreads();
52+
try
53+
{
54+
DoCpuHeavyWork(); // safe: no Python C API calls
55+
}
56+
finally
57+
{
58+
PythonEngine.EndAllowThreads(threadState);
59+
}
60+
61+
Inside the ``BeginAllowThreads``/``EndAllowThreads`` block you must not touch
62+
any Python object. If you need to talk to Python from worker threads spawned
63+
in this region, those threads must acquire the GIL themselves with
64+
``Py.GIL()``.
65+
66+
Calling .NET from Python threads
67+
--------------------------------
68+
69+
Calling a managed method from a Python ``threading.Thread`` works
70+
transparently — Python.NET handles GIL acquisition/release around the
71+
managed call. The managed code sees the GIL held on entry and is free to
72+
release it via ``BeginAllowThreads`` if it does its own blocking work.
73+
74+
Calling Python from CLR threads
75+
-------------------------------
76+
77+
A CLR thread that was *not* spawned by Python (a thread-pool task, a
78+
``Thread`` started in C#, an ``async`` continuation that resumed on a
79+
different thread, etc.) must acquire the GIL before touching any
80+
``PyObject``::
81+
82+
Task.Run(() =>
83+
{
84+
using (Py.GIL())
85+
{
86+
// safe to use PyObjects here
87+
}
88+
});
89+
90+
Forgetting this is the most common pythonnet threading bug. Symptoms range
91+
from immediate segfaults to subtle refcount corruption that crashes much
92+
later.
93+
94+
Reference counting and finalizers
95+
---------------------------------
96+
97+
``PyObject`` follows the .NET ``IDisposable`` pattern. ``Dispose()`` (or the
98+
end of a ``using`` block) drops the underlying Python reference; the GC
99+
finalizer queues the same release for the next time Python.NET is on the GIL.
100+
101+
Two practical consequences:
102+
103+
* **Don't share a single ``PyObject`` instance across threads without
104+
serialising access.** ``PyObject`` is not internally locked. If multiple
105+
threads concurrently dispose the same instance, the underlying refcount can
106+
go negative.
107+
108+
* **Don't rely on the GC finalizer running promptly.** The PyObject is only
109+
freed when a Python.NET API later reacquires the GIL. If your application
110+
shuts down without that happening, finalizable PyObjects can be reported as
111+
leaked.
112+
113+
Free-threaded Python (PEP 703)
114+
------------------------------
115+
116+
Starting with the free-threaded CPython 3.13+ build (``Py_GIL_DISABLED``),
117+
the GIL is no longer the serialisation point for Python C API calls.
118+
Python.NET is tested against the ``3.14t`` (free-threaded) interpreter and
119+
behaves as follows under that build:
120+
121+
* ``Py.GIL()`` still acquires a thread state. It is functionally a no-op
122+
for mutual exclusion but is still required for thread-state attachment.
123+
Existing code that uses ``using (Py.GIL())`` continues to work without
124+
changes.
125+
* ``PythonEngine.BeginAllowThreads`` / ``EndAllowThreads`` similarly
126+
manage the thread state and are still needed if you want the GC and
127+
other Python threads to run while you're in long-running unmanaged code.
128+
* Internal Python.NET caches (the reflection cache, generic-type binding
129+
cache, dynamic-dispatch cache, module attribute cache, the interned-
130+
string table, etc.) are thread-safe. You may read and call CLR types
131+
concurrently from any number of threads without external locking.
132+
* The reference-counting protocol uses CPython's ``Py_REFCNT`` symbol on
133+
3.14+, which returns the merged biased + shared refcount; values you read
134+
from ``PyObject.Refcount`` are correct under free-threading.
135+
136+
Behaviour that is *unchanged* between GIL and free-threaded builds:
137+
138+
* A managed object exposed to Python (e.g. via ``System.Object`` or a
139+
Python subclass of a CLR type) is still owned by a single CLR side: you
140+
must not mutate its plain CLR fields from multiple threads without your
141+
own locking. Python.NET only protects its own bookkeeping, not your
142+
domain data.
143+
* Operations on a single ``PyObject`` instance still require external
144+
serialisation — see "Reference counting" above.
145+
146+
Patterns
147+
--------
148+
149+
Concurrent CLR access from Python
150+
"""""""""""""""""""""""""""""""""
151+
152+
Hammering CLR attributes / generic types from many threads is supported::
153+
154+
from threading import Thread
155+
import System
156+
from System.Collections.Generic import List
157+
158+
def worker():
159+
for _ in range(1000):
160+
_ = System.String.Empty
161+
_ = List[int]()
162+
163+
threads = [Thread(target=worker) for _ in range(8)]
164+
for t in threads: t.start()
165+
for t in threads: t.join()
166+
167+
This works on both GIL and free-threaded builds.
168+
169+
Python callback invoked from a managed thread
170+
"""""""""""""""""""""""""""""""""""""""""""""
171+
172+
If a managed component calls back into a Python delegate from a thread it
173+
spawned, that callback path acquires the GIL internally — you do not need to
174+
add ``Py.GIL()`` around the Python code in the delegate.
175+
176+
Spawning a managed thread from inside ``Py.GIL()``
177+
""""""""""""""""""""""""""""""""""""""""""""""""""
178+
179+
If you start a managed thread while holding the GIL and the thread needs to
180+
call back into Python, release the GIL first so the new thread can acquire
181+
it::
182+
183+
using (Py.GIL())
184+
{
185+
var pyCallback = scope.Get("on_done");
186+
PythonEngine.BeginAllowThreads(); // let workers acquire the GIL
187+
try
188+
{
189+
// spawn workers, wait for them...
190+
}
191+
finally
192+
{
193+
PythonEngine.EndAllowThreads(...);
194+
}
195+
}
196+
197+
Without the ``BeginAllowThreads`` the spawned thread blocks forever waiting
198+
for the GIL the parent thread is still holding.
199+
200+
Common pitfalls
201+
---------------
202+
203+
* Holding ``Py.GIL()`` across ``Task.Run`` / ``await`` boundaries. Async
204+
continuations can resume on a different thread; the GIL handle is
205+
thread-bound and must be released on the same thread that acquired it.
206+
* Passing a ``PyObject`` to a managed worker without taking ownership. If
207+
the producer disposes its handle while the consumer is still using it,
208+
the worker will operate on a freed object. Wrap the producer's
209+
``PyObject`` with ``new PyObject(value)`` before handing it off, or use
210+
``NewReference()``.
211+
* Calling a Python callable that does CPU-bound work without releasing the
212+
GIL. Other Python threads cannot make progress in that case, even on a
213+
free-threaded build where the GIL is otherwise a no-op (the callable
214+
itself may still touch contended Python state).

0 commit comments

Comments
 (0)