Skip to content

Commit 731e189

Browse files
ma8mancoghlan
ma8ma
authored and
ncoghlan
committed
bpo-25658: Implement PEP 539 for Thread Specific Storage (TSS) API (pythonGH-1362)
See PEP 539 for details. Highlights of changes: - Add Thread Specific Storage (TSS) API - Document the Thread Local Storage (TLS) API as deprecated - Update code that used TLS API to use TSS API
1 parent b8ab9d3 commit 731e189

18 files changed

Lines changed: 651 additions & 108 deletions

‎Doc/c-api/init.rst‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1192,3 +1192,160 @@ These functions are only intended to be used by advanced debugging tools.
11921192
Return the next thread state object after *tstate* from the list of all such
11931193
objects belonging to the same :c:type:`PyInterpreterState` object.
11941194
1195+
1196+
.. _thread-local-storage:
1197+
1198+
Thread Local Storage Support
1199+
============================
1200+
1201+
.. sectionauthor:: Masayuki Yamamoto <ma3yuki.8mamo10@gmail.com>
1202+
1203+
The Python interpreter provides low-level support for thread-local storage
1204+
(TLS) which wraps the underlying native TLS implementation to support the
1205+
Python-level thread local storage API (:class:`threading.local`). The
1206+
CPython C level APIs are similar to those offered by pthreads and Windows:
1207+
use a thread key and functions to associate a :c:type:`void\*` value per
1208+
thread.
1209+
1210+
The GIL does *not* need to be held when calling these functions; they supply
1211+
their own locking.
1212+
1213+
Note that :file:`Python.h` does not include the declaration of the TLS APIs,
1214+
you need to include :file:`pythread.h` to use thread-local storage.
1215+
1216+
.. note::
1217+
None of these API functions handle memory management on behalf of the
1218+
:c:type:`void\*` values. You need to allocate and deallocate them yourself.
1219+
If the :c:type:`void\*` values happen to be :c:type:`PyObject\*`, these
1220+
functions don't do refcount operations on them either.
1221+
1222+
.. _thread-specific-storage-api:
1223+
1224+
Thread Specific Storage (TSS) API
1225+
---------------------------------
1226+
1227+
TSS API is introduced to supersede the use of the existing TLS API within the
1228+
CPython interpreter. This API uses a new type :c:type:`Py_tss_t` instead of
1229+
:c:type:`int` to represent thread keys.
1230+
1231+
.. versionadded:: 3.7
1232+
1233+
.. seealso:: "A New C-API for Thread-Local Storage in CPython" (:pep:`539`)
1234+
1235+
1236+
.. c:type:: Py_tss_t
1237+
1238+
This data structure represents the state of a thread key, the definition of
1239+
which may depend on the underlying TLS implementation, and it has an
1240+
internal field representing the key's initialization state. There are no
1241+
public members in this structure.
1242+
1243+
When :ref:`Py_LIMITED_API <stable>` is not defined, static allocation of
1244+
this type by :c:macro:`Py_tss_NEEDS_INIT` is allowed.
1245+
1246+
1247+
.. c:macro:: Py_tss_NEEDS_INIT
1248+
1249+
This macro expands to the default value for :c:type:`Py_tss_t` variables.
1250+
Note that this macro won't be defined with :ref:`Py_LIMITED_API <stable>`.
1251+
1252+
1253+
Dynamic Allocation
1254+
~~~~~~~~~~~~~~~~~~
1255+
1256+
Dynamic allocation of the :c:type:`Py_tss_t`, required in extension modules
1257+
built with :ref:`Py_LIMITED_API <stable>`, where static allocation of this type
1258+
is not possible due to its implementation being opaque at build time.
1259+
1260+
1261+
.. c:function:: Py_tss_t* PyThread_tss_alloc()
1262+
1263+
Return a value which is the same state as a value initialized with
1264+
:c:macro:`Py_tss_NEEDS_INIT`, or *NULL* in the case of dynamic allocation
1265+
failure.
1266+
1267+
1268+
.. c:function:: void PyThread_tss_free(Py_tss_t *key)
1269+
1270+
Free the given *key* allocated by :c:func:`PyThread_tss_alloc`, after
1271+
first calling :c:func:`PyThread_tss_delete` to ensure any associated
1272+
thread locals have been unassigned. This is a no-op if the *key*
1273+
argument is `NULL`.
1274+
1275+
.. note::
1276+
A freed key becomes a dangling pointer, you should reset the key to
1277+
`NULL`.
1278+
1279+
1280+
Methods
1281+
~~~~~~~
1282+
1283+
The parameter *key* of these functions must not be *NULL*. Moreover, the
1284+
behaviors of :c:func:`PyThread_tss_set` and :c:func:`PyThread_tss_get` are
1285+
undefined if the given :c:type:`Py_tss_t` has not been initialized by
1286+
:c:func:`PyThread_tss_create`.
1287+
1288+
1289+
.. c:function:: int PyThread_tss_is_created(Py_tss_t *key)
1290+
1291+
Return a non-zero value if the given :c:type:`Py_tss_t` has been initialized
1292+
by :c:func:`PyThread_tss_create`.
1293+
1294+
1295+
.. c:function:: int PyThread_tss_create(Py_tss_t *key)
1296+
1297+
Return a zero value on successful initialization of a TSS key. The behavior
1298+
is undefined if the value pointed to by the *key* argument is not
1299+
initialized by :c:macro:`Py_tss_NEEDS_INIT`. This function can be called
1300+
repeatedly on the same key -- calling it on an already initialized key is a
1301+
no-op and immediately returns success.
1302+
1303+
1304+
.. c:function:: void PyThread_tss_delete(Py_tss_t *key)
1305+
1306+
Destroy a TSS key to forget the values associated with the key across all
1307+
threads, and change the key's initialization state to uninitialized. A
1308+
destroyed key is able to be initialized again by
1309+
:c:func:`PyThread_tss_create`. This function can be called repeatedly on
1310+
the same key -- calling it on an already destroyed key is a no-op.
1311+
1312+
1313+
.. c:function:: int PyThread_tss_set(Py_tss_t *key, void *value)
1314+
1315+
Return a zero value to indicate successfully associating a :c:type:`void\*`
1316+
value with a TSS key in the current thread. Each thread has a distinct
1317+
mapping of the key to a :c:type:`void\*` value.
1318+
1319+
1320+
.. c:function:: void* PyThread_tss_get(Py_tss_t *key)
1321+
1322+
Return the :c:type:`void\*` value associated with a TSS key in the current
1323+
thread. This returns *NULL* if no value is associated with the key in the
1324+
current thread.
1325+
1326+
1327+
.. _thread-local-storage-api:
1328+
1329+
Thread Local Storage (TLS) API
1330+
------------------------------
1331+
1332+
.. deprecated:: 3.7
1333+
This API is superseded by
1334+
:ref:`Thread Specific Storage (TSS) API <thread-specific-storage-api>`.
1335+
1336+
.. note::
1337+
This version of the API does not support platforms where the native TLS key
1338+
is defined in a way that cannot be safely cast to ``int``. On such platforms,
1339+
:c:func:`PyThread_create_key` will return immediately with a failure status,
1340+
and the other TLS functions will all be no-ops on such platforms.
1341+
1342+
Due to the compatibility problem noted above, this version of the API should not
1343+
be used in new code.
1344+
1345+
.. c:function:: int PyThread_create_key()
1346+
.. c:function:: void PyThread_delete_key(int key)
1347+
.. c:function:: int PyThread_set_key_value(int key, void *value)
1348+
.. c:function:: void* PyThread_get_key_value(int key)
1349+
.. c:function:: void PyThread_delete_key_value(int key)
1350+
.. c:function:: void PyThread_ReInitTLS()
1351+

‎Doc/whatsnew/3.7.rst‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,38 @@ built-in ``breakpoint()``.
127127
PEP written and implemented by Barry Warsaw
128128

129129

130+
.. _whatsnew37-pep539:
131+
132+
PEP 539: A New C-API for Thread-Local Storage in CPython
133+
--------------------------------------------------------
134+
135+
While Python provides a C API for thread-local storage support; the existing
136+
:ref:`Thread Local Storage (TLS) API <thread-local-storage-api>` has used
137+
:c:type:`int` to represent TLS keys across all platforms. This has not
138+
generally been a problem for officially-support platforms, but that is neither
139+
POSIX-compliant, nor portable in any practical sense.
140+
141+
:pep:`539` changes this by providing a new :ref:`Thread Specific Storage (TSS)
142+
API <thread-specific-storage-api>` to CPython which supersedes use of the
143+
existing TLS API within the CPython interpreter, while deprecating the existing
144+
API. The TSS API uses a new type :c:type:`Py_tss_t` instead of :c:type:`int`
145+
to represent TSS keys--an opaque type the definition of which may depend on
146+
the underlying TLS implementation. Therefore, this will allow to build CPython
147+
on platforms where the native TLS key is defined in a way that cannot be safely
148+
cast to :c:type:`int`.
149+
150+
Note that on platforms where the native TLS key is defined in a way that cannot
151+
be safely cast to :c:type:`int`, all functions of the existing TLS API will be
152+
no-op and immediately return failure. This indicates clearly that the old API
153+
is not supported on platforms where it cannot be used reliably, and that no
154+
effort will be made to add such support.
155+
156+
.. seealso::
157+
158+
:pep:`539` -- A New C-API for Thread-Local Storage in CPython
159+
PEP written by Erik M. Bray; implementation by Masayuki Yamamoto.
160+
161+
130162
Other Language Changes
131163
======================
132164

‎Include/internal/pystate.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ struct _gilstate_runtime_state {
2626
*/
2727
/* TODO: Given interp_main, it may be possible to kill this ref */
2828
PyInterpreterState *autoInterpreterState;
29-
int autoTLSkey;
29+
Py_tss_t autoTSSkey;
3030
};
3131

3232
/* hook for PyEval_GetFrame(), requested for Psyco */

‎Include/pythread.h‎

Lines changed: 63 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@ PyAPI_FUNC(unsigned long) PyThread_get_thread_ident(void);
2929
PyAPI_FUNC(PyThread_type_lock) PyThread_allocate_lock(void);
3030
PyAPI_FUNC(void) PyThread_free_lock(PyThread_type_lock);
3131
PyAPI_FUNC(int) PyThread_acquire_lock(PyThread_type_lock, int);
32-
#define WAIT_LOCK 1
33-
#define NOWAIT_LOCK 0
32+
#define WAIT_LOCK 1
33+
#define NOWAIT_LOCK 0
3434

3535
/* PY_TIMEOUT_T is the integral type used to specify timeouts when waiting
3636
on a lock (see PyThread_acquire_lock_timed() below).
@@ -77,15 +77,69 @@ PyAPI_FUNC(int) PyThread_set_stacksize(size_t);
7777
PyAPI_FUNC(PyObject*) PyThread_GetInfo(void);
7878
#endif
7979

80-
/* Thread Local Storage (TLS) API */
81-
PyAPI_FUNC(int) PyThread_create_key(void);
82-
PyAPI_FUNC(void) PyThread_delete_key(int);
83-
PyAPI_FUNC(int) PyThread_set_key_value(int, void *);
84-
PyAPI_FUNC(void *) PyThread_get_key_value(int);
85-
PyAPI_FUNC(void) PyThread_delete_key_value(int key);
80+
81+
/* Thread Local Storage (TLS) API
82+
TLS API is DEPRECATED. Use Thread Specific Storage (TSS) API.
83+
84+
The existing TLS API has used int to represent TLS keys across all
85+
platforms, but it is not POSIX-compliant. Therefore, the new TSS API uses
86+
opaque data type to represent TSS keys to be compatible (see PEP 539).
87+
*/
88+
PyAPI_FUNC(int) PyThread_create_key(void) Py_DEPRECATED(3.7);
89+
PyAPI_FUNC(void) PyThread_delete_key(int key) Py_DEPRECATED(3.7);
90+
PyAPI_FUNC(int) PyThread_set_key_value(int key, void *value) Py_DEPRECATED(3.7);
91+
PyAPI_FUNC(void *) PyThread_get_key_value(int key) Py_DEPRECATED(3.7);
92+
PyAPI_FUNC(void) PyThread_delete_key_value(int key) Py_DEPRECATED(3.7);
8693

8794
/* Cleanup after a fork */
88-
PyAPI_FUNC(void) PyThread_ReInitTLS(void);
95+
PyAPI_FUNC(void) PyThread_ReInitTLS(void) Py_DEPRECATED(3.7);
96+
97+
98+
#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03070000
99+
/* New in 3.7 */
100+
/* Thread Specific Storage (TSS) API */
101+
102+
typedef struct _Py_tss_t Py_tss_t; /* opaque */
103+
104+
#ifndef Py_LIMITED_API
105+
#if defined(_POSIX_THREADS)
106+
/* Darwin needs pthread.h to know type name the pthread_key_t. */
107+
# include <pthread.h>
108+
# define NATIVE_TSS_KEY_T pthread_key_t
109+
#elif defined(NT_THREADS)
110+
/* In Windows, native TSS key type is DWORD,
111+
but hardcode the unsigned long to avoid errors for include directive.
112+
*/
113+
# define NATIVE_TSS_KEY_T unsigned long
114+
#else
115+
# error "Require native threads. See https://bugs.python.org/issue31370"
116+
#endif
117+
118+
/* When Py_LIMITED_API is not defined, the type layout of Py_tss_t is
119+
exposed to allow static allocation in the API clients. Even in this case,
120+
you must handle TSS keys through API functions due to compatibility.
121+
*/
122+
struct _Py_tss_t {
123+
int _is_initialized;
124+
NATIVE_TSS_KEY_T _key;
125+
};
126+
127+
#undef NATIVE_TSS_KEY_T
128+
129+
/* When static allocation, you must initialize with Py_tss_NEEDS_INIT. */
130+
#define Py_tss_NEEDS_INIT {0}
131+
#endif /* !Py_LIMITED_API */
132+
133+
PyAPI_FUNC(Py_tss_t *) PyThread_tss_alloc(void);
134+
PyAPI_FUNC(void) PyThread_tss_free(Py_tss_t *key);
135+
136+
/* The parameter key must not be NULL. */
137+
PyAPI_FUNC(int) PyThread_tss_is_created(Py_tss_t *key);
138+
PyAPI_FUNC(int) PyThread_tss_create(Py_tss_t *key);
139+
PyAPI_FUNC(void) PyThread_tss_delete(Py_tss_t *key);
140+
PyAPI_FUNC(int) PyThread_tss_set(Py_tss_t *key, void *value);
141+
PyAPI_FUNC(void *) PyThread_tss_get(Py_tss_t *key);
142+
#endif /* New in 3.7 */
89143

90144
#ifdef __cplusplus
91145
}

‎Misc/NEWS.d/next/C API/2017-06-24-14-30-44.bpo-25658.vm8vGE.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Implement PEP 539 for Thread Specific Stroage (TSS) API: it is a new Thread
2+
Local Storage (TLS) API to CPython which would supersede use of the existing
3+
TLS API within the CPython interpreter, while deprecating the existing API.
4+
PEP written by Erik M. Bray, patch by Masayuki Yamamoto.

‎Modules/_testcapimodule.c‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4306,6 +4306,61 @@ py_w_stopcode(PyObject *self, PyObject *args)
43064306
#endif
43074307

43084308

4309+
static PyObject *
4310+
test_pythread_tss_key_state(PyObject *self, PyObject *args)
4311+
{
4312+
Py_tss_t tss_key = Py_tss_NEEDS_INIT;
4313+
if (PyThread_tss_is_created(&tss_key)) {
4314+
return raiseTestError("test_pythread_tss_key_state",
4315+
"TSS key not in an uninitialized state at "
4316+
"creation time");
4317+
}
4318+
if (PyThread_tss_create(&tss_key) != 0) {
4319+
PyErr_SetString(PyExc_RuntimeError, "PyThread_tss_create failed");
4320+
return NULL;
4321+
}
4322+
if (!PyThread_tss_is_created(&tss_key)) {
4323+
return raiseTestError("test_pythread_tss_key_state",
4324+
"PyThread_tss_create succeeded, "
4325+
"but with TSS key in an uninitialized state");
4326+
}
4327+
if (PyThread_tss_create(&tss_key) != 0) {
4328+
return raiseTestError("test_pythread_tss_key_state",
4329+
"PyThread_tss_create unsuccessful with "
4330+
"an already initialized key");
4331+
}
4332+
#define CHECK_TSS_API(expr) \
4333+
(void)(expr); \
4334+
if (!PyThread_tss_is_created(&tss_key)) { \
4335+
return raiseTestError("test_pythread_tss_key_state", \
4336+
"TSS key initialization state was not " \
4337+
"preserved after calling " #expr); }
4338+
CHECK_TSS_API(PyThread_tss_set(&tss_key, NULL));
4339+
CHECK_TSS_API(PyThread_tss_get(&tss_key));
4340+
#undef CHECK_TSS_API
4341+
PyThread_tss_delete(&tss_key);
4342+
if (PyThread_tss_is_created(&tss_key)) {
4343+
return raiseTestError("test_pythread_tss_key_state",
4344+
"PyThread_tss_delete called, but did not "
4345+
"set the key state to uninitialized");
4346+
}
4347+
4348+
Py_tss_t *ptr_key = PyThread_tss_alloc();
4349+
if (ptr_key == NULL) {
4350+
PyErr_SetString(PyExc_RuntimeError, "PyThread_tss_alloc failed");
4351+
return NULL;
4352+
}
4353+
if (PyThread_tss_is_created(ptr_key)) {
4354+
return raiseTestError("test_pythread_tss_key_state",
4355+
"TSS key not in an uninitialized state at "
4356+
"allocation time");
4357+
}
4358+
PyThread_tss_free(ptr_key);
4359+
ptr_key = NULL;
4360+
Py_RETURN_NONE;
4361+
}
4362+
4363+
43094364
static PyMethodDef TestMethods[] = {
43104365
{"raise_exception", raise_exception, METH_VARARGS},
43114366
{"raise_memoryerror", (PyCFunction)raise_memoryerror, METH_NOARGS},
@@ -4518,6 +4573,7 @@ static PyMethodDef TestMethods[] = {
45184573
#ifdef W_STOPCODE
45194574
{"W_STOPCODE", py_w_stopcode, METH_VARARGS},
45204575
#endif
4576+
{"test_pythread_tss_key_state", test_pythread_tss_key_state, METH_VARARGS},
45214577
{NULL, NULL} /* sentinel */
45224578
};
45234579

0 commit comments

Comments
 (0)