Skip to content

Commit d5cacbb

Browse files
ncoghlanncoghlan
committed
PEP 489: Multi-phase extension module initialization
Known limitations of the current implementation: - documentation changes are incomplete - there's a reference leak I haven't tracked down yet The leak is most visible by running: ./python -m test -R3:3 test_importlib However, you can also see it by running: ./python -X showrefcount Importing the array or _testmultiphase modules, and then deleting them from both sys.modules and the local namespace shows significant increases in the total number of active references each cycle. By contrast, with _testcapi (which continues to use single-phase initialisation) the global refcounts stabilise after a couple of cycles.
1 parent ec219ba commit d5cacbb

34 files changed

Lines changed: 4451 additions & 3113 deletions

‎Doc/c-api/module.rst‎

Lines changed: 66 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,6 @@ Module Objects
77

88
.. index:: object: module
99

10-
There are only a few functions special to module objects.
11-
1210

1311
.. c:var:: PyTypeObject PyModule_Type
1412
@@ -109,6 +107,14 @@ There are only a few functions special to module objects.
109107
unencodable filenames, use :c:func:`PyModule_GetFilenameObject` instead.
110108
111109
110+
Per-interpreter module state
111+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
112+
113+
Single-phase initialization creates singleton modules that can store additional
114+
information as part of the interpreter, allow that state to be retrieved later
115+
with only a reference to the module definition, rather than to the module
116+
itself.
117+
112118
.. c:function:: void* PyModule_GetState(PyObject *module)
113119
114120
Return the "state" of the module, that is, a pointer to the block of memory
@@ -146,27 +152,6 @@ There are only a few functions special to module objects.
146152
Initializing C modules
147153
^^^^^^^^^^^^^^^^^^^^^^
148154
149-
These functions are usually used in the module initialization function.
150-
151-
.. c:function:: PyObject* PyModule_Create(PyModuleDef *module)
152-
153-
Create a new module object, given the definition in *module*. This behaves
154-
like :c:func:`PyModule_Create2` with *module_api_version* set to
155-
:const:`PYTHON_API_VERSION`.
156-
157-
158-
.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version)
159-
160-
Create a new module object, given the definition in *module*, assuming the
161-
API version *module_api_version*. If that version does not match the version
162-
of the running interpreter, a :exc:`RuntimeWarning` is emitted.
163-
164-
.. note::
165-
166-
Most uses of this function should be using :c:func:`PyModule_Create`
167-
instead; only use this if you are sure you need it.
168-
169-
170155
.. c:type:: PyModuleDef
171156
172157
This struct holds all information that is needed to create a module object.
@@ -210,9 +195,10 @@ These functions are usually used in the module initialization function.
210195
A pointer to a table of module-level functions, described by
211196
:c:type:`PyMethodDef` values. Can be *NULL* if no functions are present.
212197
213-
.. c:member:: inquiry m_reload
198+
.. c:member:: PyModuleDef_Slot* m_slots
214199
215-
Currently unused, should be *NULL*.
200+
An array of slot definitions for multi-phase initialization, terminated by
201+
a *NULL* entry.
216202
217203
.. c:member:: traverseproc m_traverse
218204
@@ -229,14 +215,68 @@ These functions are usually used in the module initialization function.
229215
A function to call during deallocation of the module object, or *NULL* if
230216
not needed.
231217
218+
The module initialization function may create and return the module object
219+
directly. This is referred to as "single-phase initialization", and uses one
220+
of the following two module creation functions:
221+
222+
.. c:function:: PyObject* PyModule_Create(PyModuleDef *module)
223+
224+
Create a new module object, given the definition in *module*. This behaves
225+
like :c:func:`PyModule_Create2` with *module_api_version* set to
226+
:const:`PYTHON_API_VERSION`.
227+
228+
229+
.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version)
230+
231+
Create a new module object, given the definition in *module*, assuming the
232+
API version *module_api_version*. If that version does not match the version
233+
of the running interpreter, a :exc:`RuntimeWarning` is emitted.
234+
235+
.. note::
236+
237+
Most uses of this function should be using :c:func:`PyModule_Create`
238+
instead; only use this if you are sure you need it.
239+
240+
241+
Alternatively, the module initialization function may instead return a
242+
:c:type:`PyModuleDef` instance with a non-empty ``m_slots`` array. This is
243+
referred to as "multi-phase initialization", and ``PyModuleDef`` instance
244+
should be initialized with the following function:
245+
246+
.. c:function:: PyObject* PyModuleDef_Init(PyModuleDef *module)
247+
248+
Ensures a module definition is a properly initialized Python object that
249+
correctly reports its type and reference count.
250+
251+
.. XXX (ncoghlan): It's not clear if it makes sense to document PyModule_ExecDef
252+
PyModule_FromDefAndSpec or PyModule_FromDefAndSpec2 here, as end user code
253+
generally shouldn't be calling those.
254+
255+
The module initialization function (if using single phase initialization) or
256+
a function called from a module execution slot (if using multiphase
257+
initialization), can use the following functions to help initialize the module
258+
state:
259+
260+
.. c:function:: int PyModule_SetDocString(PyObject *module, const char *docstring)
261+
262+
Set the docstring for *module* to *docstring*. Return ``-1`` on error, ``0``
263+
on success.
264+
265+
.. c:function:: int PyModule_AddFunctions(PyObject *module, PyMethodDef *functions)
266+
267+
Add the functions from the ``NULL`` terminated *functions* array to *module*.
268+
Refer to the :c:type:`PyMethodDef` documentation for details on individual
269+
entries (due to the lack of a shared module namespace, module level
270+
"functions" implemented in C typically receive the module as their first
271+
parameter, making them similar to instance methods on Python classes).
272+
232273
233274
.. c:function:: int PyModule_AddObject(PyObject *module, const char *name, PyObject *value)
234275
235276
Add an object to *module* as *name*. This is a convenience function which can
236277
be used from the module's initialization function. This steals a reference to
237278
*value*. Return ``-1`` on error, ``0`` on success.
238279
239-
240280
.. c:function:: int PyModule_AddIntConstant(PyObject *module, const char *name, long value)
241281
242282
Add an integer constant to *module* as *name*. This convenience function can be

‎Doc/library/importlib.rst‎

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,12 @@ generically as an :term:`importer`) to participate in the import process.
5555
:pep:`451`
5656
A ModuleSpec Type for the Import System
5757

58+
:pep:`488`
59+
Elimination of PYO files
60+
61+
:pep:`489`
62+
Multi-phase extension module initialization
63+
5864
:pep:`3120`
5965
Using UTF-8 as the Default Source Encoding
6066

@@ -756,9 +762,9 @@ find and load modules.
756762
Only class methods are defined by this class to alleviate the need for
757763
instantiation.
758764

759-
.. note::
760-
Due to limitations in the extension module C-API, for now
761-
BuiltinImporter does not implement :meth:`Loader.exec_module`.
765+
.. versionchanged:: 3.5
766+
As part of :pep:`489`, the builtin importer now implements
767+
:meth:`Loader.create_module` and :meth:`Loader.exec_module`
762768

763769

764770
.. class:: FrozenImporter
@@ -973,14 +979,18 @@ find and load modules.
973979

974980
Path to the extension module.
975981

976-
.. method:: load_module(name=None)
982+
.. method:: create_module(spec)
983+
984+
Creates the module object from the given specification in accordance
985+
with :pep:`489`.
986+
987+
.. versionadded:: 3.5
988+
989+
.. method:: exec_module(module)
977990

978-
Loads the extension module if and only if *fullname* is the same as
979-
:attr:`name` or is ``None``.
991+
Initializes the given module object in accordance with :pep:`489`.
980992

981-
.. note::
982-
Due to limitations in the extension module C-API, for now
983-
ExtensionFileLoader does not implement :meth:`Loader.exec_module`.
993+
.. versionadded:: 3.5
984994

985995
.. method:: is_package(fullname)
986996

‎Doc/whatsnew/3.5.rst‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ Implementation improvements:
9393
(:issue:`19977`).
9494

9595
* :pep:`488`, the elimination of ``.pyo`` files.
96+
* :pep:`489`, multi-phase initialization of extension modules.
9697

9798
Significantly Improved Library Modules:
9899

@@ -265,6 +266,21 @@ updated API to help with this change.
265266
:pep:`488` -- Elimination of PYO files
266267

267268

269+
PEP 489: Multi-phase extension module initialization
270+
----------------------------------------------------
271+
272+
:pep:`489` updates extension module initialization to take advantage of the
273+
two step module loading mechanism introduced by :pep:`451` in Python 3.4.
274+
275+
This change brings the import semantics of extension modules that opt-in to
276+
using the new mechanism much closer to those of Python source and bytecode
277+
modules, including the ability to any valid identifier as a module name,
278+
rather than being restricted to ASCII.
279+
280+
.. seealso::
281+
282+
:pep:`488` -- Multi-phase extension module initialization
283+
268284
Other Language Changes
269285
======================
270286

@@ -667,7 +683,7 @@ time
667683
tkinter
668684
-------
669685

670-
* The :module:`tkinter._fix` module used for setting up the Tcl/Tk environment
686+
* The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment
671687
on Windows has been replaced by a private function in the :module:`_tkinter`
672688
module which makes no permanent changes to environment variables.
673689
(Contributed by Zachary Ware in :issue:`20035`.)
@@ -1012,7 +1028,6 @@ Changes in the Python API
10121028
program depends on patching the module level variable to capture the debug
10131029
output, you will need to update it to capture sys.stderr instead.
10141030

1015-
10161031
Changes in the C API
10171032
--------------------
10181033

‎Include/modsupport.h‎

Lines changed: 39 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,13 @@ extern "C" {
1212
/* If PY_SSIZE_T_CLEAN is defined, each functions treats #-specifier
1313
to mean Py_ssize_t */
1414
#ifdef PY_SSIZE_T_CLEAN
15-
#define PyArg_Parse _PyArg_Parse_SizeT
16-
#define PyArg_ParseTuple _PyArg_ParseTuple_SizeT
17-
#define PyArg_ParseTupleAndKeywords _PyArg_ParseTupleAndKeywords_SizeT
18-
#define PyArg_VaParse _PyArg_VaParse_SizeT
19-
#define PyArg_VaParseTupleAndKeywords _PyArg_VaParseTupleAndKeywords_SizeT
20-
#define Py_BuildValue _Py_BuildValue_SizeT
21-
#define Py_VaBuildValue _Py_VaBuildValue_SizeT
15+
#define PyArg_Parse _PyArg_Parse_SizeT
16+
#define PyArg_ParseTuple _PyArg_ParseTuple_SizeT
17+
#define PyArg_ParseTupleAndKeywords _PyArg_ParseTupleAndKeywords_SizeT
18+
#define PyArg_VaParse _PyArg_VaParse_SizeT
19+
#define PyArg_VaParseTupleAndKeywords _PyArg_VaParseTupleAndKeywords_SizeT
20+
#define Py_BuildValue _Py_BuildValue_SizeT
21+
#define Py_VaBuildValue _Py_VaBuildValue_SizeT
2222
#else
2323
PyAPI_FUNC(PyObject *) _Py_VaBuildValue_SizeT(const char *, va_list);
2424
#endif
@@ -49,6 +49,9 @@ PyAPI_FUNC(int) PyModule_AddIntConstant(PyObject *, const char *, long);
4949
PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char *);
5050
#define PyModule_AddIntMacro(m, c) PyModule_AddIntConstant(m, #c, c)
5151
#define PyModule_AddStringMacro(m, c) PyModule_AddStringConstant(m, #c, c)
52+
PyAPI_FUNC(int) PyModule_SetDocString(PyObject *, const char *);
53+
PyAPI_FUNC(int) PyModule_AddFunctions(PyObject *, PyMethodDef *);
54+
PyAPI_FUNC(int) PyModule_ExecDef(PyObject *module, PyModuleDef *def);
5255

5356
#define Py_CLEANUP_SUPPORTED 0x20000
5457

@@ -67,35 +70,35 @@ PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char
6770
Please add a line or two to the top of this log for each API
6871
version change:
6972
70-
22-Feb-2006 MvL 1013 PEP 353 - long indices for sequence lengths
73+
22-Feb-2006 MvL 1013 PEP 353 - long indices for sequence lengths
7174
72-
19-Aug-2002 GvR 1012 Changes to string object struct for
73-
interning changes, saving 3 bytes.
75+
19-Aug-2002 GvR 1012 Changes to string object struct for
76+
interning changes, saving 3 bytes.
7477
75-
17-Jul-2001 GvR 1011 Descr-branch, just to be on the safe side
78+
17-Jul-2001 GvR 1011 Descr-branch, just to be on the safe side
7679
7780
25-Jan-2001 FLD 1010 Parameters added to PyCode_New() and
7881
PyFrame_New(); Python 2.1a2
7982
8083
14-Mar-2000 GvR 1009 Unicode API added
8184
82-
3-Jan-1999 GvR 1007 Decided to change back! (Don't reuse 1008!)
85+
3-Jan-1999 GvR 1007 Decided to change back! (Don't reuse 1008!)
8386
84-
3-Dec-1998 GvR 1008 Python 1.5.2b1
87+
3-Dec-1998 GvR 1008 Python 1.5.2b1
8588
86-
18-Jan-1997 GvR 1007 string interning and other speedups
89+
18-Jan-1997 GvR 1007 string interning and other speedups
8790
88-
11-Oct-1996 GvR renamed Py_Ellipses to Py_Ellipsis :-(
91+
11-Oct-1996 GvR renamed Py_Ellipses to Py_Ellipsis :-(
8992
90-
30-Jul-1996 GvR Slice and ellipses syntax added
93+
30-Jul-1996 GvR Slice and ellipses syntax added
9194
92-
23-Jul-1996 GvR For 1.4 -- better safe than sorry this time :-)
95+
23-Jul-1996 GvR For 1.4 -- better safe than sorry this time :-)
9396
94-
7-Nov-1995 GvR Keyword arguments (should've been done at 1.3 :-( )
97+
7-Nov-1995 GvR Keyword arguments (should've been done at 1.3 :-( )
9598
96-
10-Jan-1995 GvR Renamed globals to new naming scheme
99+
10-Jan-1995 GvR Renamed globals to new naming scheme
97100
98-
9-Jan-1995 GvR Initial version (incompatible with older API)
101+
9-Jan-1995 GvR Initial version (incompatible with older API)
99102
*/
100103

101104
/* The PYTHON_ABI_VERSION is introduced in PEP 384. For the lifetime of
@@ -105,21 +108,34 @@ PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char
105108
#define PYTHON_ABI_STRING "3"
106109

107110
#ifdef Py_TRACE_REFS
108-
/* When we are tracing reference counts, rename PyModule_Create2 so
111+
/* When we are tracing reference counts, rename module creation functions so
109112
modules compiled with incompatible settings will generate a
110113
link-time error. */
111114
#define PyModule_Create2 PyModule_Create2TraceRefs
115+
#define PyModule_FromDefAndSpec2 PyModule_FromDefAndSpec2TraceRefs
112116
#endif
113117

114118
PyAPI_FUNC(PyObject *) PyModule_Create2(struct PyModuleDef*,
115119
int apiver);
116120

117121
#ifdef Py_LIMITED_API
118122
#define PyModule_Create(module) \
119-
PyModule_Create2(module, PYTHON_ABI_VERSION)
123+
PyModule_Create2(module, PYTHON_ABI_VERSION)
120124
#else
121125
#define PyModule_Create(module) \
122-
PyModule_Create2(module, PYTHON_API_VERSION)
126+
PyModule_Create2(module, PYTHON_API_VERSION)
127+
#endif
128+
129+
PyAPI_FUNC(PyObject *) PyModule_FromDefAndSpec2(PyModuleDef *def,
130+
PyObject *spec,
131+
int module_api_version);
132+
133+
#ifdef Py_LIMITED_API
134+
#define PyModule_FromDefAndSpec(module, spec) \
135+
PyModule_FromDefAndSpec2(module, spec, PYTHON_ABI_VERSION)
136+
#else
137+
#define PyModule_FromDefAndSpec(module, spec) \
138+
PyModule_FromDefAndSpec2(module, spec, PYTHON_API_VERSION)
123139
#endif
124140

125141
#ifndef Py_LIMITED_API

‎Include/moduleobject.h‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,9 @@ PyAPI_FUNC(void) _PyModule_ClearDict(PyObject *);
3030
PyAPI_FUNC(struct PyModuleDef*) PyModule_GetDef(PyObject*);
3131
PyAPI_FUNC(void*) PyModule_GetState(PyObject*);
3232

33+
PyAPI_FUNC(PyObject *) PyModuleDef_Init(struct PyModuleDef*);
34+
PyTypeObject PyModuleDef_Type;
35+
3336
typedef struct PyModuleDef_Base {
3437
PyObject_HEAD
3538
PyObject* (*m_init)(void);
@@ -44,18 +47,29 @@ typedef struct PyModuleDef_Base {
4447
NULL, /* m_copy */ \
4548
}
4649

50+
typedef struct PyModuleDef_Slot{
51+
int slot;
52+
void *value;
53+
} PyModuleDef_Slot;
54+
4755
typedef struct PyModuleDef{
4856
PyModuleDef_Base m_base;
4957
const char* m_name;
5058
const char* m_doc;
5159
Py_ssize_t m_size;
5260
PyMethodDef *m_methods;
53-
inquiry m_reload;
61+
PyModuleDef_Slot* m_slots;
5462
traverseproc m_traverse;
5563
inquiry m_clear;
5664
freefunc m_free;
5765
}PyModuleDef;
5866

67+
#define Py_mod_create 1
68+
#define Py_mod_exec 2
69+
70+
#ifndef Py_LIMITED_API
71+
#define _Py_mod_LAST_SLOT 2
72+
#endif
5973

6074
#ifdef __cplusplus
6175
}

0 commit comments

Comments
 (0)