Skip to content

Commit 4d70562

Browse files
vstinnervstinner
committed
Issue python#3329: Add new APIs to customize memory allocators
* Add a new PyMemAllocators structure * New functions: - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory allocator functions - PyMem_GetRawAllocators(), PyMem_SetRawAllocators() - PyMem_GetAllocators(), PyMem_SetAllocators() - PyMem_SetupDebugHooks() - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators() * Add unit test for PyMem_Malloc(0) and PyObject_Malloc(0) * Add unit test for new get/set allocators functions * PyObject_Malloc() now falls back on PyMem_Malloc() instead of malloc() if size is bigger than SMALL_REQUEST_THRESHOLD, and PyObject_Realloc() falls back on PyMem_Realloc() instead of realloc() * PyMem_Malloc() and PyMem_Realloc() now always call malloc() and realloc(), instead of calling PyObject_Malloc() and PyObject_Realloc() in debug mode
1 parent 8c18da2 commit 4d70562

6 files changed

Lines changed: 772 additions & 212 deletions

File tree

‎Doc/c-api/memory.rst‎

Lines changed: 120 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,15 +84,59 @@ the C library allocator as shown in the previous example, the allocated memory
8484
for the I/O buffer escapes completely the Python memory manager.
8585

8686

87+
Raw Memory Interface
88+
====================
89+
90+
The following function are wrappers to system allocators: :c:func:`malloc`,
91+
:c:func:`realloc`, :c:func:`free`. These functions are thread-safe, the
92+
:term:`GIL <global interpreter lock>` does not need to be held to use these
93+
functions.
94+
95+
The behaviour of requesting zero bytes is not defined: return *NULL* or a
96+
distinct non-*NULL* pointer depending on the platform. Use
97+
:c:func:`PyMem_Malloc` and :c:func:`PyMem_Realloc` to have a well defined
98+
behaviour.
99+
100+
.. versionadded:: 3.4
101+
102+
.. c:function:: void* PyMem_RawMalloc(size_t n)
103+
104+
Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
105+
allocated memory, or *NULL* if the request fails. The memory
106+
will not have been initialized in any way.
107+
108+
109+
.. c:function:: void* PyMem_RawRealloc(void *p, size_t n)
110+
111+
Resizes the memory block pointed to by *p* to *n* bytes. The contents will
112+
be unchanged to the minimum of the old and the new sizes. If *p* is *NULL*,
113+
the call is equivalent to ``PyMem_RawMalloc(n)``. Unless *p* is *NULL*, it
114+
must have been returned by a previous call to :c:func:`PyMem_RawMalloc` or
115+
:c:func:`PyMem_RawRealloc`. If the request fails, :c:func:`PyMem_RawRealloc`
116+
returns *NULL* and *p* remains a valid pointer to the previous memory area.
117+
118+
119+
.. c:function:: void PyMem_RawFree(void *p)
120+
121+
Frees the memory block pointed to by *p*, which must have been returned by a
122+
previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`.
123+
Otherwise, or if ``PyMem_Free(p)`` has been called before, undefined
124+
behavior occurs. If *p* is *NULL*, no operation is performed.
125+
126+
87127
.. _memoryinterface:
88128
89129
Memory Interface
90130
================
91131
92132
The following function sets, modeled after the ANSI C standard, but specifying
93133
behavior when requesting zero bytes, are available for allocating and releasing
94-
memory from the Python heap:
134+
memory from the Python heap.
95135
136+
.. warning::
137+
138+
The :term:`GIL <global interpreter lock>` must be held when using these
139+
functions.
96140
97141
.. c:function:: void* PyMem_Malloc(size_t n)
98142
@@ -155,6 +199,81 @@ versions and is therefore deprecated in extension modules.
155199
:c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`.
156200
157201
202+
Customize Memory Allocators
203+
===========================
204+
205+
.. versionadded:: 3.4
206+
207+
.. c:type:: PyMemAllocators
208+
209+
Structure used to describe memory allocator. This structure has
210+
four fields:
211+
212+
+----------------------------------------------------------+-----------------+
213+
| Field | Meaning |
214+
+==========================================================+=================+
215+
| ``void *ctx`` | user data |
216+
+----------------------------------------------------------+-----------------+
217+
| ``void* malloc(void *ctx, size_t size)`` | allocate memory |
218+
+----------------------------------------------------------+-----------------+
219+
| ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate memory |
220+
| | or resize a |
221+
| | memory block |
222+
+----------------------------------------------------------+-----------------+
223+
| ``void free(void *ctx, void *ptr)`` | release memory |
224+
+----------------------------------------------------------+-----------------+
225+
226+
.. c:function:: void PyMem_GetRawAllocators(PyMemAllocators *allocators)
227+
228+
Get internal functions of :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`
229+
and :c:func:`PyMem_RawFree`.
230+
231+
.. c:function:: void PyMem_SetRawAllocators(PyMemAllocators *allocators)
232+
233+
Set internal functions of :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`
234+
and :c:func:`PyMem_RawFree`.
235+
236+
:c:func:`PyMem_SetupDebugHooks` should be called to reinstall debug hooks if
237+
new functions do no call original functions anymore.
238+
239+
.. c:function:: void PyMem_GetAllocators(PyMemAllocators *allocators)
240+
241+
Get internal functions of :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`
242+
and :c:func:`PyMem_Free`.
243+
244+
.. c:function:: void PyMem_SetAllocators(PyMemAllocators *allocators)
245+
246+
Set internal functions of :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`
247+
and :c:func:`PyMem_Free`.
248+
249+
``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return *NULL*: it
250+
would be treated as an error.
251+
252+
:c:func:`PyMem_SetupDebugHooks` should be called to reinstall debug hooks if
253+
new functions do no call original functions anymore.
254+
255+
.. c:function:: void PyMem_SetupDebugHooks(void)
256+
257+
Setup hooks to detect bugs in the following Python memory allocator
258+
functions:
259+
260+
- :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`,
261+
:c:func:`PyMem_RawFree`
262+
- :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Free`
263+
- :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`,
264+
:c:func:`PyObject_Free`
265+
266+
Newly allocated memory is filled with the byte ``0xCB``, freed memory is
267+
filled with the byte ``0xDB``. Additionnal checks:
268+
269+
- detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
270+
allocated by :c:func:`PyMem_Malloc`
271+
- detect write before the start of the buffer (buffer underflow)
272+
- detect write after the end of the buffer (buffer overflow)
273+
274+
The function does nothing if Python is not compiled is debug mode.
275+
276+
158277
.. _memoryexamples:
159278
160279
Examples

‎Include/objimpl.h‎

Lines changed: 39 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -94,9 +94,9 @@ PyObject_{New, NewVar, Del}.
9494
the object gets initialized via PyObject_{Init, InitVar} after obtaining
9595
the raw memory.
9696
*/
97-
PyAPI_FUNC(void *) PyObject_Malloc(size_t);
98-
PyAPI_FUNC(void *) PyObject_Realloc(void *, size_t);
99-
PyAPI_FUNC(void) PyObject_Free(void *);
97+
PyAPI_FUNC(void *) PyObject_Malloc(size_t size);
98+
PyAPI_FUNC(void *) PyObject_Realloc(void *ptr, size_t new_size);
99+
PyAPI_FUNC(void) PyObject_Free(void *ptr);
100100

101101
/* This function returns the number of allocated memory blocks, regardless of size */
102102
PyAPI_FUNC(Py_ssize_t) _Py_GetAllocatedBlocks(void);
@@ -106,41 +106,46 @@ PyAPI_FUNC(Py_ssize_t) _Py_GetAllocatedBlocks(void);
106106
#ifndef Py_LIMITED_API
107107
PyAPI_FUNC(void) _PyObject_DebugMallocStats(FILE *out);
108108
#endif /* #ifndef Py_LIMITED_API */
109-
#ifdef PYMALLOC_DEBUG /* WITH_PYMALLOC && PYMALLOC_DEBUG */
110-
PyAPI_FUNC(void *) _PyObject_DebugMalloc(size_t nbytes);
111-
PyAPI_FUNC(void *) _PyObject_DebugRealloc(void *p, size_t nbytes);
112-
PyAPI_FUNC(void) _PyObject_DebugFree(void *p);
113-
PyAPI_FUNC(void) _PyObject_DebugDumpAddress(const void *p);
114-
PyAPI_FUNC(void) _PyObject_DebugCheckAddress(const void *p);
115-
PyAPI_FUNC(void *) _PyObject_DebugMallocApi(char api, size_t nbytes);
116-
PyAPI_FUNC(void *) _PyObject_DebugReallocApi(char api, void *p, size_t nbytes);
117-
PyAPI_FUNC(void) _PyObject_DebugFreeApi(char api, void *p);
118-
PyAPI_FUNC(void) _PyObject_DebugCheckAddressApi(char api, const void *p);
119-
PyAPI_FUNC(void *) _PyMem_DebugMalloc(size_t nbytes);
120-
PyAPI_FUNC(void *) _PyMem_DebugRealloc(void *p, size_t nbytes);
121-
PyAPI_FUNC(void) _PyMem_DebugFree(void *p);
122-
#define PyObject_MALLOC _PyObject_DebugMalloc
123-
#define PyObject_Malloc _PyObject_DebugMalloc
124-
#define PyObject_REALLOC _PyObject_DebugRealloc
125-
#define PyObject_Realloc _PyObject_DebugRealloc
126-
#define PyObject_FREE _PyObject_DebugFree
127-
#define PyObject_Free _PyObject_DebugFree
128-
129-
#else /* WITH_PYMALLOC && ! PYMALLOC_DEBUG */
109+
#endif
110+
111+
/* Macros */
130112
#define PyObject_MALLOC PyObject_Malloc
131113
#define PyObject_REALLOC PyObject_Realloc
132114
#define PyObject_FREE PyObject_Free
133-
#endif
134-
135-
#else /* ! WITH_PYMALLOC */
136-
#define PyObject_MALLOC PyMem_MALLOC
137-
#define PyObject_REALLOC PyMem_REALLOC
138-
#define PyObject_FREE PyMem_FREE
139-
140-
#endif /* WITH_PYMALLOC */
141-
142115
#define PyObject_Del PyObject_Free
143-
#define PyObject_DEL PyObject_FREE
116+
#define PyObject_DEL PyObject_Free
117+
118+
/* Get internal functions of PyObject_Malloc(), PyObject_Realloc() and
119+
PyObject_Free(). *ctx_p is an arbitrary user value. */
120+
PyAPI_FUNC(void) PyObject_GetAllocators(PyMemAllocators *allocators);
121+
122+
/* Set internal functions of PyObject_Malloc(), PyObject_Realloc() and PyObject_Free().
123+
ctx is an arbitrary user value.
124+
125+
malloc(ctx, 0) and realloc(ctx, ptr, 0) must not return NULL: it would be
126+
treated as an error.
127+
128+
PyMem_SetupDebugHooks() should be called to reinstall debug hooks if new
129+
functions do no call original functions anymore. */
130+
PyAPI_FUNC(void) PyObject_SetAllocators(PyMemAllocators *allocators);
131+
132+
/* Get internal functions allocating and deallocating arenas for
133+
PyObject_Malloc(), PyObject_Realloc() and PyObject_Free().
134+
*ctx_p is an arbitrary user value. */
135+
PyAPI_FUNC(void) _PyObject_GetArenaAllocators(
136+
void **ctx_p,
137+
void* (**malloc_p) (void *ctx, size_t size),
138+
void (**free_p) (void *ctx, void *ptr, size_t size)
139+
);
140+
141+
/* Get internal functions allocating and deallocating arenas for
142+
PyObject_Malloc(), PyObject_Realloc() and PyObject_Free().
143+
ctx is an arbitrary user value. */
144+
PyAPI_FUNC(void) _PyObject_SetArenaAllocators(
145+
void *ctx,
146+
void* (*malloc) (void *ctx, size_t size),
147+
void (*free) (void *ctx, void *ptr, size_t size)
148+
);
144149

145150
/*
146151
* Generic object allocator interface

‎Include/pymem.h‎

Lines changed: 71 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,40 @@
1111
extern "C" {
1212
#endif
1313

14+
typedef struct {
15+
/* user context passed as the first argument to the 3 functions */
16+
void *ctx;
17+
18+
/* allocate memory */
19+
void* (*malloc) (void *ctx, size_t size);
20+
21+
/* allocate memory or resize a memory buffer */
22+
void* (*realloc) (void *ctx, void *ptr, size_t new_size);
23+
24+
/* release memory */
25+
void (*free) (void *ctx, void *ptr);
26+
} PyMemAllocators;
27+
28+
/* Raw memory allocators, system functions: malloc(), realloc(), free().
29+
30+
These functions are thread-safe, the GIL does not need to be held. */
31+
32+
/* Get internal functions of PyMem_RawMalloc(), PyMem_RawRealloc() and
33+
PyMem_RawFree(). *ctx_p is an arbitrary user value. */
34+
PyAPI_FUNC(void) PyMem_GetRawAllocators(PyMemAllocators *allocators);
35+
36+
/* Set internal functions of PyMem_RawMalloc(), PyMem_RawRealloc() and
37+
PyMem_RawFree(). ctx is an arbitrary user value.
38+
39+
PyMem_SetupDebugHooks() should be called to reinstall debug hooks if new
40+
functions do no call original functions anymore. */
41+
PyAPI_FUNC(void) PyMem_SetRawAllocators(PyMemAllocators *allocators);
42+
43+
PyAPI_FUNC(void *) PyMem_RawMalloc(size_t size);
44+
PyAPI_FUNC(void *) PyMem_RawRealloc(void *ptr, size_t new_size);
45+
PyAPI_FUNC(void) PyMem_RawFree(void *ptr);
46+
47+
1448
/* BEWARE:
1549
1650
Each interface exports both functions and macros. Extension modules should
@@ -49,35 +83,21 @@ extern "C" {
4983
performed on failure (no exception is set, no warning is printed, etc).
5084
*/
5185

52-
PyAPI_FUNC(void *) PyMem_Malloc(size_t);
53-
PyAPI_FUNC(void *) PyMem_Realloc(void *, size_t);
54-
PyAPI_FUNC(void) PyMem_Free(void *);
55-
56-
/* Starting from Python 1.6, the wrappers Py_{Malloc,Realloc,Free} are
57-
no longer supported. They used to call PyErr_NoMemory() on failure. */
86+
PyAPI_FUNC(void *) PyMem_Malloc(size_t size);
87+
PyAPI_FUNC(void *) PyMem_Realloc(void *ptr, size_t new_size);
88+
PyAPI_FUNC(void) PyMem_Free(void *ptr);
5889

5990
/* Macros. */
60-
#ifdef PYMALLOC_DEBUG
61-
/* Redirect all memory operations to Python's debugging allocator. */
62-
#define PyMem_MALLOC _PyMem_DebugMalloc
63-
#define PyMem_REALLOC _PyMem_DebugRealloc
64-
#define PyMem_FREE _PyMem_DebugFree
65-
66-
#else /* ! PYMALLOC_DEBUG */
6791

6892
/* PyMem_MALLOC(0) means malloc(1). Some systems would return NULL
6993
for malloc(0), which would be treated as an error. Some platforms
7094
would return a pointer with no memory behind it, which would break
7195
pymalloc. To solve these problems, allocate an extra byte. */
7296
/* Returns NULL to indicate error if a negative size or size larger than
7397
Py_ssize_t can represent is supplied. Helps prevents security holes. */
74-
#define PyMem_MALLOC(n) ((size_t)(n) > (size_t)PY_SSIZE_T_MAX ? NULL \
75-
: malloc((n) ? (n) : 1))
76-
#define PyMem_REALLOC(p, n) ((size_t)(n) > (size_t)PY_SSIZE_T_MAX ? NULL \
77-
: realloc((p), (n) ? (n) : 1))
78-
#define PyMem_FREE free
79-
80-
#endif /* PYMALLOC_DEBUG */
98+
#define PyMem_MALLOC(n) PyMem_Malloc(n)
99+
#define PyMem_REALLOC(p, n) PyMem_Realloc(p, n)
100+
#define PyMem_FREE(p) PyMem_Free(p)
81101

82102
/*
83103
* Type-oriented memory interface
@@ -115,6 +135,37 @@ PyAPI_FUNC(void) PyMem_Free(void *);
115135
#define PyMem_Del PyMem_Free
116136
#define PyMem_DEL PyMem_FREE
117137

138+
/* Get internal functions of PyMem_Malloc(), PyMem_Realloc()
139+
and PyMem_Free() */
140+
PyAPI_FUNC(void) PyMem_GetAllocators(PyMemAllocators *allocators);
141+
142+
/* Set internal functions of PyMem_Malloc(), PyMem_Realloc() and PyMem_Free().
143+
144+
malloc(ctx, 0) and realloc(ctx, ptr, 0) must not return NULL: it would be
145+
treated as an error.
146+
147+
PyMem_SetupDebugHooks() should be called to reinstall debug hooks if new
148+
functions do no call original functions anymore. */
149+
PyAPI_FUNC(void) PyMem_SetAllocators(PyMemAllocators *allocators);
150+
151+
/* Setup hooks to detect bugs in the following Python memory allocator
152+
functions:
153+
154+
- PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree()
155+
- PyMem_Malloc(), PyMem_Realloc(), PyMem_Free()
156+
- PyObject_Malloc(), PyObject_Realloc() and PyObject_Free()
157+
158+
Newly allocated memory is filled with the byte 0xCB, freed memory is filled
159+
with the byte 0xDB. Additionnal checks:
160+
161+
- detect API violations, ex: PyObject_Free() called on a buffer allocated
162+
by PyMem_Malloc()
163+
- detect write before the start of the buffer (buffer underflow)
164+
- detect write after the end of the buffer (buffer overflow)
165+
166+
The function does nothing if Python is not compiled is debug mode. */
167+
PyAPI_FUNC(void) PyMem_SetupDebugHooks(void);
168+
118169
#ifdef __cplusplus
119170
}
120171
#endif

0 commit comments

Comments
 (0)