@@ -84,15 +84,65 @@ the C library allocator as shown in the previous example, the allocated memory
8484for the I/O buffer escapes completely the Python memory manager.
8585
8686
87+ Raw Memory Interface
88+ ====================
89+
90+ The following function sets are wrappers to the system allocator. These
91+ functions are thread-safe, the :term: `GIL <global interpreter lock> ` does not
92+ need to be held.
93+
94+ The default raw memory block allocator uses the following functions:
95+ :c:func: `malloc `, :c:func: `realloc ` and :c:func: `free `; call ``malloc(1) `` when
96+ requesting zero bytes.
97+
98+ .. versionadded :: 3.4
99+
100+ .. c :function :: void * PyMem_RawMalloc (size_t n)
101+
102+ Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
103+ allocated memory, or *NULL* if the request fails. Requesting zero bytes
104+ returns a distinct non-*NULL* pointer if possible, as if
105+ ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have
106+ been initialized in any way.
107+
108+
109+ .. c:function:: void* PyMem_RawRealloc(void *p, size_t n)
110+
111+ Resizes the memory block pointed to by *p * to *n * bytes. The contents will
112+ be unchanged to the minimum of the old and the new sizes. If *p * is *NULL *,
113+ the call is equivalent to ``PyMem_RawMalloc(n) ``; else if *n * is equal to
114+ zero, the memory block is resized but is not freed, and the returned pointer
115+ is non-*NULL *. Unless *p * is *NULL *, it must have been returned by a
116+ previous call to :c:func: `PyMem_RawMalloc ` or :c:func: `PyMem_RawRealloc `. If
117+ the request fails, :c:func: `PyMem_RawRealloc ` returns *NULL * and *p * remains
118+ a valid pointer to the previous memory area.
119+
120+
121+ .. c :function :: void PyMem_RawFree (void *p)
122+
123+ Frees the memory block pointed to by *p *, which must have been returned by a
124+ previous call to :c:func: `PyMem_RawMalloc ` or :c:func: `PyMem_RawRealloc `.
125+ Otherwise, or if ``PyMem_Free(p) `` has been called before, undefined
126+ behavior occurs. If *p * is *NULL *, no operation is performed.
127+
128+
87129.. _memoryinterface :
88130
89131Memory Interface
90132================
91133
92134The following function sets, modeled after the ANSI C standard, but specifying
93135behavior when requesting zero bytes, are available for allocating and releasing
94- memory from the Python heap:
136+ memory from the Python heap.
137+
138+ The default memory block allocator uses the following functions:
139+ :c:func: `malloc `, :c:func: `realloc ` and :c:func: `free `; call ``malloc(1) `` when
140+ requesting zero bytes.
141+
142+ .. warning ::
95143
144+ The :term: `GIL <global interpreter lock> ` must be held when using these
145+ functions.
96146
97147.. c :function :: void * PyMem_Malloc (size_t n)
98148
@@ -155,6 +205,125 @@ versions and is therefore deprecated in extension modules.
155205:c:func: `PyMem_NEW `, :c:func: `PyMem_RESIZE `, :c:func: `PyMem_DEL `.
156206
157207
208+ Customize Memory Allocators
209+ ===========================
210+
211+ .. versionadded :: 3.4
212+
213+ .. c :type :: PyMemAllocator
214+
215+
216+ four fields:
217+
218+ +----------------------------------------------------------+---------------------------------------+
219+ | Field | Meaning |
220+ +==========================================================+=======================================+
221+ | ``void *ctx `` | user context passed as first argument |
222+ +----------------------------------------------------------+---------------------------------------+
223+ | ``void* malloc(void *ctx, size_t size) `` | allocate a memory block |
224+ +----------------------------------------------------------+---------------------------------------+
225+ | ``void* realloc(void *ctx, void *ptr, size_t new_size) `` | allocate or resize a memory block |
226+ +----------------------------------------------------------+---------------------------------------+
227+ | ``void free(void *ctx, void *ptr) `` | free a memory block |
228+ +----------------------------------------------------------+---------------------------------------+
229+
230+ .. c :type :: PyMemAllocatorDomain
231+
232+
233+
234+ * :c:data: `PYMEM_DOMAIN_RAW `: functions :c:func: `PyMem_RawMalloc `,
235+ :c:func: `PyMem_RawRealloc ` and :c:func: `PyMem_RawFree `
236+ * :c:data: `PYMEM_DOMAIN_MEM `: functions :c:func: `PyMem_Malloc `,
237+ :c:func: `PyMem_Realloc ` and :c:func: `PyMem_Free `
238+ * :c:data: `PYMEM_DOMAIN_OBJ `: functions :c:func: `PyObject_Malloc `,
239+ :c:func: `PyObject_Realloc ` and :c:func: `PyObject_Free `
240+
241+
242+ .. c :function :: void PyMem_GetAllocator (PyMemAllocatorDomain domain, PyMemAllocator *allocator)
243+
244+ Get the memory block allocator of the specified domain.
245+
246+
247+ .. c :function :: void PyMem_SetAllocator (PyMemAllocatorDomain domain, PyMemAllocator *allocator)
248+
249+ Set the memory block allocator of the specified domain.
250+
251+ The new allocator must return a distinct non-NULL pointer when requesting
252+ zero bytes.
253+
254+ For the :c:data: `PYMEM_DOMAIN_RAW ` domain, the allocator must be
255+ thread-safe: the :term: `GIL <global interpreter lock> ` is not held when the
256+ allocator is called.
257+
258+ If the new allocator is not a hook (does not call the previous allocator),
259+ the :c:func:`PyMem_SetupDebugHooks` function must be called to reinstall the
260+ debug hooks on top on the new allocator.
261+
262+
263+ .. c:function:: void PyMem_SetupDebugHooks(void)
264+
265+ Setup hooks to detect bugs in the following Python memory allocator
266+ functions:
267+
268+ - :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`,
269+ :c:func:`PyMem_RawFree`
270+ - :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Free`
271+ - :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`,
272+ :c:func:`PyObject_Free`
273+
274+ Newly allocated memory is filled with the byte ``0xCB``, freed memory is
275+ filled with the byte ``0xDB``. Additionnal checks:
276+
277+ - detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
278+ allocated by :c:func:`PyMem_Malloc`
279+ - detect write before the start of the buffer (buffer underflow)
280+ - detect write after the end of the buffer (buffer overflow)
281+
282+ The function does nothing if Python is not compiled is debug mode.
283+
284+
285+ Customize PyObject Arena Allocator
286+ ==================================
287+
288+ Python has a *pymalloc* allocator for allocations smaller than 512 bytes. This
289+ allocator is optimized for small objects with a short lifetime. It uses memory
290+ mappings called "arenas" with a fixed size of 256 KB. It falls back to
291+ :c:func:`PyMem_Malloc` and :c:func:`PyMem_Realloc` for allocations larger than
292+ 512 bytes. *pymalloc* is the default allocator used by
293+ :c:func:`PyObject_Malloc`.
294+
295+ The default arena allocator uses the following functions:
296+
297+ * :c:func:`VirtualAlloc` and :c:func:`VirtualFree` on Windows,
298+ * :c:func:`mmap` and :c:func:`munmap` if available,
299+ * :c:func:`malloc` and :c:func:`free` otherwise.
300+
301+ .. versionadded:: 3.4
302+
303+ .. c:type:: PyObjectArenaAllocator
304+
305+ Structure used to describe an arena allocator. The structure has
306+ three fields:
307+
308+ +--------------------------------------------------+---------------------------------------+
309+ | Field | Meaning |
310+ +==================================================+=======================================+
311+ | ``void *ctx`` | user context passed as first argument |
312+ +--------------------------------------------------+---------------------------------------+
313+ | ``void* alloc(void *ctx, size_t size)`` | allocate an arena of size bytes |
314+ +--------------------------------------------------+---------------------------------------+
315+ | ``void free(void *ctx, size_t size, void *ptr)`` | free an arena |
316+ +--------------------------------------------------+---------------------------------------+
317+
318+ .. c :function :: PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)
319+
320+
321+
322+ .. c :function :: PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)
323+
324+
325+
326+
158327.. _memoryexamples :
159328
160329Examples
0 commit comments