Skip to content

Commit 796564c

Browse files
pitroupitrou
committed
Issue python#18112: PEP 442 implementation (safe object finalization).
1 parent c5d95b1 commit 796564c

File tree

25 files changed

+1256
-323
lines changed

25 files changed

+1256
-323
lines changed

‎Doc/c-api/typeobj.rst‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -465,6 +465,14 @@ type objects) *must* have the :attr:`ob_size` field.
465465
:const:`Py_TPFLAGS_HAVE_VERSION_TAG`.
466466

467467

468+
.. data:: Py_TPFLAGS_HAVE_FINALIZE
469+
470+
This bit is set when the :attr:`tp_finalize` slot is present in the
471+
type structure.
472+
473+
.. versionadded:: 3.4
474+
475+
468476
.. c:member:: char* PyTypeObject.tp_doc
469477
470478
An optional pointer to a NUL-terminated C string giving the docstring for this
@@ -968,6 +976,47 @@ type objects) *must* have the :attr:`ob_size` field.
968976
This field is not inherited; it is calculated fresh by :c:func:`PyType_Ready`.
969977

970978

979+
.. c:member:: destructor PyTypeObject.tp_finalize
980+
981+
An optional pointer to an instance finalization function. Its signature is
982+
:c:type:`destructor`::
983+
984+
void tp_finalize(PyObject *)
985+
986+
If :attr:`tp_finalize` is set, the interpreter calls it once when
987+
finalizing an instance. It is called either from the garbage
988+
collector (if the instance is part of an isolated reference cycle) or
989+
just before the object is deallocated. Either way, it is guaranteed
990+
to be called before attempting to break reference cycles, ensuring
991+
that it finds the object in a sane state.
992+
993+
:attr:`tp_finalize` should not mutate the current exception status;
994+
therefore, a recommended way to write a non-trivial finalizer is::
995+
996+
static void
997+
local_finalize(PyObject *self)
998+
{
999+
PyObject *error_type, *error_value, *error_traceback;
1000+
1001+
/* Save the current exception, if any. */
1002+
PyErr_Fetch(&error_type, &error_value, &error_traceback);
1003+
1004+
/* ... */
1005+
1006+
/* Restore the saved exception. */
1007+
PyErr_Restore(error_type, error_value, error_traceback);
1008+
}
1009+
1010+
For this field to be taken into account (even through inheritance),
1011+
you must also set the :const:`Py_TPFLAGS_HAVE_FINALIZE` flags bit.
1012+
1013+
This field is inherited by subtypes.
1014+
1015+
.. versionadded:: 3.4
1016+
1017+
.. seealso:: "Safe object finalization" (:pep:`442`)
1018+
1019+
9711020
.. c:member:: PyObject* PyTypeObject.tp_cache
9721021
9731022
Unused. Not inherited. Internal use only.

‎Doc/extending/newtypes.rst‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -157,7 +157,8 @@ to :const:`Py_TPFLAGS_DEFAULT`. ::
157157
Py_TPFLAGS_DEFAULT, /* tp_flags */
158158

159159
All types should include this constant in their flags. It enables all of the
160-
members defined by the current version of Python.
160+
members defined until at least Python 3.3. If you need further members,
161+
you will need to OR the corresponding flags.
161162

162163
We provide a doc string for the type in :attr:`tp_doc`. ::
163164

@@ -928,8 +929,9 @@ Finalization and De-allocation
928929

929930
This function is called when the reference count of the instance of your type is
930931
reduced to zero and the Python interpreter wants to reclaim it. If your type
931-
has memory to free or other clean-up to perform, put it here. The object itself
932-
needs to be freed here as well. Here is an example of this function::
932+
has memory to free or other clean-up to perform, you can put it here. The
933+
object itself needs to be freed here as well. Here is an example of this
934+
function::
933935

934936
static void
935937
newdatatype_dealloc(newdatatypeobject * obj)
@@ -981,6 +983,22 @@ done. This can be done using the :c:func:`PyErr_Fetch` and
981983
Py_TYPE(obj)->tp_free((PyObject*)self);
982984
}
983985

986+
.. note::
987+
There are limitations to what you can safely do in a deallocator function.
988+
First, if your type supports garbage collection (using :attr:`tp_traverse`
989+
and/or :attr:`tp_clear`), some of the object's members can have been
990+
cleared or finalized by the time :attr:`tp_dealloc` is called. Second, in
991+
:attr:`tp_dealloc`, your object is in an unstable state: its reference
992+
count is equal to zero. Any call to a non-trivial object or API (as in the
993+
example above) might end up calling :attr:`tp_dealloc` again, causing a
994+
double free and a crash.
995+
996+
Starting with Python 3.4, it is recommended not to put any complex
997+
finalization code in :attr:`tp_dealloc`, and instead use the new
998+
:c:member:`~PyTypeObject.tp_finalize` type method.
999+
1000+
.. seealso::
1001+
:pep:`442` explains the new finalization scheme.
9841002

9851003
.. index::
9861004
single: string; object representation

‎Doc/includes/typestruct.h‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,4 +70,11 @@ typedef struct _typeobject {
7070
PyObject *tp_subclasses;
7171
PyObject *tp_weaklist;
7272

73+
destructor tp_del;
74+
75+
/* Type attribute cache version tag. Added in version 2.6 */
76+
unsigned int tp_version_tag;
77+
78+
destructor tp_finalize;
79+
7380
} PyTypeObject;

‎Doc/library/gc.rst‎

Lines changed: 11 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -176,31 +176,24 @@ values but should not rebind them):
176176

177177
.. data:: garbage
178178

179-
A list of objects which the collector found to be unreachable but could not be
180-
freed (uncollectable objects). By default, this list contains only objects with
181-
:meth:`__del__` methods. Objects that have :meth:`__del__` methods and are
182-
part of a reference cycle cause the entire reference cycle to be uncollectable,
183-
including objects not necessarily in the cycle but reachable only from it.
184-
Python doesn't collect such cycles automatically because, in general, it isn't
185-
possible for Python to guess a safe order in which to run the :meth:`__del__`
186-
methods. If you know a safe order, you can force the issue by examining the
187-
*garbage* list, and explicitly breaking cycles due to your objects within the
188-
list. Note that these objects are kept alive even so by virtue of being in the
189-
*garbage* list, so they should be removed from *garbage* too. For example,
190-
after breaking cycles, do ``del gc.garbage[:]`` to empty the list. It's
191-
generally better to avoid the issue by not creating cycles containing objects
192-
with :meth:`__del__` methods, and *garbage* can be examined in that case to
193-
verify that no such cycles are being created.
194-
195-
If :const:`DEBUG_SAVEALL` is set, then all unreachable objects will be added
196-
to this list rather than freed.
179+
A list of objects which the collector found to be unreachable but could
180+
not be freed (uncollectable objects). Starting with Python 3.4, this
181+
list should be empty most of the time, except when using instances of
182+
C extension types with a non-NULL ``tp_del`` slot.
183+
184+
If :const:`DEBUG_SAVEALL` is set, then all unreachable objects will be
185+
added to this list rather than freed.
197186

198187
.. versionchanged:: 3.2
199188
If this list is non-empty at interpreter shutdown, a
200189
:exc:`ResourceWarning` is emitted, which is silent by default. If
201190
:const:`DEBUG_UNCOLLECTABLE` is set, in addition all uncollectable objects
202191
are printed.
203192

193+
.. versionchanged:: 3.4
194+
Following :pep:`442`, objects with a :meth:`__del__` method don't end
195+
up in :attr:`gc.garbage` anymore.
196+
204197
.. data:: callbacks
205198

206199
A list of callbacks that will be invoked by the garbage collector before and

‎Doc/library/weakref.rst‎

Lines changed: 7 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -529,22 +529,13 @@ follows::
529529
def __del__(self):
530530
self.remove()
531531

532-
This solution has a couple of serious problems:
533-
534-
* There is no guarantee that the object will be garbage collected
535-
before the program exists, so the directory might be left. This is
536-
because reference cycles containing an object with a :meth:`__del__`
537-
method can never be collected. And even if the :class:`TempDir`
538-
object is not itself part of a reference cycle, it may still be kept
539-
alive by some unkown uncollectable reference cycle.
540-
541-
* The :meth:`__del__` method may be called at shutdown after the
542-
:mod:`shutil` module has been cleaned up, in which case
543-
:attr:`shutil.rmtree` will have been replaced by :const:`None`.
544-
This will cause the :meth:`__del__` method to fail and the directory
545-
will not be removed.
546-
547-
Using finalizers we can avoid these problems::
532+
This solution has a serious problem: the :meth:`__del__` method may be
533+
called at shutdown after the :mod:`shutil` module has been cleaned up,
534+
in which case :attr:`shutil.rmtree` will have been replaced by :const:`None`.
535+
This will cause the :meth:`__del__` method to fail and the directory
536+
will not be removed.
537+
538+
Using finalizers we can avoid this problem::
548539

549540
class TempDir:
550541
def __init__(self):

‎Doc/reference/datamodel.rst‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1120,12 +1120,10 @@ Basic customization
11201120
``sys.last_traceback`` keeps the stack frame alive). The first situation
11211121
can only be remedied by explicitly breaking the cycles; the latter two
11221122
situations can be resolved by storing ``None`` in ``sys.last_traceback``.
1123-
Circular references which are garbage are detected when the option cycle
1124-
detector is enabled (it's on by default), but can only be cleaned up if
1125-
there are no Python- level :meth:`__del__` methods involved. Refer to the
1126-
documentation for the :mod:`gc` module for more information about how
1127-
:meth:`__del__` methods are handled by the cycle detector, particularly
1128-
the description of the ``garbage`` value.
1123+
Circular references which are garbage are detected and cleaned up when
1124+
the cyclic garbage collector is enabled (it's on by default). Refer to the
1125+
documentation for the :mod:`gc` module for more information about this
1126+
topic.
11291127

11301128
.. warning::
11311129

‎Doc/whatsnew/3.4.rst‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,21 @@ The :pep:`445` adds new Application Programming Interfaces (API) to customize
119119
Python memory allocators.
120120

121121

122+
.. _pep-442:
123+
124+
PEP 442: Safe object finalization
125+
=================================
126+
127+
This PEP removes the current limitations and quirks of object finalization.
128+
With it, objects with :meth:`__del__` methods, as well as generators
129+
with :keyword:`finally` clauses, can be finalized when they are part of a
130+
reference cycle.
131+
132+
.. seealso::
133+
134+
:pep:`442` - Safe object finalization
135+
PEP written and implemented by Antoine Pitrou
136+
122137

123138
Other Language Changes
124139
======================

‎Include/object.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -408,6 +408,8 @@ typedef struct _typeobject {
408408
/* Type attribute cache version tag. Added in version 2.6 */
409409
unsigned int tp_version_tag;
410410

411+
destructor tp_finalize;
412+
411413
#ifdef COUNT_ALLOCS
412414
/* these must be last and never explicitly initialized */
413415
Py_ssize_t tp_allocs;
@@ -529,6 +531,8 @@ PyAPI_FUNC(int) PyObject_Not(PyObject *);
529531
PyAPI_FUNC(int) PyCallable_Check(PyObject *);
530532

531533
PyAPI_FUNC(void) PyObject_ClearWeakRefs(PyObject *);
534+
PyAPI_FUNC(void) PyObject_CallFinalizer(PyObject *);
535+
PyAPI_FUNC(int) PyObject_CallFinalizerFromDealloc(PyObject *);
532536

533537
/* Same as PyObject_Generic{Get,Set}Attr, but passing the attributes
534538
dict as the last parameter. */
@@ -646,6 +650,12 @@ given type object has a specified feature.
646650
Py_TPFLAGS_HAVE_VERSION_TAG | \
647651
0)
648652

653+
/* NOTE: The following flags reuse lower bits (removed as part of the
654+
* Python 3.0 transition). */
655+
656+
/* Type structure has tp_finalize member (3.4) */
657+
#define Py_TPFLAGS_HAVE_FINALIZE (1UL << 0)
658+
649659
#ifdef Py_LIMITED_API
650660
#define PyType_HasFeature(t,f) ((PyType_GetFlags(t) & (f)) != 0)
651661
#else

‎Include/objimpl.h‎

Lines changed: 29 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -256,6 +256,30 @@ extern PyGC_Head *_PyGC_generation0;
256256

257257
#define _Py_AS_GC(o) ((PyGC_Head *)(o)-1)
258258

259+
/* Bit 0 is set when tp_finalize is called */
260+
#define _PyGC_REFS_MASK_FINALIZED (1 << 0)
261+
/* The (N-1) most significant bits contain the gc state / refcount */
262+
#define _PyGC_REFS_SHIFT (1)
263+
#define _PyGC_REFS_MASK (((size_t) -1) << _PyGC_REFS_SHIFT)
264+
265+
#define _PyGCHead_REFS(g) ((g)->gc.gc_refs >> _PyGC_REFS_SHIFT)
266+
#define _PyGCHead_SET_REFS(g, v) do { \
267+
(g)->gc.gc_refs = ((g)->gc.gc_refs & ~_PyGC_REFS_MASK) \
268+
| (v << _PyGC_REFS_SHIFT); \
269+
} while (0)
270+
#define _PyGCHead_DECREF(g) ((g)->gc.gc_refs -= 1 << _PyGC_REFS_SHIFT)
271+
272+
#define _PyGCHead_FINALIZED(g) (((g)->gc.gc_refs & _PyGC_REFS_MASK_FINALIZED) != 0)
273+
#define _PyGCHead_SET_FINALIZED(g, v) do { \
274+
(g)->gc.gc_refs = ((g)->gc.gc_refs & ~_PyGC_REFS_MASK_FINALIZED) \
275+
| (v != 0); \
276+
} while (0)
277+
278+
#define _PyGC_FINALIZED(o) _PyGCHead_FINALIZED(_Py_AS_GC(o))
279+
#define _PyGC_SET_FINALIZED(o, v) _PyGCHead_SET_FINALIZED(_Py_AS_GC(o), v)
280+
281+
#define _PyGC_REFS(o) _PyGCHead_REFS(_Py_AS_GC(o))
282+
259283
#define _PyGC_REFS_UNTRACKED (-2)
260284
#define _PyGC_REFS_REACHABLE (-3)
261285
#define _PyGC_REFS_TENTATIVELY_UNREACHABLE (-4)
@@ -264,9 +288,9 @@ extern PyGC_Head *_PyGC_generation0;
264288
* collector it must be safe to call the ob_traverse method. */
265289
#define _PyObject_GC_TRACK(o) do { \
266290
PyGC_Head *g = _Py_AS_GC(o); \
267-
if (g->gc.gc_refs != _PyGC_REFS_UNTRACKED) \
291+
if (_PyGCHead_REFS(g) != _PyGC_REFS_UNTRACKED) \
268292
Py_FatalError("GC object already tracked"); \
269-
g->gc.gc_refs = _PyGC_REFS_REACHABLE; \
293+
_PyGCHead_SET_REFS(g, _PyGC_REFS_REACHABLE); \
270294
g->gc.gc_next = _PyGC_generation0; \
271295
g->gc.gc_prev = _PyGC_generation0->gc.gc_prev; \
272296
g->gc.gc_prev->gc.gc_next = g; \
@@ -279,16 +303,16 @@ extern PyGC_Head *_PyGC_generation0;
279303
*/
280304
#define _PyObject_GC_UNTRACK(o) do { \
281305
PyGC_Head *g = _Py_AS_GC(o); \
282-
assert(g->gc.gc_refs != _PyGC_REFS_UNTRACKED); \
283-
g->gc.gc_refs = _PyGC_REFS_UNTRACKED; \
306+
assert(_PyGCHead_REFS(g) != _PyGC_REFS_UNTRACKED); \
307+
_PyGCHead_SET_REFS(g, _PyGC_REFS_UNTRACKED); \
284308
g->gc.gc_prev->gc.gc_next = g->gc.gc_next; \
285309
g->gc.gc_next->gc.gc_prev = g->gc.gc_prev; \
286310
g->gc.gc_next = NULL; \
287311
} while (0);
288312

289313
/* True if the object is currently tracked by the GC. */
290314
#define _PyObject_GC_IS_TRACKED(o) \
291-
((_Py_AS_GC(o))->gc.gc_refs != _PyGC_REFS_UNTRACKED)
315+
(_PyGC_REFS(o) != _PyGC_REFS_UNTRACKED)
292316

293317
/* True if the object may be tracked by the GC in the future, or already is.
294318
This can be useful to implement some optimizations. */

‎Lib/test/test_descr.py‎

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3736,18 +3736,8 @@ def __del__(self):
37363736
# bug).
37373737
del c
37383738

3739-
# If that didn't blow up, it's also interesting to see whether clearing
3740-
# the last container slot works: that will attempt to delete c again,
3741-
# which will cause c to get appended back to the container again
3742-
# "during" the del. (On non-CPython implementations, however, __del__
3743-
# is typically not called again.)
37443739
support.gc_collect()
37453740
self.assertEqual(len(C.container), 1)
3746-
del C.container[-1]
3747-
if support.check_impl_detail():
3748-
support.gc_collect()
3749-
self.assertEqual(len(C.container), 1)
3750-
self.assertEqual(C.container[-1].attr, 42)
37513741

37523742
# Make c mortal again, so that the test framework with -l doesn't report
37533743
# it as a leak.

0 commit comments

Comments
 (0)