gh-155742: Check singletons consistencty in the garbage collector - #155801

This check should detect data corruption such as issue gh-155702.

The check is run on a garbage collection, since it's easy to add there: the GC is collected infrequently, so the overhead should be limited.

These checks are part of a long series of changes to detect data corruptions earlier and (if possible) in a reliable way in C extensions:

• I impemented faulthandler which registers a signal handler for fatal signals such as SIGSEGV, SIGBUS, SIGABRT (abort()) and SIGFPE (like a division by zero).

• I merged the Red Hat work done by David Malcolm to implement more sanity checks during a garbage collection: issue Try to print repr() when an C-level assert fails (in the garbage collector, beyond?) #53509 (commit a4b2bc7). visit_decref() now checks _PyObject_IsFreed(op) to detect usage of freed memory.

• I added _PyMem_IsPtrFreed() and _PyObject_IsFreed() to detect usage of uninitialized or freed memory.

• I enhanced Py_FatalError(): display the calling function name and dump the list of C extensions (exclude stdlib extensions).

• I added functions to check objects consistency and call them in debug mode:

  • _PyUnicode_CheckConsistency()
  • _PyObject_CheckConsistency()
  • _PyType_CheckConsistency()
  • _PyDict_CheckConsistency()
  • _PyThreadState_CheckConsistency()
  • _PyWideStringList_CheckConsistency()

• And other changes that I forgot.

I ran a quick benchmark:

import pyperf
import gc
import ctypes.util

runner = pyperf.Runner()

@ctypes.util.wrap_dll_function(ctypes.pythonapi)
def _Py_CheckSingletons() -> None:
    pass

runner.bench_func('gc.collect', gc.collect)
runner.bench_func('_Py_CheckSingletons', _Py_CheckSingletons)

With this change:

diff --git a/Objects/object.c b/Objects/object.c
index 0837b8e81d7..657d180a53f 100644
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -3595,6 +3595,9 @@ check_singleton_unicode(PyObject *obj, Py_ssize_t length, Py_UCS4 ch)
 //
 // Since the hash is computed lazily, don't check the hash, except for empty
 // tuple.
+
+PyAPI_FUNC(void) _Py_CheckSingletons(void);
+
 void
 _Py_CheckSingletons(void)
 {

I ran the benchmark on Fedora 44 with Python built in debug mode (./configure --cache-file=../python-config.cache --with-pydebug --with-system-expat) with gcc -Og.

Results without the change (_Py_CheckSingletons is missing, it only exists in the PR):

gc.collect: Mean +- std dev: 4.66 ms +- 0.10 ms

Results with the change:

gc.collect: Mean +- std dev: 4.83 ms +- 0.35 ms
_Py_CheckSingletons: Mean +- std dev: 21.4 us +- 1.1 us

This change makes gc.collect() around 1.03x slower: 4.66 ms => 4.83 ms (+ 17 us).