Skip to content
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
0ebdc69
Initial implementation for glibc
ZeroIntensity Dec 21, 2024
0ccf9fb
Make it cross-platform buildable.
ZeroIntensity Dec 21, 2024
52945e5
Call it on Windows.
ZeroIntensity Dec 21, 2024
9bdd802
Add a whatsnew entry
ZeroIntensity Dec 21, 2024
66f2641
Add dump_c_stack
ZeroIntensity Dec 21, 2024
0591013
Reverse the loop.
ZeroIntensity Dec 21, 2024
64707a2
Add documentation entries.
ZeroIntensity Dec 21, 2024
8a5b784
Make the whatsnew entry more specific.
ZeroIntensity Dec 21, 2024
4f09358
Add tests.
ZeroIntensity Dec 21, 2024
e1aa619
Add blurb from whatsnew.
ZeroIntensity Dec 21, 2024
6b515d3
Fix failing Sphinx build.
ZeroIntensity Dec 21, 2024
c69ee9d
Fix PyArg_ParseTupleAndKeywords format string.
ZeroIntensity Dec 21, 2024
f08b1dd
Fix Sphinx warnings.
ZeroIntensity Dec 21, 2024
dbb6d25
Add ignore to test_inspect
ZeroIntensity Dec 21, 2024
faf1a3e
Fix name.
ZeroIntensity Dec 21, 2024
ec832aa
Ignore the reentrant variable in the C analyzer.
ZeroIntensity Dec 21, 2024
524f167
Apply suggestions from code review
ZeroIntensity Dec 21, 2024
dd08bcb
Use manpage references.
ZeroIntensity Dec 21, 2024
bda3dcd
Add issue number to whatsnew.
ZeroIntensity Dec 21, 2024
a3564a5
Merge branch 'c-faulthandler' of https://github.com/ZeroIntensity/cpy…
ZeroIntensity Dec 21, 2024
f3fcea1
Rename macros.
ZeroIntensity Dec 21, 2024
db97dd2
Reduce number of printed calls.
ZeroIntensity Dec 21, 2024
c17457f
Apply suggestions from code review
ZeroIntensity Dec 21, 2024
d5f7d4b
Address code review.
ZeroIntensity Dec 21, 2024
e79e661
Merge branch 'c-faulthandler' of https://github.com/ZeroIntensity/cpy…
ZeroIntensity Dec 21, 2024
0c84f8a
Explicitly check for backtrace() and backtrace_symbols()
ZeroIntensity Dec 22, 2024
8198997
Handle no frames returned.
ZeroIntensity Dec 22, 2024
0f670f0
Move the function that calls backtrace() to within the core.
ZeroIntensity Dec 25, 2024
079f186
Merge into main.
ZeroIntensity Jan 26, 2025
892a085
Fix test case.
ZeroIntensity Jan 26, 2025
896abd1
Run make regen-configure
ZeroIntensity Jan 26, 2025
cab079e
Extract behavior in own function.
ZeroIntensity Jan 27, 2025
1784071
PEP-7
ZeroIntensity Jan 27, 2025
3304e2a
Update pycore_traceback.h
ZeroIntensity Feb 4, 2025
aa97f24
Use address_expr
ZeroIntensity Feb 5, 2025
50b4964
Fix wrong argument parsing keycode.
ZeroIntensity Feb 5, 2025
533b1db
Add tests for dump_c_stack()
ZeroIntensity Feb 5, 2025
58b3580
Start homemade implementation of backtrace_symbols()
ZeroIntensity Feb 26, 2025
f62dac8
Merge branch 'main' into c-faulthandler
ZeroIntensity Mar 31, 2025
4feaf09
Fix merge conflicts.
ZeroIntensity Apr 13, 2025
c95369f
Merge branch 'c-faulthandler' of https://github.com/ZeroIntensity/cpy…
ZeroIntensity Apr 13, 2025
95833e7
Untested implementation.
ZeroIntensity Apr 13, 2025
3e2701d
Sorta works.
ZeroIntensity Apr 13, 2025
56127fa
NULL-initialize the arrays.
ZeroIntensity Apr 13, 2025
b70bd43
Use dprintf()
ZeroIntensity Apr 13, 2025
7a070ab
Add proper configure guards.
ZeroIntensity Apr 13, 2025
e9c3d7c
Use faulthandler formatting.
ZeroIntensity Apr 13, 2025
195a539
Protect against compilers without VLAs.
ZeroIntensity Apr 13, 2025
9dd6c3b
Fix incorrect formatting when no symbol name is available.
ZeroIntensity Apr 13, 2025
ce9c39f
Don't use GNU extension.
ZeroIntensity Apr 13, 2025
c344ad7
Handle NULL.
ZeroIntensity Apr 13, 2025
e899792
Fix newlines in the whatsnew.
ZeroIntensity Apr 14, 2025
b136f71
Fix stray newline change.
ZeroIntensity Apr 14, 2025
2c381b9
Add configure guard for dladdr1()
ZeroIntensity Apr 14, 2025
52c0748
Add documentation about compatibility.
ZeroIntensity Apr 14, 2025
bd47026
Fix sphinx role.
ZeroIntensity Apr 14, 2025
07a20d0
Fix styling.
ZeroIntensity Apr 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion Doc/library/faulthandler.rst
Original file line number Diff line number Diff line change
Expand Up @@ -66,10 +66,23 @@ Dumping the traceback
Added support for passing file descriptor to this function.


Dumping the C stack
-------------------

.. versionadded:: next

.. function:: dump_c_stack(file=sys.stderr)
Comment thread
ZeroIntensity marked this conversation as resolved.

Dump the C stack trace of the current thread into *file*.

If the system does not support the C-level :manpage:`backtrace(3)`
or :manpage:`backtrace_symbols(3)` functions, then an error message
is displayed instead of the C stack.

Fault handler state
-------------------

.. function:: enable(file=sys.stderr, all_threads=True)
.. function:: enable(file=sys.stderr, all_threads=True, c_stack=True)

Enable the fault handler: install handlers for the :const:`~signal.SIGSEGV`,
:const:`~signal.SIGFPE`, :const:`~signal.SIGABRT`, :const:`~signal.SIGBUS`
Expand All @@ -81,6 +94,10 @@ Fault handler state
The *file* must be kept open until the fault handler is disabled: see
:ref:`issue with file descriptors <faulthandler-fd>`.

If *c_stack* is ``True``, then the C stack trace is printed after the Python
traceback, unless the system does not support it. See :func:`dump_c_stack` for
more information on compatibility.

.. versionchanged:: 3.5
Added support for passing file descriptor to this function.

Expand All @@ -95,6 +112,9 @@ Fault handler state
Only the current thread is dumped if the :term:`GIL` is disabled to
prevent the risk of data races.

.. versionchanged:: next
The dump now displays the C stack trace if *c_stack* is true.

.. function:: disable()

Disable the fault handler: uninstall the signal handlers installed by
Expand Down
8 changes: 8 additions & 0 deletions Doc/whatsnew/3.14.rst
Original file line number Diff line number Diff line change
Expand Up @@ -399,6 +399,14 @@ errno
(Contributed by James Roy in :gh:`126585`.)


faulthandler
------------

* Add support for printing the C stack trace on systems that support it via
Comment thread
ZeroIntensity marked this conversation as resolved.
Outdated
:func:`faulthandler.dump_c_stack` or via the *c_stack* argument in :func:`faulthandler.enable`.
Comment thread
ZeroIntensity marked this conversation as resolved.
Outdated
(Contributed by Peter Bierma in :gh:`127604`.)


fractions
---------

Expand Down
1 change: 1 addition & 0 deletions Include/internal/pycore_faulthandler.h
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ struct _faulthandler_runtime_state {
#ifdef MS_WINDOWS
void *exc_handler;
#endif
int c_stack;
} fatal_error;

struct {
Expand Down
4 changes: 4 additions & 0 deletions Include/internal/pycore_traceback.h
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,10 @@ extern int _PyTraceBack_Print(
extern int _Py_WriteIndentedMargin(int, const char*, PyObject *);
extern int _Py_WriteIndent(int, PyObject *);

// Export for the faulthandler module
PyAPI_FUNC(void)
_Py_DumpStack(int fd);
Comment thread
ZeroIntensity marked this conversation as resolved.
Outdated

#ifdef __cplusplus
}
#endif
Expand Down
11 changes: 9 additions & 2 deletions Lib/test/test_faulthandler.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,22 +93,24 @@ def check_error(self, code, lineno, fatal_error, *,
fd=None, know_current_thread=True,
py_fatal_error=False,
garbage_collecting=False,
c_stack=True,
Comment thread
ZeroIntensity marked this conversation as resolved.
function='<module>'):
"""
Check that the fault handler for fatal errors is enabled and check the
traceback from the child process output.

Raise an error if the output doesn't match the expected format.
"""
address_expr = "0x[0-9a-f]+"
all_threads_disabled = (
all_threads
and (not sys._is_gil_enabled())
)
if all_threads and not all_threads_disabled:
if know_current_thread:
header = 'Current thread 0x[0-9a-f]+'
header = f'Current thread {address_expr}'
else:
header = 'Thread 0x[0-9a-f]+'
header = f'Thread {address_expr}'
else:
header = 'Stack'
regex = [f'^{fatal_error}']
Expand All @@ -118,12 +120,17 @@ def check_error(self, code, lineno, fatal_error, *,
if all_threads_disabled and not py_fatal_error:
regex.append("<Cannot show all threads while the GIL is disabled>")
regex.append(fr'{header} \(most recent call first\):')
if garbage_collecting:
regex.append(' Garbage-collecting')
if support.Py_GIL_DISABLED and py_fatal_error and not know_current_thread:
regex.append(" <tstate is freed>")
else:
if garbage_collecting and not all_threads_disabled:
regex.append(' Garbage-collecting')
regex.append(fr' File "<string>", line {lineno} in {function}')
if c_stack:
regex.append("Current thread's C stack (most recent call first):")
regex.append(r" (\/.+\(\+.+\) \[0x[0-9a-f]+\])|(<.+>)")
Comment thread
ZeroIntensity marked this conversation as resolved.
Outdated
regex = '\n'.join(regex)

if other_regex:
Expand Down
2 changes: 1 addition & 1 deletion Lib/test/test_inspect/test_inspect.py
Original file line number Diff line number Diff line change
Expand Up @@ -5723,7 +5723,7 @@ def test_errno_module_has_signatures(self):

def test_faulthandler_module_has_signatures(self):
import faulthandler
unsupported_signature = {'dump_traceback', 'dump_traceback_later', 'enable'}
unsupported_signature = {'dump_traceback', 'dump_traceback_later', 'enable', 'dump_c_stack'}
unsupported_signature |= {name for name in ['register']
if hasattr(faulthandler, name)}
self._test_module_has_signatures(faulthandler, unsupported_signature=unsupported_signature)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Add support for printing the C stack trace on systems that support it via
:func:`faulthandler.dump_c_stack` or via the *c_stack* argument in
:func:`faulthandler.enable`.
58 changes: 56 additions & 2 deletions Modules/faulthandler.c
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
#ifdef HAVE_UNISTD_H
# include <unistd.h> // _exit()
#endif

#include <signal.h> // sigaction()
#include <stdlib.h> // abort()
#if defined(HAVE_PTHREAD_SIGMASK) && !defined(HAVE_BROKEN_PTHREAD_SIGMASK) && defined(HAVE_PTHREAD_H)
Expand Down Expand Up @@ -218,6 +219,25 @@ faulthandler_dump_traceback(int fd, int all_threads,
reentrant = 0;
}

static void
faulthandler_dump_c_stack(int fd)
{
static volatile int reentrant = 0;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it is better to use atomics here? AFAIU volatile have different behavior for msvc for different platforms.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be, but regardless, that should be done in a different PR.


if (reentrant) {
return;
}

reentrant = 1;
Comment thread
ZeroIntensity marked this conversation as resolved.

if (fatal_error.c_stack) {
PUTS(fd, "\n");
_Py_DumpStack(fd);
}

reentrant = 0;
}

static PyObject*
faulthandler_dump_traceback_py(PyObject *self,
PyObject *args, PyObject *kwargs)
Expand Down Expand Up @@ -264,6 +284,32 @@ faulthandler_dump_traceback_py(PyObject *self,
Py_RETURN_NONE;
}

static PyObject *
faulthandler_dump_c_stack_py(PyObject *self,
Comment thread
picnixz marked this conversation as resolved.
PyObject *args, PyObject *kwargs)
{
static char *kwlist[] = {"file", NULL};
PyObject *file = NULL;
Comment thread
ZeroIntensity marked this conversation as resolved.

if (!PyArg_ParseTupleAndKeywords(args, kwargs,
"|p:dump_c_stack", kwlist,
&file))
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PEP 7: add braces

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this got brought up before. The rest of the file avoids the braces, it's probably better to just stay consistent.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I respectfully disagree, we are adding new functions so it's better to go closer to the standard that deviate more from it

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the author of the original code, I would prefer that new code respects PEP 7 (add braces) :-)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do. cc @picnixz if you want to bicker about styling (I'm pretty sure you were the one that originally brought this up).

return NULL;
Comment thread
ZeroIntensity marked this conversation as resolved.

int fd = faulthandler_get_fileno(&file);
if (fd < 0) {
return NULL;
}

_Py_DumpStack(fd);

if (PyErr_CheckSignals()) {
return NULL;
}

Py_RETURN_NONE;
}

static void
faulthandler_disable_fatal_handler(fault_handler_t *handler)
{
Expand Down Expand Up @@ -354,6 +400,7 @@ faulthandler_fatal_error(int signum)

faulthandler_dump_traceback(fd, deduce_all_threads(),
fatal_error.interp);
faulthandler_dump_c_stack(fd);

_Py_DumpExtensionModules(fd, fatal_error.interp);

Expand Down Expand Up @@ -430,6 +477,7 @@ faulthandler_exc_handler(struct _EXCEPTION_POINTERS *exc_info)

faulthandler_dump_traceback(fd, deduce_all_threads(),
fatal_error.interp);
faulthandler_dump_c_stack(fd);

/* call the next exception handler */
return EXCEPTION_CONTINUE_SEARCH;
Expand Down Expand Up @@ -524,14 +572,15 @@ faulthandler_enable(void)
static PyObject*
faulthandler_py_enable(PyObject *self, PyObject *args, PyObject *kwargs)
{
static char *kwlist[] = {"file", "all_threads", NULL};
static char *kwlist[] = {"file", "all_threads", "c_stack", NULL};
PyObject *file = NULL;
int all_threads = 1;
int fd;
int c_stack = 1;
Comment thread
ZeroIntensity marked this conversation as resolved.
PyThreadState *tstate;

if (!PyArg_ParseTupleAndKeywords(args, kwargs,
"|Op:enable", kwlist, &file, &all_threads))
"|Opp:enable", kwlist, &file, &all_threads, &c_stack))
return NULL;

fd = faulthandler_get_fileno(&file);
Expand All @@ -547,6 +596,7 @@ faulthandler_py_enable(PyObject *self, PyObject *args, PyObject *kwargs)
fatal_error.fd = fd;
fatal_error.all_threads = all_threads;
fatal_error.interp = PyThreadState_GetInterpreter(tstate);
fatal_error.c_stack = c_stack;

if (faulthandler_enable() < 0) {
return NULL;
Expand Down Expand Up @@ -1237,6 +1287,10 @@ static PyMethodDef module_methods[] = {
PyDoc_STR("dump_traceback($module, /, file=sys.stderr, all_threads=True)\n--\n\n"
"Dump the traceback of the current thread, or of all threads "
"if all_threads is True, into file.")},
{"dump_c_stack",
_PyCFunction_CAST(faulthandler_dump_c_stack_py), METH_VARARGS|METH_KEYWORDS,
PyDoc_STR("dump_c_stack($module, /, file=sys.stderr)\n--\n\n"
"Dump the C stack of the current thread.")},
{"dump_traceback_later",
_PyCFunction_CAST(faulthandler_dump_traceback_later), METH_VARARGS|METH_KEYWORDS,
PyDoc_STR("dump_traceback_later($module, /, timeout, repeat=False, file=sys.stderr, exit=False)\n--\n\n"
Expand Down
68 changes: 68 additions & 0 deletions Python/traceback.c
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@
#ifdef HAVE_UNISTD_H
# include <unistd.h> // lseek()
#endif
#ifdef HAVE_EXECINFO_H
# include <execinfo.h> // backtrace(), backtrace_symbols()
#endif


#define OFF(x) offsetof(PyTracebackObject, x)
Expand Down Expand Up @@ -1101,3 +1104,68 @@ _Py_DumpTracebackThreads(int fd, PyInterpreterState *interp,
return NULL;
}

#define TRACEBACK_ENTRY_MAX_SIZE 256

static void
format_entry(char *entry_str, const char *the_entry, Py_ssize_t *length_ptr)
{
int length = PyOS_snprintf(entry_str, TRACEBACK_ENTRY_MAX_SIZE, " %s\n", the_entry);
if (length == TRACEBACK_ENTRY_MAX_SIZE) {
/* We exceeded the size, make it look prettier */
// Add ellipsis to last 3 characters
entry_str[TRACEBACK_ENTRY_MAX_SIZE - 5] = '.';
entry_str[TRACEBACK_ENTRY_MAX_SIZE - 4] = '.';
entry_str[TRACEBACK_ENTRY_MAX_SIZE - 3] = '.';
// Ensure trailing newline
entry_str[TRACEBACK_ENTRY_MAX_SIZE - 2] = '\n';
// Ensure that it's null-terminated
entry_str[TRACEBACK_ENTRY_MAX_SIZE - 1] = '\0';
}

*length_ptr = (Py_ssize_t)length;
Comment thread
ZeroIntensity marked this conversation as resolved.
Outdated
}

/* This is for faulthandler.
* Apparently, backtrace() doesn't play well across DLL boundaries on macOS */
#if defined(HAVE_EXECINFO_H) && defined(HAVE_BACKTRACE) && defined(HAVE_BACKTRACE_SYMBOLS)
void
_Py_DumpStack(int fd)
{
#define BACKTRACE_SIZE 32
PUTS(fd, "Current thread's C stack trace (most recent call first):\n");
void *callstack[BACKTRACE_SIZE];
int frames = backtrace(callstack, BACKTRACE_SIZE);
if (frames == 0) {
// Some systems won't return anything for the stack trace
PUTS(fd, " <system returned no stack trace>\n");
return;
}

char **strings = backtrace_symbols(callstack, BACKTRACE_SIZE);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One note (not a blocker) is that this function can be arbitrarily slow depending on the DWARF level of the binary. I don't think is a huge problem but maybe is worth documenting

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How slow are we talking? Like, a few extra milliseconds, or several seconds to minutes?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both are possible depending on how monstrous the extension modules and other shared objects that are loaded look like. Minutes will be very rare but in heavy monolithic enterprise applications is likely possible.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOL yeah the https://github.com/abseil/abseil-cpp/tree/master/absl/debugging stack trace dumper could also take a L o n g t i m e on huge C++ binaries.

if (strings == NULL) {
PUTS(fd, " <not enough memory to get stack trace>\n");
return;
}
for (int i = 0; i < frames; ++i) {
char entry_str[TRACEBACK_ENTRY_MAX_SIZE];
Py_ssize_t length;
format_entry(entry_str, strings[i], &length);
_Py_write_noraise(fd, entry_str, length);
}

if (frames == BACKTRACE_SIZE) {
PUTS(fd, " <truncated rest of calls>\n");
}

free(strings);
Comment thread
ZeroIntensity marked this conversation as resolved.
Outdated
#undef BACKTRACE_SIZE
#undef TRACEBACK_ENTRY_MAX_SIZE
}
#else
void
_Py_DumpStack(int fd)
{
PUTS(fd, "Current thread's C stack trace (most recent call first):\n");
PUTS(fd, " <cannot get C stack on this system>\n");
}
#endif
1 change: 1 addition & 0 deletions Tools/c-analyzer/cpython/ignored.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@ Python/sysmodule.c - _preinit_xoptions -
# thread-safety
# XXX need race protection?
Modules/faulthandler.c faulthandler_dump_traceback reentrant -
Modules/faulthandler.c faulthandler_dump_c_stack reentrant -
Python/pylifecycle.c _Py_FatalErrorFormat reentrant -
Python/pylifecycle.c fatal_error reentrant -

Expand Down
Loading