Skip to content

bpo-45947: Place dict and values pointer at fixed (negative) offset just before GC header. #29879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
markshannon merged 21 commits into from
Dec 7, 2021
Merged

bpo-45947: Place dict and values pointer at fixed (negative) offset just before GC header. #29879

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
ee59772
Remove unsafe _PyObject_GC_Calloc function.
markshannon Aug 5, 2021
72b71cc
Place __dict__ immediately before GC header for variable sized object…
markshannon Aug 5, 2021
cd22dac
Restore documented behavior of tp_dictoffset.
markshannon Nov 29, 2021
4c83f77
Merge branch 'main' into regular-dict-placement
markshannon Nov 29, 2021
8a8593c
Fix up lazy dict creation logic to use managed dict pointers.
markshannon Nov 29, 2021
34b5cea
Manage values pointer, placing them directly before managed dict poin…
markshannon Nov 30, 2021
e8c74ab
Refactor a bit.
markshannon Nov 30, 2021
1bf13b0
Fix specialization of managed values.
markshannon Nov 30, 2021
a025dfb
Convert hint-based load/store attr specialization target managed dict…
markshannon Nov 30, 2021
5a012a8
Specialize LOAD_METHOD for managed dict objects.
markshannon Nov 30, 2021
e7734b8
Merge branch 'main' into regular-dict-placement
markshannon Dec 1, 2021
14d41ab
Use newer API internally.
markshannon Dec 1, 2021
123171a
Add NEWS.
markshannon Dec 1, 2021
48d6a58
Use inline functions instead of magic constants.
markshannon Dec 1, 2021
79e61bf
Remove unsafe _PyObject_GC_Malloc() function.
markshannon Dec 1, 2021
ce0f65b
Remove invalid assert.
markshannon Dec 2, 2021
98ddaed
Add comment explaning use of Py_TPFLAGS_MANAGED_DICT.
markshannon Dec 3, 2021
0f376b5
Use inline function, not magic constant.
markshannon Dec 7, 2021
d724812
Tidy up struct layout a bit.
markshannon Dec 7, 2021
9435bae
Tidy up gdb/libpython.py.
markshannon Dec 7, 2021
302f46f
Fix whitespace.
markshannon Dec 7, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove unsafe _PyObject_GC_Malloc() function.
  • Loading branch information
@markshannon
markshannon committed Dec 1, 2021
commit 79e61bf5f643fe0e8fae084ce4b09378e2f74abe
2 changes: 0 additions & 2 deletions Include/cpython/objimpl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,6 @@ PyAPI_FUNC(int) PyObject_IS_GC(PyObject *obj);
# define _PyGC_FINALIZED(o) PyObject_GC_IsFinalized(o)
#endif

PyAPI_FUNC(PyObject *) _PyObject_GC_Malloc(size_t size);
Copy link
Copy Markdown
Member

@tiran tiran Dec 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_PyObject_GC_Malloc is part of stable ABI. AFAIK the function cannot be removed.

Copy link
Copy Markdown
Member Author

@markshannon markshannon Dec 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not part of the stable ABI. It starts with an underscore.
https://www.python.org/dev/peps/pep-0384/#excluded-functions

Copy link
Copy Markdown
Member

@methane methane Dec 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Misc/stable_abi.txt define it as stable ABI.

function _PyObject_GC_Malloc
    added 3.2
    abi_only

On the other hand, no public macro in Python/C API use it. So I doubt it is actually stable abi.

As far as this repo, only one package in top4000 packages uses it.
https://github.com/hpyproject/top4000-pypi-packages/search?q=_PyObject_GC_Malloc

Copy link
Copy Markdown
Member

@gvanrossum gvanrossum Dec 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I sent an email to python-dev asking for clarification of the status of this functions and others that start with _ but are listed in Misc/stable_abi.txt. (It is not listed in Doc/data/stable_abi.dat.)



/* Test if a type supports weak references */
#define PyType_SUPPORTS_WEAKREFS(t) ((t)->tp_weaklistoffset > 0)
Expand Down
1 change: 0 additions & 1 deletion Lib/test/test_stable_abi_ctypes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -817,7 +817,6 @@ def test_available_symbols(self):
"_PyErr_BadInternalCall",
"_PyObject_CallFunction_SizeT",
"_PyObject_CallMethod_SizeT",
"_PyObject_GC_Malloc",
"_PyObject_GC_New",
"_PyObject_GC_NewVar",
"_PyObject_GC_Resize",
Expand Down
3 changes: 0 additions & 3 deletions Misc/stable_abi.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1577,9 +1577,6 @@ function _PyObject_CallFunction_SizeT
function _PyObject_CallMethod_SizeT
added 3.2
abi_only
function _PyObject_GC_Malloc
added 3.2
abi_only
function _PyObject_GC_New
added 3.2
abi_only
Expand Down
43 changes: 14 additions & 29 deletions Modules/gcmodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2254,46 +2254,30 @@ _PyObject_GC_Link(PyObject *op)
}
}



PyObject *
_PyObject_GC_Malloc(size_t basicsize)
static PyObject *
gc_alloc(size_t basicsize, size_t presize)
{
PyThreadState *tstate = _PyThreadState_GET();
GCState *gcstate = &tstate->interp->gc;
if (basicsize > PY_SSIZE_T_MAX - sizeof(PyGC_Head)) {
if (basicsize > PY_SSIZE_T_MAX - presize) {
return _PyErr_NoMemory(tstate);
}
size_t size = sizeof(PyGC_Head) + basicsize;

PyGC_Head *g = (PyGC_Head *)PyObject_Malloc(size);
if (g == NULL) {
size_t size = presize + basicsize;
char *mem = PyObject_Malloc(size);
if (mem == NULL) {
return _PyErr_NoMemory(tstate);
}
assert(((uintptr_t)g & sizeof(uintptr_t)) == 0); // g must be correctly aligned

g->_gc_next = 0;
g->_gc_prev = 0;
gcstate->generations[0].count++; /* number of allocated GC objects */
if (gcstate->generations[0].count > gcstate->generations[0].threshold &&
gcstate->enabled &&
gcstate->generations[0].threshold &&
!gcstate->collecting &&
!_PyErr_Occurred(tstate))
{
gcstate->collecting = 1;
gc_collect_generations(tstate);
gcstate->collecting = 0;
}
PyObject *op = FROM_GC(g);
((PyObject **)mem)[0] = NULL;
((PyObject **)mem)[1] = NULL;
PyObject *op = (PyObject *)(mem + presize);
_PyObject_GC_Link(op);
return op;
}


PyObject *
_PyObject_GC_New(PyTypeObject *tp)
{
PyObject *op = _PyObject_GC_Malloc(_PyObject_SIZE(tp));
size_t presize = _PyType_PreHeaderSize(tp);
PyObject *op = gc_alloc(_PyObject_SIZE(tp), presize);
if (op == NULL) {
return NULL;
}
Expand All @@ -2311,8 +2295,9 @@ _PyObject_GC_NewVar(PyTypeObject *tp, Py_ssize_t nitems)
PyErr_BadInternalCall();
return NULL;
}
size_t presize = _PyType_PreHeaderSize(tp);
size = _PyObject_VAR_SIZE(tp, nitems);
op = (PyVarObject *) _PyObject_GC_Malloc(size);
op = (PyVarObject *)gc_alloc(size, presize);
if (op == NULL) {
return NULL;
}
Expand Down
1 change: 0 additions & 1 deletion PC/python3dll.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ EXPORT_FUNC(_PyArg_VaParseTupleAndKeywords_SizeT)
EXPORT_FUNC(_PyErr_BadInternalCall)
EXPORT_FUNC(_PyObject_CallFunction_SizeT)
EXPORT_FUNC(_PyObject_CallMethod_SizeT)
EXPORT_FUNC(_PyObject_GC_Malloc)
EXPORT_FUNC(_PyObject_GC_New)
EXPORT_FUNC(_PyObject_GC_NewVar)
EXPORT_FUNC(_PyObject_GC_Resize)
Expand Down