Skip to content

Commit 7b370b7

Browse files
encukouencukou
authored
pythongh-93274: Make vectorcall safe on mutable classes & inherit it by default (python#95437)
1 parent a613fed commit 7b370b7

8 files changed

Lines changed: 351 additions & 21 deletions

File tree

‎Doc/c-api/call.rst‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,15 @@ This bears repeating:
5757
A class supporting vectorcall **must** also implement
5858
:c:member:`~PyTypeObject.tp_call` with the same semantics.
5959

60+
.. versionchanged:: 3.12
61+
62+
The :const:`Py_TPFLAGS_HAVE_VECTORCALL` flag is now removed from a class
63+
when the class's :py:meth:`~object.__call__` method is reassigned.
64+
(This internally sets :c:member:`~PyTypeObject.tp_call` only, and thus
65+
may make it behave differently than the vectorcall function.)
66+
In earlier Python versions, vectorcall should only be used with
67+
:const:`immutable <Py_TPFLAGS_IMMUTABLETYPE>` or static types.
68+
6069
A class should not implement vectorcall if that would be slower
6170
than *tp_call*. For example, if the callee needs to convert
6271
the arguments to an args tuple and kwargs dict anyway, then there is no point

‎Doc/c-api/typeobj.rst‎

Lines changed: 20 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -720,29 +720,29 @@ and :c:type:`PyType_Type` effectively act as defaults.)
720720
with the *vectorcallfunc* function.
721721
This can be done by setting *tp_call* to :c:func:`PyVectorcall_Call`.
722722

723-
.. warning::
724-
725-
It is not recommended for :ref:`mutable heap types <heap-types>` to implement
726-
the vectorcall protocol.
727-
When a user sets :attr:`__call__` in Python code, only *tp_call* is updated,
728-
likely making it inconsistent with the vectorcall function.
729-
730723
.. versionchanged:: 3.8
731724

732725
Before version 3.8, this slot was named ``tp_print``.
733726
In Python 2.x, it was used for printing to a file.
734727
In Python 3.0 to 3.7, it was unused.
735728

729+
.. versionchanged:: 3.12
730+
731+
Before version 3.12, it was not recommended for
732+
:ref:`mutable heap types <heap-types>` to implement the vectorcall
733+
protocol.
734+
When a user sets :attr:`~type.__call__` in Python code, only *tp_call* is
735+
updated, likely making it inconsistent with the vectorcall function.
736+
Since 3.12, setting ``__call__`` will disable vectorcall optimization
737+
by clearing the :const:`Py_TPFLAGS_HAVE_VECTORCALL` flag.
738+
736739
**Inheritance:**
737740

738741
This field is always inherited.
739742
However, the :const:`Py_TPFLAGS_HAVE_VECTORCALL` flag is not
740-
always inherited. If it's not, then the subclass won't use
743+
always inherited. If it's not set, then the subclass won't use
741744
:ref:`vectorcall <vectorcall>`, except when
742745
:c:func:`PyVectorcall_Call` is explicitly called.
743-
This is in particular the case for types without the
744-
:const:`Py_TPFLAGS_IMMUTABLETYPE` flag set (including subclasses defined in
745-
Python).
746746

747747

748748
.. c:member:: getattrfunc PyTypeObject.tp_getattr
@@ -1178,12 +1178,18 @@ and :c:type:`PyType_Type` effectively act as defaults.)
11781178

11791179
**Inheritance:**
11801180

1181-
This bit is inherited for types with the
1182-
:const:`Py_TPFLAGS_IMMUTABLETYPE` flag set, if
1183-
:c:member:`~PyTypeObject.tp_call` is also inherited.
1181+
This bit is inherited if :c:member:`~PyTypeObject.tp_call` is also
1182+
inherited.
11841183

11851184
.. versionadded:: 3.9
11861185

1186+
.. versionchanged:: 3.12
1187+
1188+
This flag is now removed from a class when the class's
1189+
:py:meth:`~object.__call__` method is reassigned.
1190+
1191+
This flag can now be inherited by mutable classes.
1192+
11871193
.. data:: Py_TPFLAGS_IMMUTABLETYPE
11881194

11891195
This bit is set for type objects that are immutable: type attributes cannot be set nor deleted.

‎Doc/whatsnew/3.12.rst‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -414,6 +414,15 @@ New Features
414414
an additional metaclass argument.
415415
(Contributed by Wenzel Jakob in :gh:`93012`.)
416416

417+
* (XXX: this should be combined with :gh:`93274` when that is done)
418+
The :const:`Py_TPFLAGS_HAVE_VECTORCALL` flag is now removed from a class
419+
when the class's :py:meth:`~object.__call__` method is reassigned.
420+
This makes vectorcall safe to use with mutable types (i.e. heap types
421+
without the :const:`immutable <Py_TPFLAGS_IMMUTABLETYPE>` flag).
422+
Mutable types that do not override :c:member:`~PyTypeObject.tp_call` now
423+
inherit the :const:`Py_TPFLAGS_HAVE_VECTORCALL` flag.
424+
(Contributed by Petr Viktorin in :gh:`93012`.)
425+
417426
Porting to Python 3.12
418427
----------------------
419428

‎Lib/test/test_call.py‎

Lines changed: 63 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -606,9 +606,19 @@ def test_vectorcall_flag(self):
606606
self.assertFalse(_testcapi.MethodDescriptorNopGet.__flags__ & Py_TPFLAGS_HAVE_VECTORCALL)
607607
self.assertTrue(_testcapi.MethodDescriptor2.__flags__ & Py_TPFLAGS_HAVE_VECTORCALL)
608608

609-
# Mutable heap types should not inherit Py_TPFLAGS_HAVE_VECTORCALL
609+
# Mutable heap types should inherit Py_TPFLAGS_HAVE_VECTORCALL,
610+
# but should lose it when __call__ is overridden
610611
class MethodDescriptorHeap(_testcapi.MethodDescriptorBase):
611612
pass
613+
self.assertTrue(MethodDescriptorHeap.__flags__ & Py_TPFLAGS_HAVE_VECTORCALL)
614+
MethodDescriptorHeap.__call__ = print
615+
self.assertFalse(MethodDescriptorHeap.__flags__ & Py_TPFLAGS_HAVE_VECTORCALL)
616+
617+
# Mutable heap types should not inherit Py_TPFLAGS_HAVE_VECTORCALL if
618+
# they define __call__ directly
619+
class MethodDescriptorHeap(_testcapi.MethodDescriptorBase):
620+
def __call__(self):
621+
pass
612622
self.assertFalse(MethodDescriptorHeap.__flags__ & Py_TPFLAGS_HAVE_VECTORCALL)
613623

614624
def test_vectorcall_override(self):
@@ -621,6 +631,58 @@ def test_vectorcall_override(self):
621631
f = _testcapi.MethodDescriptorNopGet()
622632
self.assertIs(f(*args), args)
623633

634+
def test_vectorcall_override_on_mutable_class(self):
635+
"""Setting __call__ should disable vectorcall"""
636+
TestType = _testcapi.make_vectorcall_class()
637+
instance = TestType()
638+
self.assertEqual(instance(), "tp_call")
639+
instance.set_vectorcall(TestType)
640+
self.assertEqual(instance(), "vectorcall") # assume vectorcall is used
641+
TestType.__call__ = lambda self: "custom"
642+
self.assertEqual(instance(), "custom")
643+
644+
def test_vectorcall_override_with_subclass(self):
645+
"""Setting __call__ on a superclass should disable vectorcall"""
646+
SuperType = _testcapi.make_vectorcall_class()
647+
class DerivedType(SuperType):
648+
pass
649+
650+
instance = DerivedType()
651+
652+
# Derived types with its own vectorcall should be unaffected
653+
UnaffectedType1 = _testcapi.make_vectorcall_class(DerivedType)
654+
UnaffectedType2 = _testcapi.make_vectorcall_class(SuperType)
655+
656+
# Aside: Quickly check that the C helper actually made derived types
657+
self.assertTrue(issubclass(UnaffectedType1, DerivedType))
658+
self.assertTrue(issubclass(UnaffectedType2, SuperType))
659+
660+
# Initial state: tp_call
661+
self.assertEqual(instance(), "tp_call")
662+
self.assertEqual(_testcapi.has_vectorcall_flag(SuperType), True)
663+
self.assertEqual(_testcapi.has_vectorcall_flag(DerivedType), True)
664+
self.assertEqual(_testcapi.has_vectorcall_flag(UnaffectedType1), True)
665+
self.assertEqual(_testcapi.has_vectorcall_flag(UnaffectedType2), True)
666+
667+
# Setting the vectorcall function
668+
instance.set_vectorcall(SuperType)
669+
670+
self.assertEqual(instance(), "vectorcall")
671+
self.assertEqual(_testcapi.has_vectorcall_flag(SuperType), True)
672+
self.assertEqual(_testcapi.has_vectorcall_flag(DerivedType), True)
673+
self.assertEqual(_testcapi.has_vectorcall_flag(UnaffectedType1), True)
674+
self.assertEqual(_testcapi.has_vectorcall_flag(UnaffectedType2), True)
675+
676+
# Setting __call__ should remove vectorcall from all subclasses
677+
SuperType.__call__ = lambda self: "custom"
678+
679+
self.assertEqual(instance(), "custom")
680+
self.assertEqual(_testcapi.has_vectorcall_flag(SuperType), False)
681+
self.assertEqual(_testcapi.has_vectorcall_flag(DerivedType), False)
682+
self.assertEqual(_testcapi.has_vectorcall_flag(UnaffectedType1), True)
683+
self.assertEqual(_testcapi.has_vectorcall_flag(UnaffectedType2), True)
684+
685+
624686
def test_vectorcall(self):
625687
# Test a bunch of different ways to call objects:
626688
# 1. vectorcall using PyVectorcall_Call()

‎Misc/NEWS.d/next/C API/2022-07-29-15-24-45.gh-issue-93012.-DdGEy.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
The :const:`Py_TPFLAGS_HAVE_VECTORCALL` flag is now removed from a class
2+
when the class's :py:meth:`~object.__call__` method is reassigned. This
3+
makes vectorcall safe to use with mutable types (i.e. heap types without the
4+
:const:`immutable <Py_TPFLAGS_IMMUTABLETYPE>` flag). Mutable types that do
5+
not override :c:member:`~PyTypeObject.tp_call` now inherit the
6+
:const:`Py_TPFLAGS_HAVE_VECTORCALL` flag.

‎Modules/_testcapi/clinic/vectorcall.c.h‎

Lines changed: 107 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)