Environment
- Pythonnet version: 3.0.5
- Python version: 3.13
- Operating System: MacOS Sequoia 15.1.1
- .NET Runtime: 8.0.403
Details
After #2530, I took the time to test from Python some of the most common C# containers, testing in particular the methods from the standard Python abc's (Sequence, MutableSequence, Mapping, MutableMapping, ...).
Here are my results, along with a few proposed fixes. Please let me know if there are any mistakes.
Shared setup
import pythonnet
pythonnet.load('coreclr')
import clr
from System.Collections.Generic import Dictionary, List, KeyValuePair
from System.Collections.Immutable import ImmutableArray, ImmutableList, ImmutableDictionary
from System import Int32, String, Object, Nullable, Array
from System.Collections.Generic import CollectionExtensionsArray-like containers
ArrT = Array[Int32]
ImmArrT = ImmutableArray[Int32]
ListT = List[Int32]
arr = ArrT([0, 1, 2, 3, 4])
immarr = ImmutableArray.Create[Int32](0, 1, 2, 3, 4) # is this supposed to work, instead of using Create()?
lst = ListT(ArrT([0, 1, 2, 3, 4]))
immlst = ImmutableList.ToImmutableList[Int32](ArrT([0, 1, 2, 3, 4]))
What worked everywhere
__iter____reversed____contains____len__reverse()__getitem__ when the index is within boundsindex() when the element is in the container__setitem__ when the index is within bounds
What didn't work
__getitem__ when the index is negative or out of bounds
arr[-1] # OK
arr[999] # OK, raises IndexError
immarr[-1] # System.IndexOutOfRangeException: Index was outside the bounds of the array.
lst[-1] # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
immlst[-1] # System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'index')
Python expects a negative index to be interpreted as the position from the end of the container. Otherwise this breaks the implementation of pop() from abc (on top of a common assumption). This translation is only done for Array (here) for some reason.
Also, the exception when the index is out of bounds should be IndexError, otherwise this breaks the implementation of Sequence.index() among other things.
Possible fix
Add the following method to class SequenceMixin:
def __getitem__(self, key):
length = len(self)
key = key if key >= 0 else length + key
if key not in range(length):
raise IndexError('index out of range')
return self.get_Item(key)... and make sure List doesn't override it?
index() when the element is NOT in the container
arr.index(999) # OK, raises ValueError
immarr.index(999) # System.IndexOutOfRangeException: Index was outside the bounds of the array.
lst.index(999) # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
immlst.index(999) # System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'index')
This is caused by the previous point, the exceptions are thrown in Sequence.index().
__getitem__, __setitem__ and __delitem__ with slices
arr[0:3] # TypeError: array index has type slice, expected an integer
immarr[0:3] # TypeError: No method matches given arguments for ImmutableArray`1.get_Item: (<class 'slice'>)
lst[0:3] # TypeError: No method matches given arguments for List`1.get_Item: (<class 'slice'>)
immlst[0:3] # TypeError: No method matches given arguments for ImmutableList`1.get_Item: (<class 'slice'>)
This is ok, clearly not supported which is fine. Exceptions self-explanatory.
__setitem__ when the index is negative or out of bounds
arr[-1] = 9 # OK
lst[-1] = 9 # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
arr[999] = 9 # OK, raises IndexError
lst[999] = 9 # System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
OK for Array; for List, same as __getitem__: a negative index is not translated, and an out of bounds index raises the wrong exception.
Possible fix
Like for __getitem__, add the following method to class MutableSequenceMixin:
def __setitem__(self, key, value):
length = len(self)
key = key if key >= 0 else length + key
if key not in range(length):
raise IndexError('index out of range')
self.set_Item(key, value)
- methods that would result in a different
Array length
del arr[4] # SystemError: Objects/longobject.c:583: bad argument to internal function
arr.insert(0, 5) # IndexError; calls abc implementation, which is a stub!
arr.append(5) # IndexError; uses abc implementation, which falls back to insert
arr.pop() # SystemError (same as del); uses abc implementation, which calls del
arr.clear() # SystemError (same as del); uses abc implementation, which calls del
arr.extend([10, 11, 12]) # IndexError; uses abc implementation, which calls append
arr.remove(9) # SystemError (same as del); uses abc implementation, which calls del
arr += [10, 11, 12] # IndexError; uses abc implementation, which falls back to extend
This could be fine, to disallow changing an array's length, even though Array.Resize and Array.Copy could in theory allow us to have all of these working and be fairly efficient.
But the exceptions have me think that this is another kind of problem:
- the
SystemError comes from the bowels of CPython, precisely here, so I assume this is unintended behavior;
- the
IndexError is caused by MutableSequenceMixin not implementing insert(); if not supporting this is intended, it could be made more understandable by implementing it (in Array, not MutableSequenceMixin) and throwing TypeError, System.NotSupportedException or something more explanatory.
Crash output
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at Python.Runtime.BorrowedReference.DangerousGetAddress() in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/BorrowedReference.cs:line 18
at Python.Runtime.NewReference..ctor(BorrowedReference reference, Boolean canBeNull) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/NewReference.cs:line 20
at Python.Runtime.Runtime.PyTuple_SetItem(BorrowedReference pointer, IntPtr index, BorrowedReference value) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Runtime.cs:line 1482
at Python.Runtime.ClassBase.mp_ass_subscript_impl(BorrowedReference ob, BorrowedReference idx, BorrowedReference v) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Types/ClassBase.cs:line 498
[1] 37819 abort python3 delme_test_pythonnet.py
Like for Dictionary, __delitem__ on a List causes a hard crash; possibly related to #2530.
- other methods mutating
List length
lst.insert(0, 5) # IndexError; calls abc implementation, which is a stub!
lst.append(5) # IndexError; uses abc implementation, which calls insert
lst.pop() # IndexError; uses abc implementation, which calls __getitem__(-1) and del
lst.clear() # IndexError; uses abc implementation, which calls pop
lst.extend([10, 11, 12]) # IndexError; uses abc implementation, which calls append
lst.remove(3) # CRASH; uses abc implementation, which calls del
lst += [10, 11, 12] # IndexError; uses abc implementation, which falls back to extend
These should definitely work; it mostly boils down to the previous point, the missing support for negative indexes and insert() not being implemented in MutableMappingMixin.
Possible fix
Add the following method to class MutableSequenceMixin:
def insert(self, index, value):
self.Insert(index, value)And also apply some of the previous proposed fixes, on top of fixing del.
Dictionary-like containers
DictT = Dictionary[Int32, String]
DictT2 = Dictionary[String, Int32]
DictT3 = Dictionary[String, Nullable[Int32]]
d = DictT()
d[10] = "10"
d[20] = "20"
d[30] = "30"
d2 = DictT2()
d3 = DictT3()
rod = CollectionExtensions.AsReadOnly[Int32, String](DictT(d)) # ReadOnlyDictionary<int, string>
immd = ImmutableDictionary.ToImmutableDictionary[Int32, String](d) # ImmutableDictionary<int, string>
What worked everywhere
__contains____len____getitem__ when the key is in the dictionary__setitem__keysvaluesitems, although maybe it could be achieved without copyinggetclearupdate
What didn't work
This crashes with the following output; already tracked in #2530.
Crash output
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at Python.Runtime.BorrowedReference.DangerousGetAddress() in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/BorrowedReference.cs:line 18
at Python.Runtime.NewReference..ctor(BorrowedReference reference, Boolean canBeNull) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Native/NewReference.cs:line 20
at Python.Runtime.Runtime.PyTuple_SetItem(BorrowedReference pointer, IntPtr index, BorrowedReference value) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Runtime.cs:line 1482
at Python.Runtime.ClassBase.mp_ass_subscript_impl(BorrowedReference ob, BorrowedReference idx, BorrowedReference v) in /home/benedikt/.cache/uv/sdists-v6/.tmpkW03e6/pythonnet-3.0.5/src/runtime/Types/ClassBase.cs:line 498
[1] 44289 abort python3 delme_test_pythonnet.py
__iter__ returns KeyValuePairs instead of tuples
list(d) # gives a list of KeyValuePair
This contradicts the Mapping protocol which mandates that the mapping is an iterable of its keys, and breaks the popitem() implementation from collections.abc. At least the latter should be fixed.
d.popitem() # TypeError: No method matches given arguments for Dictionary`2.get_Item: (<class 'System.Collections.Generic.KeyValuePair[Int32,String]'>)
d2.popitem() # TypeError: No method matches given arguments for Dictionary`2.get_Item: (<class 'System.Collections.Generic.KeyValuePair[String,Int32]'>)
d3.popitem() # TypeError: No method matches given arguments for Dictionary`2.get_Item: (<class 'System.Collections.Generic.KeyValuePair[String,Nullable[Int32]]'>)
This is caused by the previous point, the exceptions are thrown in MutableMapping.popitem() which expects the dictionary to be an iterable of its keys.
Possible fix
Add the following to MutableMappingMixin:
def popitem(self):
try:
key = next(iter(self.Keys))
except StopIteration:
raise KeyError from None
value = self[key]
del self[key]
return key, value... on top of fixing del.
__getitem__ when the key is not in the dictionary
d[40] # KeyNotFoundException: The given key '40' was not present in the dictionary.
rod[40] # KeyNotFoundException: The given key '40' was not present in the dictionary.
immd[40] # KeyNotFoundException: The given key '40' was not present in the dictionary.
The exception is not translated to a ValueError, which might theoretically break some functions trying to catch it, but it doesn't seem as important here.
pop and setdefault for some types
d.pop(10) # OK
d2.pop("10") # TypeError: No method matches given arguments for Dictionary`2.TryGetValue: (<class 'str'>, <class 'NoneType'>)
d3.pop("10") # OK
d.setdefault(10, "10") # OK
d2.setdefault("10", 10) # TypeError: No method matches given arguments for Dictionary`2.TryGetValue: (<class 'str'>, <class 'NoneType'>)
d3.setdefault("10", 10) # OK
d3.setdefault("10") # Also OKI believe this breaks only with non-nullable value types: the exception is called from the MutableMappingMixin implementation of pop(), which calls self.TryGetValue(key, None) here.
Possible fix: call self.TryGetValue(key) instead? I believe this is supported since commit f69753c.
Environment
Details
After #2530, I took the time to test from Python some of the most common C# containers, testing in particular the methods from the standard Python abc's (
Sequence,MutableSequence,Mapping,MutableMapping, ...).Here are my results, along with a few proposed fixes. Please let me know if there are any mistakes.
Shared setup
Array-like containers
What worked everywhere
__iter____reversed____contains____len__reverse()__getitem__when the index is within boundsindex()when the element is in the container__setitem__when the index is within boundsWhat didn't work
__getitem__when the index is negative or out of boundsPython expects a negative index to be interpreted as the position from the end of the container. Otherwise this breaks the implementation of
pop()from abc (on top of a common assumption). This translation is only done for Array (here) for some reason.Also, the exception when the index is out of bounds should be
IndexError, otherwise this breaks the implementation ofSequence.index()among other things.Possible fix
Add the following method to class
SequenceMixin:... and make sure
Listdoesn't override it?index()when the element is NOT in the containerThis is caused by the previous point, the exceptions are thrown in
Sequence.index().__getitem__,__setitem__and__delitem__with slicesThis is ok, clearly not supported which is fine. Exceptions self-explanatory.
__setitem__when the index is negative or out of boundsOK for
Array; forList, same as__getitem__: a negative index is not translated, and an out of bounds index raises the wrong exception.Possible fix
Like for
__getitem__, add the following method to classMutableSequenceMixin:ArraylengthThis could be fine, to disallow changing an array's length, even though
Array.ResizeandArray.Copycould in theory allow us to have all of these working and be fairly efficient.But the exceptions have me think that this is another kind of problem:
SystemErrorcomes from the bowels of CPython, precisely here, so I assume this is unintended behavior;IndexErroris caused byMutableSequenceMixinnot implementinginsert(); if not supporting this is intended, it could be made more understandable by implementing it (inArray, notMutableSequenceMixin) and throwingTypeError,System.NotSupportedExceptionor something more explanatory.ListelementsCrash output
Like for
Dictionary,__delitem__on aListcauses a hard crash; possibly related to #2530.ListlengthThese should definitely work; it mostly boils down to the previous point, the missing support for negative indexes and
insert()not being implemented inMutableMappingMixin.Possible fix
Add the following method to class
MutableSequenceMixin:And also apply some of the previous proposed fixes, on top of fixing
del.Dictionary-like containers
What worked everywhere
__contains____len____getitem__when the key is in the dictionary__setitem__keysvaluesitems, although maybe it could be achieved without copyinggetclearupdateWhat didn't work
This crashes with the following output; already tracked in #2530.
Crash output
__iter__returnsKeyValuePairs instead of tuplesThis contradicts the Mapping protocol which mandates that the mapping is an iterable of its keys, and breaks the
popitem()implementation fromcollections.abc. At least the latter should be fixed.popitemThis is caused by the previous point, the exceptions are thrown in
MutableMapping.popitem()which expects the dictionary to be an iterable of its keys.Possible fix
Add the following to
MutableMappingMixin:... on top of fixing
del.__getitem__when the key is not in the dictionaryThe exception is not translated to a
ValueError, which might theoretically break some functions trying to catch it, but it doesn't seem as important here.popandsetdefaultfor some typesI believe this breaks only with non-nullable value types: the exception is called from the
MutableMappingMixinimplementation ofpop(), which callsself.TryGetValue(key, None)here.Possible fix: call
self.TryGetValue(key)instead? I believe this is supported since commit f69753c.