Skip to content

implement list codec #1084

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lostmsu merged 25 commits into from
Feb 18, 2021
Merged

implement list codec #1084

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
b500aa1
begin to implement list codec
koubaa Mar 10, 2020
f80ae87
add a test
koubaa Mar 10, 2020
765b6a2
improve CanDecode
koubaa Mar 11, 2020
98ce49c
improve list codec
koubaa Mar 11, 2020
b827f5a
full set of list codecs
koubaa Mar 11, 2020
224df0e
abandon rank and use three codecs
koubaa Mar 22, 2020
1600fdc
update
koubaa Mar 25, 2020
c43446e
respond to PR comments
koubaa Mar 29, 2020
32fa32d
make decoders public
koubaa Mar 31, 2020
dfc814b
Make reset public
koubaa Apr 1, 2020
f025cd1
add codec tests
koubaa Apr 1, 2020
24d78c3
respond to comments
koubaa Apr 6, 2020
ebcfa30
fix brace
koubaa Aug 30, 2020
5cc575d
use unix line endings to avoid large diff
koubaa Aug 30, 2020
17c47a7
fix compiler error
koubaa Aug 30, 2020
e8cc3d8
don't rethrow exception
koubaa Sep 13, 2020
080dbc3
cleanup
koubaa Sep 17, 2020
059ab08
fixes
koubaa Sep 18, 2020
432c09e
clean up sequence wrapper
koubaa Sep 18, 2020
3043201
respond to comments
koubaa Oct 10, 2020
57d7779
fix potential double free
koubaa Oct 20, 2020
7fb5915
fix exception
koubaa Feb 17, 2021
085c665
fix all exceptions
koubaa Feb 17, 2021
bf2d038
respond to PR feedback
koubaa Feb 18, 2021
07ee9fb
use hasattr("__iter__")
koubaa Feb 18, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
improve list codec
  • Loading branch information
@koubaa
koubaa committed Feb 16, 2021
commit 98ce49ce66bc3dc987f87b1c9a786dfcf0fa3f77
9 changes: 7 additions & 2 deletions src/embed_tests/Codecs.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,13 +90,16 @@ public void ListCodecTest()
var items = new List<PyObject>() { new PyInt(1), new PyInt(2), new PyInt(3) };

var x = new PyList(items.ToArray());
Assert.IsTrue(codec.CanDecode(x, typeof(List<int>)));
Assert.IsTrue(codec.CanDecode(x, typeof(IList<bool>)));
Assert.IsTrue(codec.CanDecode(x, typeof(System.Collections.IEnumerable)));
Assert.IsTrue(codec.CanDecode(x, typeof(IEnumerable<int>)));
Assert.IsTrue(codec.CanDecode(x, typeof(ICollection<float>)));
Assert.IsFalse(codec.CanDecode(x, typeof(bool)));

//we'd have to copy into a list to do this. not the best idea to support it.
//maybe there can be a flag on listcodec to allow it.
Assert.IsFalse(codec.CanDecode(x, typeof(List<int>)));
Copy link
Copy Markdown
Contributor Author

@koubaa koubaa Mar 11, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lostmsu With your idea of multiple codecs I can have a DeepCopyListCodec that takes any ienumerable and converts it by copying to a list. This is what we do right now in master and if we get rid of that implicit conversion the user can just register this codec if that causes an undesirable behavior change for them. @filmor what do you think?

Copy link
Copy Markdown
Member

@lostmsu lostmsu Mar 11, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am thinking, that Python.Runtime.dll should only contain codecs, that perform lossless conversions, perhaps even enabled by default. E.g. if a type is exactly builtins.list it is safe to convert it to IList<T> via some safe wrapper type PythonList<T>, that would contain a reference to the original object. Preferably, when this wrapper is passed back to Python it should either just pass the wrapped object, or behave exactly like a normal list would.

The codecs, that are not lossless, like converting a builtins.list to T[], and codecs, that act on relatively uncommon types like System.Uri should go to a separate extra package (created yesterday after seeing your PR): https://github.com/pythonnet/codecs

Copy link
Copy Markdown
Contributor Author

@koubaa koubaa Mar 22, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lostmsu done. I didn't go as far as to enable by default but that can be done as a followup PR. Please look closely at the test cases as I tried to capture the behavior of each of the three codecs to make sure it meets your expectation.


Action<System.Collections.IEnumerable> checkPlainEnumerable = (System.Collections.IEnumerable enumerable) =>
{
Assert.IsNotNull(enumerable);
Expand Down Expand Up @@ -145,7 +148,9 @@ raise StopIteration
Assert.IsFalse(codec.CanDecode(foo, typeof(ICollection<int>)));
Assert.IsFalse(codec.CanDecode(foo, typeof(IList<int>)));


IEnumerable<int> intEnumerable = null;
Assert.DoesNotThrow(() => { codec.TryDecode<IEnumerable<int>>(x, out intEnumerable); });
checkPlainEnumerable(intEnumerable);
}
}

Expand Down
84 changes: 54 additions & 30 deletions src/runtime/Codecs/ListCodec.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,11 @@ private CollectionRank GetRank(PyObject objectType)
return CollectionRank.Iterable;
}

private CollectionRank GetRank(Type targetType)
private Tuple<CollectionRank, Type> GetRankAndType(Type collectionType)
{
//if it is a plain IEnumerable, we can decode it using sequence protocol.
if (targetType == typeof(System.Collections.IEnumerable))
return CollectionRank.Iterable;
if (collectionType == typeof(System.Collections.IEnumerable))
return new Tuple<CollectionRank, Type>(CollectionRank.Iterable, typeof(object));

Func<Type, CollectionRank> getRankOfType = (Type type) => {
if (type.GetGenericTypeDefinition() == typeof(IList<>))
Expand All @@ -53,32 +53,25 @@ private CollectionRank GetRank(Type targetType)
return CollectionRank.None;
};

if (targetType.IsGenericType)
{
var thisRank = getRankOfType(targetType);
if (thisRank != CollectionRank.None)
return thisRank;
}

var maxRank = CollectionRank.None;
//if it implements any of the standard C# collection interfaces, we can decode it.
foreach (Type itf in targetType.GetInterfaces())
if (collectionType.IsGenericType)
{
if (!itf.IsGenericType) continue;

var thisRank = getRankOfType(itf);
//for compatibility we *could* do this and copy the value but probably not the best option.
/*if (collectionType.GetGenericTypeDefinition() == typeof(List<>))
return new Tuple<CollectionRank, Type>(CollectionRank.List, elementType);*/

//this is the most specialized type. return early
if (thisRank == CollectionRank.List) return thisRank;

//if it is more specialized, assign to max rank
if ((int)thisRank > (int)maxRank)
maxRank = thisRank;
var elementType = collectionType.GetGenericArguments()[0];
var thisRank = getRankOfType(collectionType);
if (thisRank != CollectionRank.None)
return new Tuple<CollectionRank, Type>(thisRank, elementType);
}

return maxRank;
return null;
}

private CollectionRank? GetRank(Type targetType)
{
return GetRankAndType(targetType)?.Item1;
}

public bool CanDecode(PyObject objectType, Type targetType)
{
Expand All @@ -89,7 +82,7 @@ public bool CanDecode(PyObject objectType, Type targetType)

//get the clr object rank
var clrRank = GetRank(targetType);
if (clrRank == CollectionRank.None)
if (clrRank == null || clrRank == CollectionRank.None)
return false;

//if it is a plain IEnumerable, we can decode it using sequence protocol.
Expand All @@ -99,15 +92,16 @@ public bool CanDecode(PyObject objectType, Type targetType)
return (int)pyRank >= (int)clrRank;
}

private class PyEnumerable : System.Collections.IEnumerable
private class GenericPyEnumerable<T> : IEnumerable<T>
{
PyObject iterObject;
internal PyEnumerable(PyObject pyObj)
protected PyObject iterObject;

internal GenericPyEnumerable(PyObject pyObj)
{
iterObject = new PyObject(Runtime.PyObject_GetIter(pyObj.Handle));
}

public IEnumerator GetEnumerator()
IEnumerator IEnumerable.GetEnumerator()
{
IntPtr item;
while ((item = Runtime.PyIter_Next(iterObject.Handle)) != IntPtr.Zero)
Expand All @@ -123,11 +117,32 @@ public IEnumerator GetEnumerator()
yield return obj;
}
}

public IEnumerator<T> GetEnumerator()
{
IntPtr item;
while ((item = Runtime.PyIter_Next(iterObject.Handle)) != IntPtr.Zero)
{
object obj = null;
if (!Converter.ToManaged(item, typeof(T), out obj, true))
{
Runtime.XDecref(item);
break;
}

Runtime.XDecref(item);
yield return (T)obj;
}
}
}

private object ToPlainEnumerable(PyObject pyObj)
{
return new PyEnumerable(pyObj);
return new GenericPyEnumerable<object>(pyObj);
}
private object ToEnumerable<T>(PyObject pyObj)
{
return new GenericPyEnumerable<T>(pyObj);
}

public bool TryDecode<T>(PyObject pyObj, out T value)
Expand All @@ -136,7 +151,16 @@ public bool TryDecode<T>(PyObject pyObj, out T value)
//first see if T is a plan IEnumerable
if (typeof(T) == typeof(System.Collections.IEnumerable))
{
var = ToPlainEnumerable(pyObj);
var = new GenericPyEnumerable<object>(pyObj);
}

//next use the rank to return the appropriate type
var clrRank = GetRank(typeof(T));
if (clrRank == CollectionRank.Iterable)
var = new GenericPyEnumerable<int>(pyObj);
else
{
//var = null;
}

value = (T)var;
Expand Down