Implicit List conversion breaks backwards compatibility + numpy support

Environment

Computer 1:

Pythonnet version: 2.3, installed via pip I think
Python version: 2.7.12
Operating System: Ubuntu 16.04.2

Computer 2:

Pythonnet version: manual build on master from commit ce14424 (currently 11 commits behind)
Python version: 2.7.12
Operating System: Ubuntu 16.04.2

Details

Description: Python calls .NET function which returns a List. Python then passes the return value, without modification, to a second .NET function which accepts a List. Computer 1 executes this code just fine. On Computer 2, there is no .NET function found that matches the arguments because the return value of the .NET function has been transformed into a Python list.

Python code:

import clr
from MyDotNetProject import PythonInterop

x = PythonInterop.GetDoubleList()
PythonInterop.PrintDoubleList(x)

.NET code:

    public class PythonInterop
    {
        public static List<Double> GetDoubleList() {
            var result = new List<Double>();
            result.Add(1);
            result.Add(2);
            result.Add(3);
            return result;
        }

        public static void PrintDoubleList(List<Double> list) {
            Console.WriteLine("You list has " + list.Count + " elements");
        }
    }

The Python code works fine on Computer 1. On Computer 2, the PrintDoubleList call produces
TypeError: No method matches given arguments

If I print type(x) in Python, Computer 1 gives me a .NET List type while Computer 2 gives me a Python list. I can print x.Count on Computer 1, but I get a missing attribute error on Computer 2.

If I build manually from the 2.3 tag, I get the same (good) behavior as on Computer 1.

It seems that some feature has been partially added to automatically convert .NET objects into Python objects when possible. I suppose this is ok (though I would prefer that not happen because I don't want the mandatory performance hit of converting even when I don't want to convert), but if that's the intention, there must be automatic conversion of Python objects to .NET objects also. One without the other is a critical bug.

Is this the issue you are experiencing?

#451

As far as I can tell, this isn't like #451 because the error in #451 occurs in .NET. The error in this issue occurs in Python. But, the same underlying problem may be causing both issues. Given the .NET code above, here is a different demonstration of the poor behavior:

import clr
from MyDotNetProject import PythonInterop
from System.Collections.Generic import List
from System import Double

x = PythonInterop.GetDoubleList()
assert isinstance(x, List[Double])

That assertion used to (Computer 1, version 2.3) succeed. Now (Computer 2, ce14424+ on master) it fails.

From #451, I see you commented, "The reason is because before the list is passed back to C#, it gets converted to Python list." It seems like this is a new (since 2.3) behavior, right? I suggest that this breaks backward compatibility and is generally undesirable. An explicit helper function to easily convert a generic .NET list to a Python list might be nice, but the conversion should not be performed automatically. But if it is, nonetheless, performed automatically, then the symmetric operation (automatically converting a Python list to a generic .NET list) MUST also happen.

Also from #451: "But this is of course a lot of conversion back and forth and maybe a performance issue." I agree, which is why I strongly suggest not performing automatic conversion.

EDIT: The reason I say a helper function to convert a .NET List to a Python list only might be nice is that the conversion is trivial:
y = list([x[i] for i in range(x.Count)])

@BenjaminPelletier

@BenjaminPelletier I agree that converting the CLR List by default is unexpected behavior and should be considered a regression. What do you think @vmuriart ?

At the same time we need to get this sample with numpy arrays working seamlessly:

https://github.com/pythonnet/pythonnet#example

This is why the conversion was introduced.

Calling Python from .NET is outside my use case so I'm not familiar with the logistics doing so. But "just like I'd expect" would be for pythonnet to expose explicit .NET types that map to Python types for the user to instantiate (in .NET) by some means before calling Python with them. These types would probably have helper functions for converting/wrapping from common corresponding .NET types.

FWIW, here's how I'm marshalling arrays between .NET and NumPy, and it seems like it would have been nice to not have to figure out & write this myself:

Python:

######################## Copy operations #################################

# [Python.NET] Efficient copy of .NET Array to ctypes or numpy array.
# https://mail.python.org/pipermail/pythondotnet/2014-May/001525.html

def copyNumpyToDotNetArray(srcnparray, destdotnetarray):
    ''' Copies the content in a numpy array of any dimensions with dtype=float to a one-dimensional .NET array
    The number of elements in srcnpfloat must not exceed the size of destdotnetdouble
    
    @srcnpfloat: numpy array (any number of dimensions) with dtype=float
    @destdotnetdouble: Pre-allocated .NET Double Array object (via Array.CreateInstance(Double, n) or similar) to copy the numpy array content into
    '''
    if len(srcnparray.shape) == 1 and not srcnparray.dtype == bool:
        ptr = IntPtr.__overloads__[long](srcnparray.__array_interface__['data'][0])
        Marshal.Copy(ptr, destdotnetarray, 0, srcnparray.size)
    else:
        if not srcnparray.flags.c_contiguous:
            srcnparray = srcnparray.copy(order='C')
        assert srcnparray.flags.c_contiguous
        PythonInterop.CopyFromPointer(IntPtr.__overloads__[long](srcnparray.__array_interface__['data'][0]), destdotnetarray)

def copyDotNetArrayToNumpy(srcdotnetarray, destnparray):
    ''' Copies the content of a one-dimensional .NET array into a numpy array of any dimensions
    The number of elements in srcdotnetdouble and destnpfloat must match
    
    @srcdotnetdouble: One-dimensional .NET Array of Doubles (via Array.CreateInstance(Double, n) or similar) containing source data
    @destnpfloat: numpy array (any number of dimensions) to copy the source data into
    
    Use dest = np.empty(desiredShape, dtype=float) to preinitialize if creating a new variable
    '''
    if len(srcdotnetarray) == 1:
        ptr = IntPtr.__overloads__[long](destnparray.__array_interface__['data'][0])
        Marshal.Copy(srcdotnetarray, 0, ptr, srcdotnetarray.Length)
    else:
        PythonInterop.CopyToPointer(srcdotnetarray, IntPtr.__overloads__[long](destnparray.__array_interface__['data'][0]))

######################## Conversions to .NET #############################

def dotnetArrayOf(nparray):
    ''' Creates and returns a .NET Array that mirrors the provided numpy array
    
    @nparray: numpy array
    
    @Returns: .NET Array with element type matching nparray.dtype and identical dimensions with content that matches the provided numpy array
    '''
    dims = nparray.shape
    n = len(dims)
    dims_dn = Array.CreateInstance(Int32, n)
    for i in range(n):
        dims_dn[i] = Int32(dims[i])
    if nparray.dtype == float:
        dn = Array.CreateInstance(Double, dims_dn)
    elif nparray.dtype == int: #NOTE: this check is probably invalid in Python 3.x
        dn = Array.CreateInstance(Int64, dims_dn)
    elif nparray.dtype == np.int32:
        dn = Array.CreateInstance(Int32, dims_dn)
    elif nparray.dtype == np.bool:
        dn = Array.CreateInstance(Boolean, dims_dn)
    else:
        raise NotImplementedError("dotnetArrayOf does not yet support dtype=" + str(nparray.dtype))
    copyNumpyToDotNetArray(nparray, dn)
    return dn

######################## Conversions from .NET ###########################
    
def numpyArrayOf(dotnetarray):
    ''' Creates and returns a numpy Array that mirrors the provided .NET array
    
    @dotnetarray: .NET Array
    
    @Returns: numpy Array with dtype matching dotnetarray's element type, and identical dimensions with content that matches the provided .NET Array
    '''
    dims = np.empty(dotnetarray.Rank, dtype=int)
    for i in range(dotnetarray.Rank):
        dims[i] = dotnetarray.GetLength(i)
    type_dn = dotnetarray.GetType().GetElementType().Name
    if type_dn == 'Double':
        dtype = float
    elif type_dn == 'Int32':
        dtype = np.int32
    elif type_dn == 'Int64':
        dtype = int #NOTE: probably invalid in Python 3.x
    elif type_dn == 'Boolean':
        dtype = np.bool
    else:
        raise NotImplementedError("numpyArrayOf does not yet support .NET Arrays with " + str(type_dn) + " elements")
    nparray = np.empty(dims, dtype=dtype, order='C')
    copyDotNetArrayToNumpy(dotnetarray, nparray)
    return nparray

.NET:

    public class PythonInterop
    {
        /// <summary>
        /// Copies data from a NumPy array to the destination .NET array
        /// </summary>
        /// <param name="pSource">Pointer to a NumPy array to copy from</param>
        /// <param name="dest">.NET array to be copied into</param>
        /// <remarks>This routine handles Boolean arrays in a special way because NumPy arrays have each element occupying 1 byte while .NET has them occupying 4 bytes</remarks>
		public static void CopyFromPointer(IntPtr pSource, Array dest)
        {
            Type elementType = dest.GetType().GetElementType();
            int sizeOfElement = Marshal.SizeOf(elementType);
            if (elementType == typeof(Boolean))
                sizeOfElement = 1;

            int byteCount = sizeOfElement;
            for (int i = 0; i < dest.Rank; i++)
            {
                byteCount *= dest.GetLength(i);
            }

            var gch = GCHandle.Alloc(dest);
            var tPtr = Marshal.UnsafeAddrOfPinnedArrayElement(dest, 0);
            MemCopy(pSource, tPtr, byteCount);
            gch.Free();
        }

        /// <summary>
        /// Copies data from a .NET array to a NumPy array
        /// </summary>
        /// <param name="source">.NET array to be copied from</param>
        /// <param name="pDest">Pointer to a NumPy array to copy into</param>
		public static void CopyToPointer(Array source, IntPtr pDest)
        {
            Type elementType = source.GetType().GetElementType();
            int sizeOfElement = Marshal.SizeOf(elementType);
            if (elementType == typeof(Boolean))
                sizeOfElement = 1;

            int byteCount = sizeOfElement;
            for (int i = 0; i < source.Rank; i++)
                byteCount *= source.GetLength(i);

            var gch = GCHandle.Alloc(source);
            var tPtr = Marshal.UnsafeAddrOfPinnedArrayElement(source, 0);
            MemCopy(tPtr, pDest, byteCount);
            gch.Free();
        }

        private static readonly int SIZEOFINT = Marshal.SizeOf(typeof(int));
        private static unsafe void MemCopy(IntPtr pSource, IntPtr pDest, int byteCount)
        {
            int count = byteCount / SIZEOFINT;
            int rest = byteCount % count;
            unchecked
            {
                int* ps = (int*)pSource.ToPointer(), pd = (int*)pDest.ToPointer();
                // Loop over the cnt in blocks of 4 bytes, 
                // copying an integer (4 bytes) at a time:
                for (int n = 0; n < count; n++)
                {
                    *pd = *ps;
                    pd++;
                    ps++;
                }
                // Complete the copy by moving any bytes that weren't moved in blocks of 4:
                if (rest > 0)
                {
                    byte* ps1 = (byte*)ps;
                    byte* pd1 = (byte*)pd;
                    for (int n = 0; n < rest; n++)
                    {
                        *pd1 = *ps1;
                        pd1++;
                        ps1++;
                    }
                }
            }
        }
    }

@BenjaminPelletier

@BenjaminPelletier if you would like to submit a helper function for numpy arrays, then pull requests are welcome! Here is another example of passing numpy arrays:

https://stackoverflow.com/questions/43910749/efficiently-convert-system-single-to-numpy-array

@denfromufa I also have a numpy array converter here
https://github.com/yagweb/pythonnetLab/blob/master/pynetLab/Numpy.cs

With this converter, the example https://github.com/pythonnet/pythonnet#example can be replaced with this one,
https://github.com/yagweb/pythonnetLab/blob/master/Test/TestMatplotlib.cs
This one may be a better an example, because,

.NET users can create a numpy array with a single line, like this,
var x = Numpy.NewArray(new double[]{ 1.0, 2.0, 3.0, 4.0, 5.0, 6.0 });
So, No CLR list converting to python list by default is needed.
Usage of PyScope and Matplotlib is also included in this example.

...
scope.Exec(
    "fig = plt.figure() \n" +
    "plt.plot(x[1:], y[1:]) \n"   //we can slice a numpy array
);
var fig = scope.Get("fig");
//fig.show(); //show the figure
plotdata = Matplotlib.SaveFigureToArray(fig, 200, "png"); //save the figure to a .NET array

@yagweb

@yagweb do you suggest that we should revert back the auto-conversion due to this regression and use your numpy converter?

which commit broke this ? Was it the one I added to fix the example?
If my commit broke existing behavior, then I'm ok w reverting that one and reopenning the other issues.

@vmuriart

@vmuriart this pull request broke C# List behavior:

https://github.com/pythonnet/pythonnet/pull/427/files

ok to revert behavior until better solution is found

@BenjaminPelletier

Hi, I wrote a Python-side Numpy <-> .NET conversion functions that use ctypes.memmove based on what @BenjaminPelletier provided above and the 2014 mailing list conversation between David Cook and Jeffrey Bush. It doesn't require any C# helper code. I think I've got support for all the types in System except strings. Performance seems adequate, I can do around 1 MB / ms, after the initial lazy import:

Numpy to .NET converted 8388608 bytes in 4.231 +/- 1.812 ms (mean: 4.0 ns/ele)
.NET to Numpy converted 8388608 bytes in 3.461 +/- 1.200 ms (mean: 3.3 ns/ele)

The ctypes.memmove approach is equally as fast as Marshal.Copy and doesn't have any issues with multi-dimensional arrays associated with it. Source code as follows, the __main__ block is all testing code:

import numpy as np
import ctypes
import clr, System
from System import Array, Int32
from System.Runtime.InteropServices import GCHandle, GCHandleType

_MAP_NP_NET = {
    np.dtype('float32'): System.Single,
    np.dtype('float64'): System.Double,
    np.dtype('int8')   : System.SByte,
    np.dtype('int16')  : System.Int16,
    np.dtype('int32')  : System.Int32,
    np.dtype('int64')  : System.Int64,
    np.dtype('uint8')  : System.Byte,
    np.dtype('uint16') : System.UInt16,
    np.dtype('uint32') : System.UInt32,
    np.dtype('uint64') : System.UInt64,
    np.dtype('bool')   : System.Boolean,
}
_MAP_NET_NP = {
    'Single' : np.dtype('float32'),
    'Double' : np.dtype('float64'),
    'SByte'  : np.dtype('int8'),
    'Int16'  : np.dtype('int16'), 
    'Int32'  : np.dtype('int32'),
    'Int64'  : np.dtype('int64'),
    'Byte'   : np.dtype('uint8'),
    'UInt16' : np.dtype('uint16'),
    'UInt32' : np.dtype('uint32'),
    'UInt64' : np.dtype('uint64'),
    'Boolean': np.dtype('bool'),
}

def asNumpyArray(netArray):
    '''
    Given a CLR `System.Array` returns a `numpy.ndarray`.  See _MAP_NET_NP for 
    the mapping of CLR types to Numpy dtypes.
    '''
    dims = np.empty(netArray.Rank, dtype=int)
    for I in range(netArray.Rank):
        dims[I] = netArray.GetLength(I)
    netType = netArray.GetType().GetElementType().Name

    try:
        npArray = np.empty(dims, order='C', dtype=_MAP_NET_NP[netType])
    except KeyError:
        raise NotImplementedError("asNumpyArray does not yet support System type {}".format(netType) )

    try: # Memmove 
        sourceHandle = GCHandle.Alloc(netArray, GCHandleType.Pinned)
        sourcePtr = sourceHandle.AddrOfPinnedObject().ToInt64()
        destPtr = npArray.__array_interface__['data'][0]
        ctypes.memmove(destPtr, sourcePtr, npArray.nbytes)
    finally:
        if sourceHandle.IsAllocated: sourceHandle.Free()
    return npArray

def asNetArray(npArray):
    '''
    Given a `numpy.ndarray` returns a CLR `System.Array`.  See _MAP_NP_NET for 
    the mapping of Numpy dtypes to CLR types.

    Note: `complex64` and `complex128` arrays are converted to `float32` 
    and `float64` arrays respectively with shape [m,n,...] -> [m,n,...,2]
    '''
    dims = npArray.shape
    dtype = npArray.dtype
    # For complex arrays, we must make a view of the array as its corresponding 
    # float type.
    if dtype == np.complex64:
        dtype = np.dtype('float32')
        dims.append(2)
        npArray = npArray.view(np.float32).reshape(dims)
    elif dtype == np.complex128:
        dtype = np.dtype('float64')
        dims.append(2)
        npArray = npArray.view(np.float64).reshape(dims)

    netDims = Array.CreateInstance(Int32, npArray.ndim)
    for I in range(npArray.ndim):
        netDims[I] = Int32(dims[I])
    
    if not npArray.flags.c_contiguous:
        npArray = npArray.copy(order='C')
    assert npArray.flags.c_contiguous

    try:
        netArray = Array.CreateInstance(_MAP_NP_NET[dtype], netDims)
    except KeyError:
        raise NotImplementedError("asNetArray does not yet support dtype {}".format(dtype))

    try: # Memmove 
        destHandle = GCHandle.Alloc(netArray, GCHandleType.Pinned)
        sourcePtr = npArray.__array_interface__['data'][0]
        destPtr = destHandle.AddrOfPinnedObject().ToInt64()
        ctypes.memmove(destPtr, sourcePtr, npArray.nbytes)
    finally:
        if destHandle.IsAllocated: destHandle.Free()
    return netArray

if __name__ == '__main__':
    from time import perf_counter
    import matplotlib.pyplot as plt
    import psutil

    tries = 1000
    foo = np.full([1024,1024], 2.5, dtype='float32')


    netMem = np.zeros(tries)
    t_asNet = np.zeros(tries)
    netFoo = asNetArray( foo ) # Lazy loading makes the first iteration very slow
    for I in range(tries):
        t0 = perf_counter()
        netFoo = asNetArray( foo )
        t_asNet[I] = perf_counter() - t0
        netMem[I] = psutil.virtual_memory().free / 2.0**20

    t_asNumpy = np.zeros(tries)
    numpyMem = np.zeros(tries)
    unNetFoo = asNumpyArray( netFoo ) # Lazy loading makes the first iteration very slow
    for I in range(tries):
        t0 = perf_counter()
        unNetFoo = asNumpyArray( netFoo )
        t_asNumpy[I] = perf_counter() - t0
        numpyMem[I] = psutil.virtual_memory().free / 2.0**20

    # Convert times to milliseconds
    t_asNet *= 1000
    t_asNumpy *= 1000
    np.testing.assert_array_almost_equal( unNetFoo, foo )
    print( "Numpy to .NET converted {} bytes in {:.3f} +/- {:.3f} ms (mean: {:.1f} ns/ele)".format( \
        foo.nbytes, t_asNet.mean(), t_asNet.std(), t_asNet.mean()/foo.size*1e6 ) )
    print( ".NET to Numpy converted {} bytes in {:.3f} +/- {:.3f} ms (mean: {:.1f} ns/ele)".format( \
        foo.nbytes, t_asNumpy.mean(), t_asNumpy.std(), t_asNumpy.mean()/foo.size*1e6 ) )

    plt.figure()
    plt.plot(np.arange(tries), netMem, '-', label='asNetArray')
    plt.plot(np.arange(tries), numpyMem, '-', label='asNumpyArray')
    plt.legend(loc='best')
    plt.ylabel('Free memory (MB)')
    plt.xlabel('Iteration')
    plt.show(block=True)

I don't see any evidence of memory-leaking.

Edit: one can do a zero-copy with np.frombuffer but then you have a mess of memory manged both by Python's garbage collector and C#'s garbage collector. If people here know how to deal with references in both GC's let me know.

@lostmsu

@lostmsu Hi there. Would you be able to provide an example of how to use Codecs to prevent the conversion from a System.Collections.Generic.List to a python list? I read both the Codec wiki page and this example but I didn't get it, sorry.
Is there a way to just set this behaviour by default anytime I import pythonnet?

Sorry, I don't do that. I think I got associated with #514 because I reported that Pythonnet 2.x had broken support for .NET Remoting, and whoever fixed it (maybe?) introduced an issue related to what you are talking about. So that was holding up the release containing the .NET Remoting fix. Since the fix for me has been released, I'm guessing your issue also got resolved. --------- Original Message --------- Subject: Re: [pythonnet/pythonnet] Implicit List conversion breaks backwards compatibility + numpy support (#514) From: "ddsleonardo" <notifications@github.com> Date: 10/30/20 5:38 am To: "pythonnet/pythonnet" <pythonnet@noreply.github.com> Cc: "jcarole" <github@jnpcarole.com>, "Manual" <manual@noreply.github.com> @lostmsu Hi there. Would you be able to provide an example of how to use Codecs to prevent the conversion from a System.Collections.Generic.List to a python list? I read both the Codec wiki page and this example but I didn't get it, sorry. Is there a way to just set this behaviour by default anytime I import pythonnet? - You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

@jcarole

I do not understand your reply @jcarole.
To make it clear, I am having the same problem as #1153 and also mentioned here. In both instances there are mentions either of RawProxyEncoder or the Codecs but I could not understand the usage examples. The only working solution I have found is this comment previously made in this discussion. This comment sounded encouraging, it would be really nice if we could make the conversions optional.

@ddsleonardo

@ddsleonardo have you checked the new test named test_codec (it is at the very end of the change linked above)? What part you did not understand?

Sorry to confuse you. I mean to say that I don't know anything about the issue you're referring to and tried to explain why you might have thought I was at all involved in that topic.

…

--------- Original Message --------- Subject: Re: [pythonnet/pythonnet] Implicit List conversion breaks backwards compatibility + numpy support (#514) From: "ddsleonardo" <notifications@github.com> Date: 10/30/20 9:41 am To: "pythonnet/pythonnet" <pythonnet@noreply.github.com> Cc: "jcarole" <github@jnpcarole.com>, "Mention" <mention@noreply.github.com> I do not understand your reply @jcarole. To make it clear, I am having the same problem as #1153 and also mentioned here. In both instances there are mentions either of RawProxyEncoder or the Codecs but I could not understand the usage examples. The only working solution I have found is this comment previously made in this discussion. This comment sounded encouraging, it would be really nice if we could make the conversions optional. - You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

@lostmsu

@lostmsu Thanks for the reply. My main question was relative to at what scope should I use the Codecs, i.e. do I need to define and register an instance of the Encoder class every time I open a Python Interactive and want to manipulate some C# Lists? (which I believe is the case, correct)?

I think I managed to follow test_codec this time and at least could run the steps without fail. But then I realised that I am running my own build of pythonnet now (with the commented block), so my auto-conversion is disabled anyway.
I will revert this later and give it another go.

Still following on the same question, it sounds like this has been made a lot more convoluted than maybe it should be. I had been running a previous version of pythonnet for a while to avoid this, and only encountered the issue again as I had to reset my environment. Now I appreciate that this package isn't written just for me, but I'd be interested to know the motivations behind this, as it seems that I'm not the only one who has struggled with this and the workarounds to make it work are not trivial.

At the same time, I am happy to run my own fork of this repo without the conversion, and set up a pipeline that keeps everything else up to date with the main repo.

@ddsleonardo

@ddsleonardo don't worry, the default conversion is going to be off by default in the next major version.

I was not here around the time Python.NET started, but I suspect this conversion was added to make most APIs easily callable: List<T> does not implement Python collection protocol, and Python's list does not implement IEnumerable and other stuff.

Updated my code snippet for PythonNet 2.5.1, Python 3.9.4, and NumPy 1.20.2 for those who are looking for a working solution:

https://gist.github.com/robbmcleod/73ca42da5984e6d0e5b6ad28bc4a504e

@robbmcleod

@robbmcleod I adapted your last code so that complex arrays in python convert back-and-forth with SystemArray in .NET whose element type are directly System.Numerics.Complex (instead of adding extra dimension):

https://gist.github.com/CitizenInsane/c8d3ddc5b14faceec433af3e940e22a8

@CitizenInsane

@CitizenInsane thanks although in my case I actually do need the 64-bit complex as that's what the counterparty software that I'm talking to uses.

@robbmcleod

@robbmcleod Yes it all depends. The .NET API i try to connect with is using System.Numerics.Complex arrays directly. Thanks for the great job, first time I'm doing python and this to interop with .net so it helped a lot to have your converters :).

@robbmcleod

BTW, @robbmcleod , @CitizenInsane , in pythonnet 3.0 .NET arrays will implement buffer protocol!

@robbmcleod

BTW, @robbmcleod , @CitizenInsane , in pythonnet 3.0 .NET arrays will implement buffer protocol!

Good to know, thanks, that should be a big improvement over my memcpy solution.

@yagweb

@denfromufa I also have a numpy array converter here https://github.com/yagweb/pythonnetLab/blob/master/pynetLab/Numpy.cs

With this converter, the example https://github.com/pythonnet/pythonnet#example can be replaced with this one, https://github.com/yagweb/pythonnetLab/blob/master/Test/TestMatplotlib.cs This one may be a better an example, because,

.NET users can create a numpy array with a single line, like this,
var x = Numpy.NewArray(new double[]{ 1.0, 2.0, 3.0, 4.0, 5.0, 6.0 });
So, No CLR list converting to python list by default is needed.

Usage of PyScope and Matplotlib is also included in this example.
...
scope.Exec(
    "fig = plt.figure() \n" +
    "plt.plot(x[1:], y[1:]) \n"   //we can slice a numpy array
);
var fig = scope.Get("fig");
//fig.show(); //show the figure
plotdata = Matplotlib.SaveFigureToArray(fig, 200, "png"); //save the figure to a .NET array

@yagweb Hi there, I found that in your converter, "using(Py.GIL())" is called multiple times in a function. However, every time that I call the function on my code for the second time, the code just hangs at "using(Py.GIL())". I have to restart the whole program to get the function to work for the second time. Could you please advise how the system should be set up to solve this problem? Thank you so much!

@robbmcleod

Thanks
@robbmcleod

I decided to add missing part of reply in regards to how convert numpy array from c# side.

All that left to do is load your script as module and call these functions.

Copy script that robbmcleod posted and save as array_converter.py in your directory

Put this into c# project

using Python.Runtime;
public static class NumpyUtils
{
    public static string script = File.ReadAllText("array_converter.py");
    public static dynamic ArrayConverterModule(){
        return PyModule.FromString("",script);
    }
}

Call it from c# and cast types right away

dynamic np_array = ...; //some numpy float32 array
dynamic array_converter = NumpyUtils.ArrayConverterModule();
float[] converted  array_converter.asNetArray(np_array);

Also I would like to add script that I use for slicing numpy arrays

using Python.Runtime;

public static class NumpyUtils
{
    public static string script = File.ReadAllText("array_converter.py");
    public static dynamic ArrayConverterModule(){
        return PyModule.FromString("",script);
    }
    /// <summary>
    /// Slices numpy array
    /// </summary>
    /// <param name="slice">Slice without object name and brackets []. For example : ":5,3"</param>
    public static dynamic Slice(PyObject np_array,string slice)
    {
        using var glob = new PyDict();
        glob["arr"] = np_array;
        var sliced = PythonEngine.Eval($"arr[{slice}]",
            globals: glob
        );
        return sliced;
    }
}

dynamic np_array = ...; //some numpy float32 array
var firstFive = NumpyUtils.Slice(np_array,":5");

Navigation Menu

Repository navigation

Implicit List conversion breaks backwards compatibility + numpy support #514

Environment

Details

21 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Implicit List conversion breaks backwards compatibility + numpy support #514

Description

Environment

Details

Activity

den-run-ai commented on Jul 20, 2017

BenjaminPelletier commented on Jul 20, 2017

den-run-ai commented on Jul 20, 2017

BenjaminPelletier commented on Jul 20, 2017

den-run-ai commented on Jul 20, 2017

yagweb commented on Jul 25, 2017

den-run-ai commented on Aug 5, 2017

vmuriart commented on Aug 10, 2017

den-run-ai commented on Aug 10, 2017

vmuriart commented on Aug 14, 2017

robbmcleod commented on Dec 8, 2017

21 remaining items

ddsleonardo commented on Oct 30, 2020

jcarole commented on Oct 30, 2020

ddsleonardo commented on Oct 30, 2020

lostmsu commented on Oct 30, 2020

jcarole commented on Oct 30, 2020

ddsleonardo commented on Oct 30, 2020

lostmsu commented on Oct 30, 2020

robbmcleod commented on May 6, 2021

CitizenInsane commented on Sep 23, 2021

robbmcleod commented on Sep 23, 2021

CitizenInsane commented on Sep 23, 2021

lostmsu commented on Sep 23, 2021

robbmcleod commented on Sep 23, 2021

Jiew23 commented on Dec 23, 2022

Kemsekov commented on Jan 4, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions