Add support for running pytest from dotnet core 2.0 (dotnet <path>/netcoreapp2.0/nPython.dll)

Environment

Pythonnet version: Any
Python version: Any
Operating System: Any

Details

Describe what you were trying to get done.

pytest suite now works unly under Windows/Mono but not under .Net Core 2.0
What commands did you run to trigger this issue? If you can provide a
Minimal, Complete, and Verifiable example
this will help us understand the issue.

import sys
sys.path.append('/users/isys/icronyn/pyenvs/pythonnet/lib/python2.7/site-packages/pytest-2.9.1-py2.7.egg')
sys.path.append('/users/isys/icronyn/pyenvs/pythonnet/lib/python2.7/site-packages/py-1.4.31-py2.7.egg')
import pytest
pytest.main()

============================================================================== test session starts ==============================================================================
platform linux2 -- Python 2.7.5, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: /users/isys/icronyn/git/pythonnet, inifile: 
collected 376 items 

src/tests/test_array.py .....................................F..
src/tests/test_callback.py ..
src/tests/test_class.py .......F..F..FF...
src/tests/test_clrmethod.py FFFFF
src/tests/test_compat.py ......FFFF....F.
src/tests/test_constructors.py ....
src/tests/test_conversion.py ....................
src/tests/test_delegate.py ...
Unhandled Exception: System.MissingMethodException: Constructor on type 'System.Reflection.Emit.TypeBuilder' not found.
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(Type type, Object[] args)
   at Python.Runtime.DelegateManager.GetDelegate(Type dtype, IntPtr callable) in /users/isys/icronyn/git/pythonnet/src/runtime/delegatemanager.cs:line 159
   at Python.Runtime.DelegateObject.tp_new(IntPtr tp, IntPtr args, IntPtr kw) in /users/isys/icronyn/git/pythonnet/src/runtime/delegateobject.cs:line 61
   at Python.Runtime.MetaType.tp_call(IntPtr tp, IntPtr args, IntPtr kw) in /users/isys/icronyn/git/pythonnet/src/runtime/metatype.cs:line 148
   at Python.Runtime.Runtime.Py_Main(Int32 argc, String[] argv)
   at Python.Runtime.PythonConsole.Main(String[] args) in /users/isys/icronyn/git/pythonnet/src/console/pythonconsole.cs:line 37
Aborted

Bad news:

I tried to add this support and found too many big and small problems.
So this feature requires up to a full working week.

I will publish my current work state in a new branch little bit later.

@dmitriyse

Thanks for this @dmitriyse - sorry for raising so many issues.
Let me know if there's anything I can do to help, I'm happy to run tests on your branch when you're ready

@Cronan

@Cronan thank you for a lot of testing efforts. You issues and reports are very useful. Probably I will reopen some of them later for separate problem solving.
At first look this was a one issue, but after deep dive I found at least two separate big problems.

@dmitriyse

Hi @dmitriyse , where are you with this? Is there anything I can do?

@Cronan

Hi @Cronan, unfortunately I am busy now. I forgot to publish my current progress on this issue. I will create branch today.

The pytests core 2.0 fixes branch has been created. https://github.com/pythonnet/pythonnet/tree/pytests-core

@dmitriyse

I'll take a look today, thanks @dmitriyse

@dmitriyse

@dmitriyse definitely looking better, gets a lot further. It fails a few tests, mainly due to missing classes in .Net core like System.Drawing, Hashtable, a few others, but crashes hard when it gets to here:

_____________________________ test_sys_argv_state ______________________________

filepath = <function make_filepath at 0x2226410>

    def test_sys_argv_state(filepath):
        """Test sys.argv state doesn't change after clr import.
        To better control the arguments being passed, test on a fresh python
        instance with specific arguments"""
    
        script = filepath("argv-fixture.py")
>       out = check_output([sys.executable, script, "foo", "bar"])

src/tests/test_sysargv.py:16: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/tests/_compat.py:71: in check_output
    output = subprocess.check_output(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

popenargs = (['/users/isys/icronyn/pyenvs/tmp-2eb586eb98020886/bin/python', '/users/isys/icronyn/git/pythonnet/src/tests/fixtures/netstandard2.0/argv-fixture.py', 'foo', 'bar'],)
kwargs = {}, process = <subprocess.Popen object at 0x1c04a50>, output = ''
unused_err = None, retcode = 1
cmd = ['/users/isys/icronyn/pyenvs/tmp-2eb586eb98020886/bin/python', '/users/isys/icronyn/git/pythonnet/src/tests/fixtures/netstandard2.0/argv-fixture.py', 'foo', 'bar']

    def check_output(*popenargs, **kwargs):
        r"""Run command with arguments and return its output as a byte string.
    
        If the exit code was non-zero it raises a CalledProcessError.  The
        CalledProcessError object will have the return code in the returncode
        attribute and output in the output attribute.
    
        The arguments are the same as for the Popen constructor.  Example:
    
        >>> check_output(["ls", "-l", "/dev/null"])
        'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'
    
        The stdout argument is not allowed as it is used internally.
        To capture standard error in the result, use stderr=STDOUT.
    
        >>> check_output(["/bin/sh", "-c",
        ...               "ls -l non_existent_file ; exit 0"],
        ...              stderr=STDOUT)
        'ls: non_existent_file: No such file or directory\n'
        """
        if 'stdout' in kwargs:
            raise ValueError('stdout argument not allowed, it will be overridden.')
        process = Popen(stdout=PIPE, *popenargs, **kwargs)
        output, unused_err = process.communicate()
        retcode = process.poll()
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd, output=output)
E           CalledProcessError: Command '['/users/isys/icronyn/pyenvs/tmp-2eb586eb98020886/bin/python', '/users/isys/icronyn/git/pythonnet/src/tests/fixtures/netstandard2.0/argv-fixture.py', 'foo', 'bar']' returned non-zero exit status 1

/usr/lib64/python2.7/subprocess.py:575: CalledProcessError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "/users/isys/icronyn/git/pythonnet/src/tests/fixtures/netstandard2.0/argv-fixture.py", line 10, in <module>
    import clr
ImportError: No module named clr

@dmitriyse

@dmitriyse Below please find the tests that failed "normally" - some could be fixed by choosing classes that are part of .Net Standard 2.0 ...

=================================== FAILURES ===================================
____________________ test_boxed_value_type_mutation_result _____________________

    def test_boxed_value_type_mutation_result():
        """Test behavior of boxed value types."""
    
        # This test actually exists mostly as documentation of an important
        # concern when dealing with value types. Python does not have any
        # value type semantics that can be mapped to the CLR, so it is easy
        # to accidentally write code like the following which is not really
        # mutating value types in-place but changing boxed copies.
    
>       from System.Drawing import Point
E       ImportError: No module named Drawing

src/tests/test_array.py:1146: ImportError
_____________________________ test_basic_subclass ______________________________

    def test_basic_subclass():
        """Test basic subclass of a managed class."""
>       from System.Collections import Hashtable
E       ImportError: cannot import name Hashtable

src/tests/test_class.py:64: ImportError
___________________________ test_struct_construction ___________________________

    def test_struct_construction():
        """Test construction of structs."""
>       from System.Drawing import Point
E       ImportError: No module named Drawing

src/tests/test_class.py:121: ImportError
____________________________ test_override_get_item ____________________________

    def test_override_get_item():
        """Test managed subclass overriding __getitem__."""
>       from System.Collections import Hashtable
E       ImportError: cannot import name Hashtable

src/tests/test_class.py:172: ImportError
____________________________ test_override_set_item ____________________________

    def test_override_set_item():
        """Test managed subclass overriding __setitem__."""
>       from System.Collections import Hashtable
E       ImportError: cannot import name Hashtable

src/tests/test_class.py:193: ImportError
______________________ test_set_and_get_property_from_py _______________________

    def test_set_and_get_property_from_py():
        """Test setting and getting clr-accessible properties from python."""
>       t = ExampleClrClass()
E       TypeError: no constructor matches given arguments

src/tests/test_clrmethod.py:30: TypeError
______________________ test_set_and_get_property_from_clr ______________________

    def test_set_and_get_property_from_clr():
        """Test setting and getting clr-accessible properties from the clr."""
>       t = ExampleClrClass()
E       TypeError: no constructor matches given arguments

src/tests/test_clrmethod.py:39: TypeError
__________________ test_set_and_get_property_from_clr_and_py ___________________

    def test_set_and_get_property_from_clr_and_py():
        """Test setting and getting clr-accessible properties alternatingly from the clr and from python."""
>       t = ExampleClrClass()
E       TypeError: no constructor matches given arguments

src/tests/test_clrmethod.py:49: TypeError
________________________ test_method_invocation_from_py ________________________

    def test_method_invocation_from_py():
        """Test calling a clr-accessible method from python."""
>       t = ExampleClrClass()
E       TypeError: no constructor matches given arguments

src/tests/test_clrmethod.py:67: TypeError
_______________________ test_method_invocation_from_clr ________________________

    def test_method_invocation_from_clr():
        """Test calling a clr-accessible method from the clr."""
>       t = ExampleClrClass()
E       TypeError: no constructor matches given arguments

src/tests/test_clrmethod.py:72: TypeError
_________________________ test_dotted_name_import_from _________________________

    def test_dotted_name_import_from():
        """Test dotted-name 'import from'."""
>       from CLR.System import Xml
E       ImportError: cannot import name Xml

src/tests/test_compat.py:115: ImportError
___________________ test_dotted_name_import_from_with_alias ____________________

    def test_dotted_name_import_from_with_alias():
        """Test dotted-name 'import from' with aliasing."""
>       from CLR.System import Xml as myXml
E       ImportError: cannot import name Xml

src/tests/test_compat.py:134: ImportError
_________________________ test_from_module_import_star _________________________

    def test_from_module_import_star():
        """Test from module import * behavior."""
        count = len(locals().keys())
>       m = __import__('CLR.System.Management', globals(), locals(), ['*'])
E       ImportError: No module named Management

src/tests/test_compat.py:154: ImportError
_________________________ test_explicit_assembly_load __________________________

    def test_explicit_assembly_load():
        """Test explicit assembly loading using standard CLR tools."""
        from CLR.System.Reflection import Assembly
        from CLR import System
        import sys
    
        assembly = Assembly.LoadWithPartialName('System.Data')
        assert assembly is not None
    
>       import CLR.System.Data
E       ImportError: No module named Data

src/tests/test_compat.py:176: ImportError
_____________________________ test_module_get_attr _____________________________

    def test_module_get_attr():
        """Test module getattr behavior."""
        import CLR.System as System
    
        int_type = System.Int32
        assert is_clr_class(int_type)
    
>       module = System.Xml
E       AttributeError: Xml

src/tests/test_compat.py:228: AttributeError
___________________________ test_interface_property ____________________________

    def test_interface_property():
        """Test properties of interfaces. Added after a bug report
           that an IsAbstract check was inappropriate and prevented
           use of properties when only the interface is known."""
>       from System.Collections import Hashtable, ICollection
E       ImportError: cannot import name Hashtable

src/tests/test_property.py:144: ImportError

The main problem here is the implicit assembly loading algorithm. It should be hard redesigned to allow to map class to an assembly under .Net Core 2.0. I did not try to figureout some solution. Maybe I will have some time to this at the Febrary.

I already tried to link required nuget packages but this is not enough.

To be honest, if I’m writing C# libs to be run in “full flavour” .Net 4, and in Core, I either need to stick to Standard 2.0 libraries, or I’m going to be using additional explicit references for Core. Maybe that’s the same for pythonnet? As far as the tests go, I was going to suggest changing them to use .Net Standard 2.0 types only - what do you think?

.Net Core 2.0 apps in VS2017 solve the problem by adding a heap of built-in references when you create a new project ...

Hi! I need to think about this solution.

Please publish your changes in some temporary fork.

.Net Core 2.1 preview should be released soon. So I prefere to wait it before resolve this issue, some differencies between Classic .Net Framework 4.x and .Net Core 2.0 will be resolved.

Navigation Menu

Repository navigation

Add support for running pytest from dotnet core 2.0 (dotnet <path>/netcoreapp2.0/nPython.dll) #590

Environment

Details

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add support for running pytest from dotnet core 2.0 (dotnet <path>/netcoreapp2.0/nPython.dll) #590

Description

Environment

Details

Activity

dmitriyse commented on Dec 13, 2017

Cronan commented on Dec 13, 2017

dmitriyse commented on Dec 13, 2017

Cronan commented on Jan 8, 2018

dmitriyse commented on Jan 9, 2018

dmitriyse commented on Jan 9, 2018

Cronan commented on Jan 10, 2018

Cronan commented on Jan 19, 2018

Cronan commented on Jan 19, 2018

dmitriyse commented on Jan 19, 2018

dmitriyse commented on Jan 19, 2018

Cronan commented on Jan 23, 2018

Cronan commented on Jan 23, 2018

dmitriyse commented on Jan 23, 2018

dmitriyse commented on Jan 23, 2018

dmitriyse commented on Feb 1, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions