Skip to content

CLR not working on 32 bit Linux #1210

Open
Open
CLR not working on 32 bit Linux#1210
Assignees
filmor

Description

@daniol
daniol
opened on Aug 25, 2020

Environment

  • Pythonnet version: 2.5.1 and also 3.0.0-dev
  • Python version: 3.5.3, 2.7.13 and also 3.8.5 (from Debian repositories and also if compiled locally with --enable-shared)
  • MONO Versions: 6.10, 5.16 and also 5.20
  • Operating System: Linux (Debian 9 'stretch' Live, i686)

Details

Pythonnet seems not to be working on a Linux 32-Bit environment (i686) at all. I tested it on a "Debian live" (USB) system in two different computers, using Python from the Debian repositories and also building python from source code, here are the results:

  • Using Python from the Debian repositories:
    The python execution crashes just doing import clr
    See attachment: mono_crash.4383a3eb2.0.zip, trace.txt

  • Building Python from source code (with --enable-shared):

It doesn't crash, but clr throws an error when trying to load an assembly ("AttributeError: module 'clr' has no attribute 'AddReference'"). This was reported in some places around internet, but I don't have any clr module already installed which could cause a name collision. See output:

user@debian:~$ /opt/python/python3.5.3/bin/python3
Python 3.5.3 (default, Sep 27 2018, 17:25:39) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
>>> dir(clr)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'facade']
>>> str(clr)
"<module 'clr' from '/opt/python/python3.5.3/lib/python3.5/site-packages/clr.cpython-35m-i386-linux-gnu.so'>"
>>> 
user@debian:~$ /opt/python/python3.5.3/bin/pip3 list
clang (11.0)
pip (9.0.1)
pycparser (2.20)
pythonnet (2.5.1)
setuptools (28.8.0)
wheel (0.35.1)

Notes

  • On a x64-System it works, using the same version of Debian, Pythonnet and Python.
  • If this is really a limitation of 32-Bit-Systems that cannot be fixed, it should be documented on the "readme".

Activity

  1. lostmsu commented on Aug 25, 2020

    @lostmsu
    lostmsu
    on Aug 25, 2020
    Member

    @daniol it should be working on 32 bit Linux, but we don't have CI set for it.

  2. daniol commented on Aug 25, 2020

    @daniol
    daniol
    on Aug 25, 2020
    Author

    @lostmsu maybe it should, but it doesn't 😄 you can check it with a live version of debian 9 under a 32-bit machine. I don't know if this can be also reproduced on a VM.

  3. jborbely commented on Nov 3, 2020

    @jborbely
    jborbely
    on Nov 3, 2020

    I can also confirm this issue with pythonnet 2.5.0, 2.5.1 and the master branch (84e2735). The issue does not exist in pythonnet 2.4.0.

    I am not using a VM nor a live USB stick. I have 32-bit ubuntu installed on a HDD. Below shows the crash report when using Python 3.5.2 from the OS. I also created Python 3.7.9 and 3.8.6 virtual environments and importing clr fails in these environments.

    user@ubuntu:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.7 LTS
Release:	16.04
Codename:	xenial

user@ubuntu:~$ mono --version
Mono JIT compiler version 5.20.1.34 (tarball Tue Jul 16 22:57:14 UTC 2019)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  x86
	Disabled:      none
	Misc:          softdebug 
	Interpreter:   yes
	LLVM:          yes(600)
	Suspend:       hybrid
	GC:            sgen (concurrent by default)

user@ubuntu:~$ python3
Python 3.5.2 (default, Oct  7 2020, 17:19:02) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================
...

    The complete crash report is available here

    If I don't start the Python REPL to import clr but pass it in through the command line I get the following

    user@ubuntu:~$ python3 -c "import clr"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
SystemError: initialization of clr raised unreported exception
Error in sys.excepthook:
SystemError: ../Objects/listobject.c:199: bad argument to internal function

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 57, in apport_excepthook
    from cStringIO import StringIO
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
SystemError: PyEval_EvalFrameEx returned a result with an error set

Original exception was:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
SystemError: initialization of clr raised unreported exception

  4. filmor commented on Nov 4, 2020

    @filmor
    filmor
    on Nov 4, 2020
    Member

    @jborbely Would you mind bisecting the issue? I don't have a 32bit installation lying around right now.

  5. jborbely commented on Nov 4, 2020

    @jborbely
    jborbely
    on Nov 4, 2020

    @filmor A summary of bisecting is available here

    The criteria that I used to decide if a commit was good or bad was by running the following bash script (which I named issue1210.sh) after each bisect iteration. If pytest passed then that commit was considered good.

    #!/bin/bash
export LD_PRELOAD=/lib/i386-linux-gnu/libSegFault.so
export SEGFAULT_SIGNALS=all
export PYTHONUNBUFFERED=True
export BUILD_OPTS=""
export NUNIT_PATH="./packages/NUnit.ConsoleRunner.3.7.0/tools/nunit3-console.exe"
export RUN_TESTS="mono $NUNIT_PATH"
export EMBED_TESTS_PATH=""
export PERF_TESTS_PATH=""
source ~/py37env/bin/activate
python --version
pip --version
PY_LIBDIR=$(python -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))')
export LD_LIBRARY_PATH=$PY_LIBDIR:$LD_LIBRARY_PATH
pip install --upgrade setuptools
pip install --upgrade -r requirements.txt
pip uninstall pythonnet -y
coverage run setup.py install $BUILD_OPTS
python -m pytest

    Please let me know if you'd like further information.

  6. filmor commented on Nov 4, 2020

    @filmor
    filmor
    on Nov 4, 2020
    Member

    This is great, thanks. It points to PR #971, which uses the library loading to allow overriding one of Python's flags. I'll have a closer look, the actual writingnlooks fine, but maybe the lib loading assumes 64bitness somewhere.

  7. jborbely commented on Nov 5, 2020

    @jborbely
    jborbely
    on Nov 5, 2020

    I've gone a little further to see if there are additional issues if the changes from PR #971 were temporarily removed.

    Here's what I did:

    1. git checkout 4271e57
    2. deleted the lines that correspond to the SetNoSiteFlag() function in src/runtime/runtime.cs and src/runtime/pythonengine.cs
    3. ran my issue1210.sh script and pytest passed
    4. git checkout 770fc01
    5. deleted the lines that correspond to the SetNoSiteFlag() function in src/runtime/runtime.cs and src/runtime/pythonengine.cs
    6. ran my issue1210.sh script and got this error

    770fc01 immediately follows 4271e57 so 770fc01 seems to also introduce errors on 32-bit linux.

    770fc01 passed the tests on travis (ignoring that one pip issue) so it is okay on 64-bit linux.

  8. filmor
    self-assigned this
    on Nov 16, 2020

  9. daniol commented on Nov 19, 2020

    @daniol
    daniol
    on Nov 19, 2020
    Author

    I can also confirm that the issue does not exist in pythonnet 2.4.0.
    However, in order to work on pythonnet 2.4.0, the version of mono MUST be 5.20.

    The current mono version (6.12) throws the following error when importing the clr module in pythonnet 2.4.0:
    ImportError: /usr/lib/libmonosgen-2.0.so.1: undefined symbol: __cxa_begin_catch

  10. filmor
    closed this as completedon Aug 8, 2021

  11. daniol commented on Aug 9, 2021

    @daniol
    daniol
    on Aug 9, 2021
    Author

    @filmor why was this closed?

  12. lostmsu commented on Aug 9, 2021

    @lostmsu
    lostmsu
    on Aug 9, 2021
    Member

    @daniol @jborbely can you please test if the current master still has the issue? We reworked import mechanism, which is where @jborbely error happened.

  13. filmor commented on Aug 9, 2021

    @filmor
    filmor
    on Aug 9, 2021
    Member

    @daniol Sorry for the silent closure, I just skimmed the last message (was cleaning up old issues) and saw the __cxa_begin_catch which has been solved quite a while ago. I can confirm that the issue exists in the current master as well. One additional problem is that pythonnet uses the dotnet CLI for building now, which is not available for 32bit Linux. Installing from a wheel built on a 64bit machine works, though.

  14. filmor
    reopened this on Aug 9, 2021

  15. lostmsu commented on Aug 9, 2021

    @lostmsu
    lostmsu
    on Aug 9, 2021
    Member

    @filmor we could consider separating .NET build and Python build. Then Python build would only need to download Python.Runtime.dll or extract it from downloaded NuGet package.

  16. filmor commented on Aug 9, 2021

    @filmor
    filmor
    on Aug 9, 2021
    Member

    I'm not sure that's worth the hassle (and I like the current clean setup.py ;)). Building on CI and distributing wheels should be enough, and on all platforms that are officially supported to run .NET >5, dotnet build will work.

  17. filmor
    added this to the 3.0.0 milestone on Sep 2, 2021

  18. lostmsu commented on Jan 5, 2022

    @lostmsu
    lostmsu
    on Jan 5, 2022
    Member

    Removing this from 3.0.0 milestone. @filmor if you are actively working on it - readd. Because this is not a breaking change there's no harm in releasing it with 3.0.1 or 3.1.0.

  19. lostmsu
    removed this from the 3.0.0 milestone on Jan 5, 2022

  20. filmor commented on Jan 6, 2022

    @filmor
    filmor
    on Jan 6, 2022
    Member

    @jborbely @daniol Now that the respective section was adjusted in #1659, could you please check if that already fixes the problem?

  21. lostmsu commented on Jan 6, 2022

    @lostmsu
    lostmsu
    on Jan 6, 2022
    Member

    Also, to officially support 32 bit we need it in CI.

  22. jborbely commented on Jan 11, 2022

    @jborbely
    jborbely
    on Jan 11, 2022

    Since the build now depends on the dotnet CLI (which is not available on 32-bit Linux) I built a wheel on 64-bit Linux (using commit 75f4398) and copied the wheel to 32-bit Linux. Unfortunately, the issue importing clr remains, although with a different error compared to the error that I got back in November 2020.

    $ mono --version
Mono JIT compiler version 5.20.1.34 (tarball Tue Jul 16 22:57:14 UTC 2019)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  x86
        Disabled:      none
        Misc:          softdebug
        Interpreter:   yes
        LLVM:          yes(600)
        Suspend:       hybrid
        GC:            sgen (concurrent by default)

$ pip list
Package    Version
---------- ----------
cffi       1.15.0
clr-loader 0.1.7
pip        21.3.1
pycparser  2.21
pythonnet  3.0.0.dev1
setuptools 60.5.0
wheel      0.37.1

$ python
Python 3.7.10 (default, Feb 20 2021, 21:21:24)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
Failed to initialize pythonnet: System.NullReferenceException: Object reference not set to an instance of an object.
  at Python.Runtime.NewReferenceExtensions.Borrow (Python.Runtime.NewReference& reference) [0x0000f] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.Runtime.InitPyMembers () [0x0000b] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.Runtime.Initialize (System.Boolean initSigs) [0x000a5] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.PythonEngine.Initialize (System.Collections.Generic.IEnumerable`1[T] args, System.Boolean setSysArgv, System.Boolean initSigs) [0x00012] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.PythonEngine.Initialize (System.Boolean setSysArgv, System.Boolean initSigs) [0x00005] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.PythonEngine.InitExt () [0x00032] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.Loader.Initialize (System.IntPtr data, System.Int32 size) [0x0002f] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.NewReferenceExtensions.Borrow (Python.Runtime.NewReference& reference) [0x0000f] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.Runtime.InitPyMembers () [0x0000b] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.Runtime.Initialize (System.Boolean initSigs) [0x000a5] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.PythonEngine.Initialize (System.Collections.Generic.IEnumerable`1[T] args, System.Boolean setSysArgv, System.Boolean initSigs) [0x00012] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.PythonEngine.Initialize (System.Boolean setSysArgv, System.Boolean initSigs) [0x00005] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.PythonEngine.InitExt () [0x00032] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.Loader.Initialize (System.IntPtr data, System.Int32 size) [0x0002f] in <64958489404d4e7ba32bf649886da953>:0 ValueError: Empty module name

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/venv/lib/python3.7/site-packages/clr.py", line 6, in <module>
    load()
  File "/home/user/venv/lib/python3.7/site-packages/pythonnet/__init__.py", line 43, in load
    if func(''.encode("utf8")) != 0:
  File "/home/user/venv/lib/python3.7/site-packages/clr_loader/wrappers.py", line 20, in __call__
    return self._callable(ffi.cast("void*", buf_arr), len(buf_arr))
  File "/home/user/venv/lib/python3.7/site-packages/clr_loader/mono.py", line 91, in __call__
    res = _MONO.mono_runtime_invoke(self._ptr, ffi.NULL, params, exception)
SystemError: <cdata 'MonoObject *(*)(MonoMethod *, void *, void * *, MonoObject * *)' 0xb6ec3610> returned a result with an error set
>>> import clr
Failed to initialize pythonnet: System.InvalidOperationException: This property must be set before runtime is initialized
  at Python.Runtime.Runtime.set_PythonDLL (System.String value) [0x00007] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.Loader.Initialize (System.IntPtr data, System.Int32 size) [0x00023] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.Runtime.set_PythonDLL (System.String value) [0x00007] in <64958489404d4e7ba32bf649886da953>:0
  at Python.Runtime.Loader.Initialize (System.IntPtr data, System.Int32 size) [0x00023] in <64958489404d4e7ba32bf649886da953>:0 Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/venv/lib/python3.7/site-packages/clr.py", line 6, in <module>
    load()
  File "/home/user/venv/lib/python3.7/site-packages/pythonnet/__init__.py", line 44, in load
    raise RuntimeError("Failed to initialize Python.Runtime.dll")
RuntimeError: Failed to initialize Python.Runtime.dll

    Please let me know if there is something else that you would like me to try or if there is additional information about my system that you would like know.

  23. filmor commented on Jan 12, 2022

    @filmor
    filmor
    on Jan 12, 2022
    Member

    Yes, I see the same error. The failing borrow (due to a null pointer being returned) is from the builtins import. I'll try to replace it by PyEval_GetBuiltins (which we should probably use anyhow).

  24. filmor commented on Jan 13, 2022

    @filmor
    filmor
    on Jan 13, 2022
    Member

    I have replaced the PyModule_Import by PyEval_GetBuiltins, but this didn't improve things. In the debugger I can see that the address that we get for the Python functions is correct, but the return value is crap (nothing that I can immediately identify, but relatively close to our current code execution, so it could be that a value in one of the registers or on the stack is misinterpreted). It could be that this function call is the first one, where the return value matters and that's why we are seeing this problem. It could also be that this has to do with the unmanaged function pointers that we are using nowadays (we used to use P/Invoke). I'll try to create a minimal example to reproduce the issue, maybe this is a problem in Mono.

  25. filmor commented on Jan 14, 2022

    @filmor
    filmor
    on Jan 14, 2022
    Member

    I have tested this now with Python 3.9, here it already fails on PyLong_FromLongLong which we call in Runtime.NewRun on a simple value. I have checked the stack and it looks wrong.

    What I would expect: <return address (4 bytes)> <value as LE (8 bytes)> (this is also what you can see for "native" calls)
    What I see: <some address (4 bytes)> <some other address (4 bytes)> <value as LE (8 bytes)>

    So the calli that is generated for our unmanaged delegate call (which is correct, as said, I have verified the addresses) pushes another parameter (I think) onto the stack, which is then frankensteined with the lower half of the value into a wrong value that is returned. If it does that on all calls, it will quite quickly destroy the stack.

    Does anyone have more insight here or knows a bit more about how Mono translates calli (or knows someone who does)?

  26. lostmsu commented on Jan 14, 2022

    @lostmsu
    lostmsu
    on Jan 14, 2022
    Member

    @filmor have you tried with .NET Core? If the behavior is different, I would simply open a bug in Mono.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions