Environment
- Pythonnet version: 3.0.0-dev (latest master on 18/2/2021)
- Python version: 3.8.5
- Operating System: Ubuntu 20.04
- .NET Runtime: .NET 5.0.2
Details
- Describe what you were trying to get done.
I am trying to use the latest pythonnet in a virtual environment. If I call my simple demo script using python command in PATH, pythonnet.load() crashes. It is the same with any other PATH-based pythonX command. Using a full path works fine. See the details and the analysis below.
Preparation
# Create virtual env.
python3 -m virtualenv venv
. venv/bin/activate
# Install the latest pythonnet
git clone https://github.com/pythonnet/pythonnet.git
pip install ./pythonnet/
# Get .NET 5 runtime
wget https://download.visualstudio.microsoft.com/download/pr/824175f2-5577-46ec-a675-95d2f652575f/07dafc2398e669afa7d25f2d47398c36/dotnet-runtime-5.0.2-linux-x64.tar.gz
mkdir dotnet_5.0.2
tar xfzv dotnet-runtime-5.0.2-linux-x64.tar.gz -C dotnet_5.0.2/
Runtime config (cat runtimeconfig.json)
{
"runtimeOptions": {
"tfm": "net5.0",
"framework": {
"name": "Microsoft.NETCore.App",
"version": "5.0.2"
}
}
}
Python script (cat bug.py):
import os
import clr_loader
import pythonnet
runtime_config = os.path.join(os.path.dirname(__file__), "runtimeconfig.json")
dotnet_root = os.path.join(os.path.dirname(__file__), "dotnet_5.0.2")
runtime = clr_loader.get_coreclr(runtime_config=runtime_config, dotnet_root=dotnet_root)
pythonnet.set_runtime(runtime)
pythonnet.load()
import System
from System import String
s = String("Hello World from Python.NET for .NET 5!")
System.Console.WriteLine(s)> which python
/home/takacs/src/pythonnet/venv/bin/python
>`which python` bug.py
Hello World from Python.NET for .NET 5!
> python bug.py
Traceback (most recent call last):
File "bug.py", line 12, in <module>
pythonnet.load()
File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/pythonnet/__init__.py", line 50, in load
_FFI.dlopen(libpython, posix.RTLD_NODELETE | posix.RTLD_LOCAL)
File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py", line 150, in dlopen
lib, function_cache = _make_ffi_library(self, name, flags)
File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py", line 832, in _make_ffi_library
backendlib = _load_backend_lib(backend, libname, flags)
File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py", line 827, in _load_backend_lib
raise OSError(msg)
OSError: cannot load library '/home/takacs/src/pythonnet/python': /home/takacs/src/pythonnet/python: cannot open shared object file: No such file or directory. Additionally, ctypes.util.find_library() did not manage to locate a library called '/home/takacs/src/pythonnet/python'
- If there was a crash, please include the traceback here.
Traceback (most recent call last):
File "bug.py", line 12, in <module>
pythonnet.load()
File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/pythonnet/__init__.py", line 50, in load
_FFI.dlopen(libpython, posix.RTLD_NODELETE | posix.RTLD_LOCAL)
File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py", line 150, in dlopen
lib, function_cache = _make_ffi_library(self, name, flags)
File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py", line 832, in _make_ffi_library
backendlib = _load_backend_lib(backend, libname, flags)
File "/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py", line 827, in _load_backend_lib
raise OSError(msg)
OSError: cannot load library '/home/takacs/src/pythonnet/python': /home/takacs/src/pythonnet/python: cannot open shared object file: No such file or directory. Additionally, ctypes.util.find_library() did not manage to locate a library called
Analysis
This issue is in the pythonnet.find_libpython module. The dlinfo.dli_fname string contains the actual command that was used to start the Python interpreter. In the case of pure python command, os.path.realpath fails the properly translate this back to the real executable.
|
path = os.path.realpath(dlinfo.dli_fname.decode()) |
)
Adding a few debug lines properly demonstrate this:
def _linked_libpython_unix():
libdl = ctypes.CDLL(ctypes.util.find_library("dl"))
libdl.dladdr.argtypes = [ctypes.c_void_p, ctypes.POINTER(Dl_info)]
libdl.dladdr.restype = ctypes.c_int
dlinfo = Dl_info()
retcode = libdl.dladdr(
ctypes.cast(ctypes.pythonapi.Py_GetVersion, ctypes.c_void_p),
ctypes.pointer(dlinfo))
if retcode == 0: # means error
return None
print(f"DEBUG dlinfo.dli_fname.decode(): {dlinfo.dli_fname.decode()}")
path = os.path.realpath(dlinfo.dli_fname.decode())
print(f"DEBUG path: {path}")
if path == os.path.realpath(sys.executable):
return None
return path> ./venv/bin/python bug.py
DEBUG dlinfo.dli_fname.decode(): ./venv/bin/python
DEBUG path: /usr/bin/python3.8
Hello World from Python.NET for .NET 5!
> ../pythonnet/venv/bin/python bug.py
DEBUG dlinfo.dli_fname.decode(): ../pythonnet/venv/bin/python
DEBUG path: /usr/bin/python3.8
Hello World from Python.NET for .NET 5!
>python bug.py
DEBUG dlinfo.dli_fname.decode(): python
DEBUG path: /home/takacs/src/pythonnet/python
Traceback (most recent call last):
...
The last case demonstrates that os.path.realpath just extends python with the current path which is obviously incorrect.
As a result, pythonnet.load() method's libpython variable will point to a library which cannot be loaded (
|
_FFI.dlopen(libpython, posix.RTLD_NODELETE | posix.RTLD_LOCAL) |
).
However, if this part is disabled, the application will crash later in the .NET
Python.Runtime for the same reason.
Environment
Details
I am trying to use the latest
pythonnetin a virtual environment. If I call my simple demo script usingpythoncommand in PATH, pythonnet.load() crashes. It is the same with any other PATH-basedpythonXcommand. Using a full path works fine. See the details and the analysis below.Preparation
Runtime config (
cat runtimeconfig.json)Python script (
cat bug.py):Minimal, Complete, and Verifiable example
this will help us understand the issue.
Analysis
This issue is in the
pythonnet.find_libpythonmodule. Thedlinfo.dli_fnamestring contains the actual command that was used to start the Python interpreter. In the case of purepythoncommand,os.path.realpathfails the properly translate this back to the real executable.pythonnet/pythonnet/find_libpython/__init__.py
Line 90 in 0f5e781
Adding a few debug lines properly demonstrate this:
The last case demonstrates that
os.path.realpathjust extendspythonwith the current path which is obviously incorrect.As a result,
pythonnet.load()method'slibpythonvariable will point to a library which cannot be loaded (pythonnet/pythonnet/__init__.py
Line 50 in 0f5e781
However, if this part is disabled, the application will crash later in the .NET
Python.Runtimefor the same reason.