Pythonnet crashes .Net application on exit. #245
Description
If we embed Python.Net into DotNet application, it crashes on Linux.
Crash occurs when application exits.
PythonEngine.Initialize();
try
{
using (Py.GIL())
{
dynamic sysModule = Py.Import("sys");
Console.WriteLine("Python engine version:");
Console.WriteLine(sysModule.version);
}
// This workaround reduces risk of Mono crash.
// Problem will be fixed in mono 4.6
if (Type.GetType("Mono.Runtime") != null)
{
Runtime.Py_Main(3, new[] { "/pcfgtest.exe", "-c", "exit" });
}
else
{
// Program will crash if we will try to do this under Windows.
//// Runtime.Py_Main(3, new[] { "/pcfgtest.exe", "-c", "exit" });
}
}
finally
{
PythonEngine.Shutdown();
}This is sample of code that produces crash.
Also it contains 'probably workaround".
Console output :
tux@pythonnet:/mnt/Projects/pythonnet/2/src/configtest/bin/Release$ mono pcfgtest.exe
Starting application...
Python.runtime.dll substituted by Python.Runtime-Py35-Linux64.dll.
Python engine version:
3.5.2 (default, Jul 5 2016, 12:43:10)
[GCC 5.4.0 20160609]
Shutting down finalizer thread timed out.
* Assertion at gc.c:867, condition `finalizer_thread_exited' not met
Stacktrace:
Native stacktrace:
mono() [0x4a77ca]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0) [0x7f48084463d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7f4807e8b418]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f4807e8d01a]
mono() [0x6408c9]
mono() [0x640ad7]
mono() [0x640c26]
mono() [0x5aee5e]
mono(mono_runtime_cleanup+0xe) [0x5a85de]
mono() [0x423cf7]
mono(mono_main+0x10d7) [0x47a087]
mono() [0x421818]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f4807e76830]
mono() [0x421c11]
Debug info from gdb:
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
Aborted (core dumped)I tried everything - python 2.7, python 3.5, Mono 4.2, 4.4.
It does not matter, behavior always the same.
Also i tried to build with Py_DEBUG under linux (and pass python library from python3.5-dbg package)
I receive assertions on PythonEngine.Init:
- Decref -1 Assertions
- td_type != NULL Assertion
Activity
royalstream commentedon Oct 6, 2016
YMMV but here's some potentially useful feedback on this issue.
I'm using OS X 10.11.3 and Anaconda Python (see the specific version in the outputs).
Using Mono 4.4.2 I get exactly the same error, even if shrink your code to just this:
The error is the following:
After upgrading to a newer version of Mono (4.6.1) I get a slightly different error description but I suspect deep down the problem remains the same:
Now here's the interesting part:If I abstain from using dynamic objects, it never crashes anymore (Mono 4.6.1)
In other words, I changed the code to
sysModule.GetAttr("version").ToString()and the problem is gone.Additional Info:
I noticed the problem only happened if I used the FSharp.Interop.Dynamic and Dynamitey libraries. I was about to blame it on those two until I realized the same issue happens with C# if you use dynamic objects.den-run-ai commentedon Oct 7, 2016
@royalstream @dmitriyse these issues with finalizer were reported recently in Mono:
https://bugzilla.xamarin.com/show_bug.cgi?id=41833
https://bugzilla.xamarin.com/show_bug.cgi?id=40399
https://bugzilla.xamarin.com/show_bug.cgi?id=42665
Few potential fixes have been made and also workarounds suggested, so have a look there.
But your second issue with WAIT_TIMEOUT is not fixed yet in Mono:
https://bugzilla.xamarin.com/show_bug.cgi?id=44355
If you open a new issue in Mono, then please link it here.
Can you also try explicitly disposing module like mentioned here:
https://mail.python.org/pipermail/pythondotnet/2008-May.txt
den-run-ai commentedon Oct 7, 2016
With mono 4.2.4.4 and python 2.7:
I'm getting this error:
royalstream commentedon Oct 14, 2016
Now I'm getting the same error on a different machine but without using Dynamic Objects.
So much for my theory then :|
For the time being I added
Process.GetCurrentProcess().Kill()as the last line of mymainfunction, which is absolutely horrible but avoids the crash until this gets fixed in Mono (I don't believe pythonnet is causing it)den-run-ai commentedon Oct 14, 2016
@royalstream @dmitriyse can you guys try acquiring the GIL without using
Py.GIL()?royalstream commentedon Oct 15, 2016
And how do you do that? sorry for the "noob" question but all the code examples I've seen call
Py.GIL()dmitriyse commentedon Oct 15, 2016
I am using this patched version of Runtime.Shutdown() method:
(No logic, just magic)
And this problem gone.
This solution works fine, no one issue in last 3 monthes. (On different OS and pythons include conda).
So I am assuming that bug exists in PythonNet.
P.S to use this workaround in UCS4 builds you should apply also apply this patch:
dmitriyse@8638d57
dmitriyse commentedon Oct 15, 2016
As i understand, PY_DEGUB build should be used to ensure valid reference counting, indexes access and so on. I tried to build and run PythonNet with this flag and received violations near in every place. I think that PythonNet in general requires big "Cleanup", and probably this problem will disappear.
den-run-ai commentedon Oct 15, 2016
@royalstream you could have a look at source code of
Py.GIL();)Here is a quick example:
den-run-ai commentedon Oct 15, 2016
@dmitriyse I always thought that PY_DEGUB is only helpful for debugging purposes of CPython interpreter, because it produces debug build of CPython?
https://docs.python.org/3/c-api/intro.html#debugging-builds
But let's open a separate issue for Py_DEBUG failures.
royalstream commentedon Oct 15, 2016
Sorry for the misunderstanding, I thought you meant avoiding
Py.GIL()and whatever its code was doing.Now I get a really bad malloc/free error, at least on the machine I'm using right now. I'll try this on a different machine next week.
I'm using Mono 4.6.0 (Stable 4.6.0.245/746756c Mon Sep 19 13:57:55 BST 2016)
dmitriyse commentedon Oct 15, 2016
Looks like you missed this patch.
P.S to use this workaround in UCS4 builds you should apply also apply this patch:
dmitriyse@8638d57
Also you can check binary distro: https://github.com/dmitriyse/pythonnet/releases/tag/v2.2.0-bin-RC
It should quickly show this trick in work. (Supports only Python 3.5 under x64 os)
try mono pyboot.exe
royalstream commentedon Oct 15, 2016
I didn't miss it but it looks like it doesn't apply to my current Python 2.7 installation (which seems to be UCS2):
royalstream commentedon Oct 15, 2016
UPDATE: I installed the Python 3.5 version of Anaconda (which is UCS4) and the patch works great, no more crashes.
PS: If I run the unpatched version of Pythonnet I get a SIGSEGV so the patch is definitely helping.
den-run-ai commentedon Dec 19, 2016
@dmitriyse @royalstream how are you able to embed Anaconda's CPython using pythonnet on Linux? I tried the following and it is failing with
System.DllNotFoundException: python3.5m, butimport clrworks fine:dmitriyse commentedon Dec 19, 2016
probably you need to add "CONDA HOME"\lib to LD_LIBRARY_PATH. Or create symlink from "CONDA HOME"\lib\libpython3.5m.so to <python.runtime.dll location>\libpython3.5m.so
den-run-ai commentedon Dec 19, 2016
@dmitriyse do you mean to add \lib to search path of
$LD_LIBRARY_PATH?den-run-ai commentedon Dec 19, 2016
@dmitriyse creating symlink did not resolve this. adding the libpython path to $LD_LIBRARY_PATH did not resolve the issue either.
Fix GC on PyObject during Finalizing
vmuriart commentedon Feb 23, 2017
for anyone looking into the
LD_LIBRARY_PATH. Here's how it was solved forTravis.Enable TestPyTupleInvalidAppend test
vmuriart commentedon Feb 27, 2017
@dmitriyse @royalstream can you verify if the latest master have solved your issue? Between #365 and #400 we should have fixed the issues with shutting down.
den-run-ai commentedon Feb 27, 2017
@vmuriart do we have any tests in pythonnet now for code snippets provided above, which reproduced the crash:
#245 (comment)
vmuriart commentedon Feb 27, 2017
Many :). At the beginning of
v2.2.2almost every embedded test produced this error in one way or another. This was one of the reasons we weren't able to run embedded tests before in AppVeyor/Travis.The tests in
initializein particular make sure that initialize and shutdown work well. Specific to the example listed though, we have other examples that cover it likeAssignNone.vmuriart commentedon Mar 3, 2017
Closing issue. Let us know if the issue is still present after the aforementioned fixes.
Enable TestPyTupleInvalidAppend test
Rebase with master (#2)