Using pythonnet in docker container

We are trying to support algorithms written in python in QuantConnect/Lean (which is written in C#).
So far, we were able to accomplish that locally in Windows. However, we have to make it work in a docker container in order to launch algorithms in our servers, but we could't make pythonnet work in that environment.

Here are the steps we are following:

Run a container:
docker run -it quantconnect/lean:foundation /bin/bash
Please checkout what system configuration quantconnect/lean:foundation has here.
Install python:

apt-get update 
apt-get install -y libglib2.0-dev python-all-dev python-pip git 
pip install --upgrade pip && pip install pandas scipy numpy

Install pythonnet
pip install git+https://github.com/pythonnet/pythonnet

After that, we ran the three winforms demos. All throw the same exception:

 System.TypeInitializationException: The type initializer for 'System.Windows.Forms.WindowsFormsSynchronizationContext' threw an exception.
  at System.Windows.Forms.Control..ctor () [0x00014] in <5d999439b3404adfac86e7eebb5f00d6>:0
  at System.Windows.Forms.ScrollableControl..ctor () [0x00000] in <5d999439b3404adfac86e7eebb5f00d6>:0
  at System.Windows.Forms.ContainerControl..ctor () [0x0000e] in <5d999439b3404adfac86e7eebb5f00d6>:0
  at System.Windows.Forms.Form..ctor () [0x00012] in <5d999439b3404adfac86e7eebb5f00d6>:0
  at (wrapper managed-to-native) System.Reflection.MonoCMethod:InternalInvoke (System.Reflection.MonoCMethod,object,object[],System.Exception&)
  at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00002] in <dca3b561b8ad4f9fb10141d81b39ff45>:0

Finally, we ran the example of embedding Python in .NET from pythonnet README:
mcs /reference:Python.Runtime.dll test.cs && mono test.exe
that yields the following exception:

Unhandled Exception:
System.EntryPointNotFoundException: Py_IsInitialized
  at (wrapper managed-to-native) Python.Runtime.Runtime:Py_IsInitialized ()
  at Python.Runtime.Runtime.Initialize () [0x0000d] in <3b52ecce105142dc9f5c47b1d8f64b0e>:0
  at Python.Runtime.PythonEngine.Initialize () [0x00014] in <3b52ecce105142dc9f5c47b1d8f64b0e>:0
  at Python.Runtime.Py.GIL () [0x0000a] in <3b52ecce105142dc9f5c47b1d8f64b0e>:0
  at Test.Test.Main (System.String[] args) [0x00000] in <d894aae141fa4a98b826174854b8bf77>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.EntryPointNotFoundException: Py_IsInitialized
  at (wrapper managed-to-native) Python.Runtime.Runtime:Py_IsInitialized ()
  at Python.Runtime.Runtime.Initialize () [0x0000d] in <3b52ecce105142dc9f5c47b1d8f64b0e>:0
  at Python.Runtime.PythonEngine.Initialize () [0x00014] in <3b52ecce105142dc9f5c47b1d8f64b0e>:0
  at Python.Runtime.Py.GIL () [0x0000a] in <3b52ecce105142dc9f5c47b1d8f64b0e>:0
  at Test.Test.Main (System.String[] args) [0x00000] in <d894aae141fa4a98b826174854b8bf77>:0

@AlexCatarino

@AlexCatarino

winforms is not available on linux.
can you try with latest master of pythonnet? we fixed some issues with wrong libpython settings in some linux distros.
can you replicate the settings from .travis.yml? particularly use mono version 4.2.4.4, due to bugs in later and previous versions.

I am using the last master, when I install pythonnet using pip+git, right?
pip install git+https://github.com/pythonnet/pythonnet
Let me try mono 4.2.44! Thanks.

yes, pip+git should get you the latest. but you should also add -U and --force flags to force update the package.

…

On Wed, Dec 7, 2016 at 5:53 PM, Alexandre Catarino ***@***.*** > wrote: I am using the last master, when I install pythonnet using pip+git, right? pip install git+https://github.com/pythonnet/pythonnet Let me try mono 4.2.44! Thanks. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#303 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgZ5a9TL9fAkOxM8hnryXmZn4R3pDP7ks5rF0b5gaJpZM4LHQPR> .

I apologize, WinForms is available on Mono, but I have not tested this:

http://www.mono-project.com/docs/gui/winforms/

From my base image: docker run -it phusion/baseimage:0.9.19 /bin/bash
I have replicated the setting from .travis.yml. It did pass the test:

> python src/tests/runtests.py
Load clr import hook
.............................................................Python.Test.SubclassConstructorTest
..........................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 343 tests in 2.956s

OK

However, when I ran the example of embedding Python in .NET from pythonnet README, I got the same exception described above.

@denfromufa any thoughts on how to fix this? Its blocking using Python.Net in our platform / https://github.com/QuantConnect/Lean (It works fine in windows)

I have similar issues with embedding CPython using pythonnet on Linux: #245 Scroll to bottom of the issue. I will try more, but I do have expertise on how Python works on Linux, particularly dlopen behavior. I got it working in the past, but somehow it broke again. Perhaps posting on mailing list or SO will get more useful suggestions. Also some pythonnet developers are more knowledgeable about Linux than me. On Tue, Dec 20, 2016, 5:34 PM Jared <notifications@github.com> wrote: @denfromufa <https://github.com/denfromufa> any thoughts on how to fix this? Its blocking using Python.Net in our platform / https://github.com/QuantConnect/Lean (It works fine in windows) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#303 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgZ5fqEe1ltWR_Lah0oXB1BoEMZ3rB8ks5rKGXrgaJpZM4LHQPR> .

QuantConnect/Lean is able to support python in Linux with pythonnet!

In order to do it, we needed to discarded changes made by #300. Since ldd does not show libpython, pythonnet is compiled with the PYTHON_WITHOUT_ENABLE_SHARED flag that results in the reported exception. On the other hand, without that flag, import clr fails.

However, if we get Python.Runtime.dll, Python.Runtime.dll.config, clr.so, create a dynamic link to the library (ln -s /usr/lib/python2.7/config-x86_64-linux-gnu/libpython2.7.so .) and set the environment to the directory these files are found, we are able to embed Python in .NET:

$ mcs test.cs /r:Python.Runtime.dll
$ mono test.exe
$ 1/3/2017 11:30:57 PM 3.14159265359

Files:

// test.cs
namespace TEST
{
   using System;
   using System.IO;
   using Python.Runtime;
   class TEST
   {
       public static void Main(string[] args)
       {
           Environment.SetEnvironmentVariable("PYTHONPATH", Environment.CurrentDirectory);
           PythonEngine.Initialize();

           try
           {
               using (Py.GIL())
               {
                   dynamic clrModule = Py.Import("pytest");
                   clrModule.print_now();
               }
           }
           finally
           {
               PythonEngine.Shutdown();
           }
       }
   }
}

# pytest.py
import clr
import numpy as np
clr.AddReference("System")
import System

def print_now():
    print System.DateTime.Now, np.pi

@AlexCatarino

@AlexCatarino this is good finding! would you mind putting a pull request for this?

My "fix" is basically to undo #300. It would break what #300 fixes...
I am thinking that putting a compilation flag could be a proper fix.

I thought that you fixed both embedding and extending pythonnet? What compilation flag do you mean?

…

On Thu, Jan 5, 2017, 7:21 PM Alexandre Catarino ***@***.***> wrote: My "fix" is basically to undo #300 <#300>. It would break what #300 <#300> fixes... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#303 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgZ5RWW1vxZ9qSDNnR8bj8OAp3ZlSmHks5rPZcLgaJpZM4LHQPR> .

@AlexCatarino

@AlexCatarino @jaredbroad I can confirm your problem and this is also affecting our testing on Travis CI. Let me look for solutions. The worst we may have to do is to recompile CPython with the correct flags or find Python dsitribution that uses these settings.

@AlexCatarino

@AlexCatarino @jaredbroad I tried Miniconda 3.5 64bit on Linux and it works in both directions - embedding and extending pythonnet!

Can you use Miniconda or Anaconda in your docker images?

@denfromufa I think this item from the old TODO document might be related. I don't know if it helps.

Let's talk to the Debian and Ubuntu maintainers for Python in order to convince
them to build Python with --enable-shared. Ubuntu's Python is semi static and not
linked against libpython2.x.so. This causes trouble with clr.so.

@vmuriart

so @vmuriart found a solution to this problem that works with system python, no miniconda necessary:

#245 (comment)

@AlexCatarino

@AlexCatarino were you able to resolve your issue? There's a couple docker images/recipes on https://hub.docker.com/r/pythonnet/pythonnet/tags/

We have resolved our issue and will keep an eye on those images/recipes.
Thank you very much!

Hello gents, I need help with python3.6...

Thank you so much for sharing the conversation. I am trying to run this with Miniconda 3.6 but I get the following error:
Unhandled Exception: System.Exception: dlsym: dotnet: undefined symbol: _PyObject_NextNotImplemented
at Python.Runtime.NativeMethods.GetProcAddress(IntPtr dllHandle, String name) in runtime.cs:line 69
at Python.Runtime.Runtime.Initialize() in runtime.cs:line 322
at Python.Runtime.PythonEngine.Initialize(IEnumerable`1 args, Boolean setSysArgv) in pythonengine.cs:line 168
at Python.Runtime.Py.GIL() in pythonengine.cs:line 547

Building the docker:
RUN apt-get -qq update && apt-get -qq -y install curl bzip2
#&& curl -sSL https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -o /tmp/miniconda.sh
&& bash /tmp/miniconda.sh -bfp /usr/local
&& rm -rf /tmp/miniconda.sh
&& conda install -y python=3
&& conda update conda
&& apt-get -qq -y remove curl bzip2
&& apt-get -qq -y autoremove
&& apt-get autoclean
&& rm -rf /var/lib/apt/lists/* /var/log/dpkg.log
&& conda clean --all --yes

RUN ln -s /lib/x86_64-linux-gnu/libdl.so.2 libdl.so
RUN ln -s /usr/local/lib/libpython3.6m.so .

python -V works and give me 3.6.4 as expected.

PS: Miniconda 2.7 doesn't work either....
what does work is:
apt-get update
apt-get install -y libglib2.0-dev python-all-dev python-pip git
pip install --upgrade pip && pip install pandas scipy numpy

Any ideas of how to make work? I need anything that makes python3.6 work...
THANKS

Ok, I found a clue, the problem seems to be specific to how Anaconda is installed on a Debian/Stretch Docker.
I get this to work with the python coming from debian packages. Even python 3.5 works when using the apt-get python3 command.

Anaconda doesn't work for me neither 2 or 3.....

Navigation Menu

Repository navigation

Using pythonnet in docker container #303

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Using pythonnet in docker container #303

Description

Activity

den-run-ai commented on Dec 7, 2016

AlexCatarino commented on Dec 7, 2016

den-run-ai commented on Dec 8, 2016

den-run-ai commented on Dec 8, 2016

AlexCatarino commented on Dec 8, 2016

jaredbroad commented on Dec 20, 2016

den-run-ai commented on Dec 21, 2016

AlexCatarino commented on Jan 4, 2017

den-run-ai commented on Jan 5, 2017

AlexCatarino commented on Jan 6, 2017

den-run-ai commented on Jan 6, 2017

den-run-ai commented on Feb 5, 2017

den-run-ai commented on Feb 5, 2017

vmuriart commented on Feb 5, 2017

den-run-ai commented on Feb 27, 2017

vmuriart commented on Mar 3, 2017

AlexCatarino commented on Mar 6, 2017

CoFlows commented on Jan 13, 2018

CoFlows commented on Jan 14, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions