Repository navigation
.NET Core support and CoreCLR embedding - cross-platform API #96
Description
Looks like with CoreCLR the unmanaged exports "hack" for embedding CLR can be extended by cross-platform hosting API:
https://github.com/dotnet/coreclr/issues/1256
This would allow simple C/C++ code to embed CLR
Current state of .NET Core support:
Activity
den-run-ai commentedon Sep 6, 2015
Note that for .NET 4+ still have to use unmanaged exports.
den-run-ai commentedon Jul 22, 2016
apparently @lstratman has extensive experience porting Mono to CoreCLR embedding.
lstratman commentedon Jul 25, 2016
Thanks @denfromufa :).
It's funny, I actually had a fork of this repo from a while back with the goal of looking into this. But, since my company has only a single customer that might want to integrate our .NET server application into their Python application on Linux and, even then, only at some point in the future, it's down towards the bottom of my priority list for development unfortunately. However, I'm happy to offer some technical guidance.
I implemented basically this same functionality for Node.js for the Edge.js project and a lot of the code that I wrote there can probably be re-used or easily re-purposed for pythonnet. https://github.com/tjanczuk/edge/blob/master/src/CoreCLREmbedding/coreclrembedding.cpp#L192 contains the code that spins up the .NET Core CLR in any process. Towards the end of the method (https://github.com/tjanczuk/edge/blob/master/src/CoreCLREmbedding/coreclrembedding.cpp#L673), we get pointers to a bunch of .NET methods that serve as the managed end of the Node.js->.NET interop process. These function pointers are then invoked anytime that we need to call from Node.js to .NET. Implementations of those .NET methods are in https://github.com/tjanczuk/edge/blob/master/src/double/Edge.js/dotnetcore/coreclrembedding.cs and are included in the .NET Core application that's run from within Edge.js through a NuGet package that needs to be referenced (see https://github.com/tjanczuk/edge#how-to-specify-additional-clr-assembly-references-in-c-code).
This works well for Edge.js because the interaction between Node.js and .NET is all encapsulated in functions in one language or the other. So to call between the two domains, you're always invoking functions, never creating variables, setting properties, etc. However for pythonnet, we're obviously more "integrated" into the Python syntax, so we can do things like create variables, set properties, etc. You can write more methods in the .NET end of the interop process to facilitate this, or try to stay in unmanaged code as much as possible by calling the .NET Core CLR native methods to do things like create variables and set properties. That would be the more performant, but potentially more difficult, route. This will have to be done through the ICLRRuntimeHost instance that you get back from calling coreclr_initialize(). It should theoretically be possible, but is left as an exercise to the interested reader.
Anyway, that concludes my brain dump, I hope that it helps. Feel free to post questions and I'll do my best to answer them. At some point, I may have some time for actual development as well, but can't offer any promises.
den-run-ai commentedon Sep 30, 2016
@lstratman thank you for very detailed response, there is a question from MS if CoreCLR C-API is good enough, would you mind replying there? MS is considering standardizing C-API for .NET ecosystem.
https://blogs.msdn.microsoft.com/dotnet/2016/09/26/introducing-net-standard/#comment-138395
Are the four methods (Initialize, Shutdown, Create_Delegate, Execute_Assembly), mentioned enough for embedding CoreCLR in pythonnet?
lstratman commentedon Oct 1, 2016
I've replied to that thread that those methods are sufficient. Looking further into the existing pythonnet code, I realized that it does all of the things that I mentioned earlier (create variables, set properties, index arrays, etc.) in managed code anyway, so once the "spinning up CoreCLR" code is integrated from Edge.js, pythonnet should just be able to re-use a lot of its existing codebase.
den-run-ai commentedon Oct 5, 2016
Another thing holding of CoreCLR port is Reflection.Emit used in pythonnet in one place:
https://github.com/dotnet/corefx/issues/4491#issuecomment-189756092
But we could possibly just do conditional compilation for this feature.
den-run-ai commentedon Oct 5, 2016
otherwise try ikvm.reflection.emit or mono.cecil
var1ap commentedon Nov 23, 2016
Hi, I started work toward Python.Net Core edition in this fork https://github.com/var1ap/pythonnet , stay tuned.
den-run-ai commentedon Dec 4, 2016
@var1ap this is great news, do you agree to transition from Zope to MIT license, if your port is successful?
var1ap commentedon Dec 5, 2016
@denfromufa of course agree.
var1ap commentedon Dec 5, 2016
Greate news!
https://github.com/var1ap/pythonnet/commit/2b508fd3fda1f67db6785b03a409b9edd0e67686
I managed to get pythonnet work under dotnet core 1.1(Win x64)
var1ap commentedon Dec 27, 2016
Also pythonnet successfully ported to dotnet core 1.1(Linux x64 Ubuntu 16.04) (in the same fork).
This is just Proof of concept. So somebody should port this project to CoreCLR with good code quality required for production projects.
dmitriyse commentedon Jan 1, 2017
Hi guys! Pythonnet really works under dotnet core >= 1.0(NetStandard1.5). Thank you very much var1ap for your research. I continued to work on porting pythonnet to dotnet core in this repo https://github.com/dmitriyse/pythonnet/tree/coreclr. Also I plan to improve my binary distribution and avoid any dll load magic. Finally we should get good nuget based distribution.
den-run-ai commentedon Feb 22, 2017
@var1ap @dmitriyse can you prepare a pull request with compilation flags for coreclr? much like .NET/Mono flags that we have currently.
34 remaining items
MatthiasKauer commentedon Nov 25, 2018
Hi, I'm trying to test calling from dotnet core into CPython.
If I understand @denfromufa correctly, that should be working.
But how do I set it up?
pip install pythonnetfails with the following message:Installing the source version with
python setup.py build_ext --inplaceor just executingmono tools/nuget/nuget.exeleads to the same error. My interpretation attempt:monoon my Linuxmint 18.2 system could be too old.I also tried installing via nuget but got (for framework netcoreapp2.0)
Questions:
monothrough their apt repository the way to go?pipnot via nuget?den-run-ai commentedon Nov 26, 2018
@MatthiasKauer current dotnet support is listed for master branch, mono should be installed according to official mono installation guide (see also .travis.yml), pip is primary channel for full installation, nuget support is limited right now, e.g. see #165.
MrM40 commentedon Dec 7, 2018
Sorry for my ignorance. I'm trying to get embedding to work. I've installed pythonnet from nuget (2.3.0-py27-dotnet). But this Python.Runtime.dll doesn't seem to be .Net Standard or -Core. Tried to compile the source code \pythonnet-master\src\runtime\Python.Runtime.csproj to Standard or Core but it seem to be a .Net 4.x. Also tried the Python.Runtime.dll in the Wheel package from here., same problem.
Can someone point me in a direction on how to get my hands on a .Net Standard or -Core version of Python.Runtime.dll?
I read a lot of places that the master branch support .Core....but I really don't know what to do with this information beside what I already tried?
den-run-ai commentedon Dec 7, 2018
@MrM40 .net core builds are in this solution:
https://github.com/pythonnet/pythonnet/blob/master/pythonnet.15.sln
But I recommend building from python side by passing the
--xplatflag to setup.py:https://github.com/pythonnet/pythonnet/blob/master/appveyor.yml#L60
You can also grab the builds/wheels from appveyor artifacts.
It is not possible to keep all 3 major runtimes (.NET Framework, .NET Core, and Mono) in the same solution/project.
MrM40 commentedon Dec 8, 2018
Still running against a wall. I only see release/debug to Mono and Win in those projects?
And I assume I need to compile the "Console.15" project, right
By the way, why not target .NET Standard in these projects?
den-run-ai commentedon Dec 9, 2018
@MrM40 see this: https://github.com/pythonnet/pythonnet/blob/master/src/runtime/Python.Runtime.15.csproj#L4
MrM40 commentedon Dec 9, 2018
Thanks :-)
I'm running on Linux(Debian) and now I get this errir:
Start Unhandled Exception: System.DllNotFoundException: Unable to load DLL 'python27': The specified module or one of its dependencies could not be found. (Exception from HRESULT: 0x8007007E) at Python.Runtime.Runtime.Py_IsInitialized() at Python.Runtime.Runtime.Initialize(Boolean initSigs) in D:\Development\~.Net libraries (3.part)\phytonnet\2018-12-09\pythonnet-master\src\runtime\runtime.cs:line 274 at Python.Runtime.PythonEngine.Initialize(IEnumerable1 args, Boolean setSysArgv, Boolean initSigs) in \pythonnet-master\src\runtime\pythonengine.cs:line 169 at Python.Runtime.Py.GIL() in \pythonnet-master\src\runtime\pythonengine.cs:line 619 at pythonnet_w.Program.Main(String[] args) in \pythonnet_test\Program.cs:line 9 AbortedI made two links: python2.7 and python27 in the same folder as Python.Runtime.dll pointing to "/usr/bin/python2.7".
Not I got this error:
Start Unhandled Exception: System.EntryPointNotFoundException: Unable to find an entry point named 'PyUnicodeUCS2_FromUnicode' in DLL 'python27'. at Python.Runtime.Runtime.PyUnicode_FromUnicode(String s, IntPtr size) at Python.Runtime.Runtime.Initialize(Boolean initSigs) in \pythonnet-master\src\runtime\runtime.cs:line 337 at Python.Runtime.PythonEngine.Initialize(IEnumerable 1 args, Boolean setSysArgv, Boolean initSigs) in pythonnet-master\src\runtime\pythonengine.cs:line 169 at Python.Runtime.Py.GIL() in \pythonnet-master\src\runtime\pythonengine.cs:line 619 at pythonnet_w.Program.Main(String[] args) in \pythonnet_test\Program.cs:line 9 Abortedden-run-ai commentedon Dec 9, 2018
@MrM40 please don't sidetrack this .NET Core (CoreCLR) status with other issues. In a separate issue, document how you are building and how you are using pythonnet to make a reproducible case. Also search the issue tracker for duplicate issues before posting.
emilmuller commentedon Mar 12, 2019
I built https://github.com/pythonnet/pythonnet/blob/master/pythonnet.15.sln with the following compilation symbols:
XPLAT;;PYTHON3;PYTHON37;UCS4;NETSTANDARD;Running this in a .NET Core application on Windows 8.1 causes the program to fatally crash:
The program '[11180] dotnet.exe' has exited with code -1073740791 (0xc0000409).Without any more information. I'm sure more information can be obtained, but I'm unsure how.
Kind regards, Emil
dmitriyse commentedon Mar 12, 2019
Please try UCS2
emilmuller commentedon Mar 12, 2019
@dmitriyse Thanks! Changed to UCS2, now getting:
The program '[12304] dotnet.exe' has exited with code 255 (0xff).dmitriyse commentedon Mar 12, 2019
Unfortunately, I haven't tested this solution widely. Win 8.1 setup hasn't been tested. So you need to dive deeply with VS debugging.
emilmuller commentedon Mar 12, 2019
@dmitriyse Hah! I got it working :) I had this line hanging around:
Removing it fixed the issue. Even though I think the path is correct.
filmor commentedon Nov 8, 2019
Closed in favour of #984.