Implement CLR loading in pure Python

To simplify the build process (immensely), I'd like to move the CLR loading for Mono and CoreCLR to Python. I have a working prototype at https://gist.github.com/filmor/cf8a02aefff58be09a57ad2eca3cc246 for CoreCLR using CFFI and will work on an equivalent Mono implementation for this as well. For .NET Framework, I don't know of any public API to embed it into the currently running process, so we'll keep the existing loader for this one.

This implements the first section of the Plan(tm) (https://github.com/pythonnet/pythonnet/wiki/Plan#clr-loading).

@spinwards

I tried the loader from your gist in #857 and it caused a hard crash on windows nano server w/ .net core 2.2 and miniconda.

Originally posted by @spinwards in #612 (comment)

I've tested it using docker with the mcr.microsoft.com/dotnet/core/sdk:2.2 image with both miniconda and a vanilla python install and a copy of Python.Runtime.dll that targets .net standard 2. In all cases when I call func(), python immediately crashes with a generic "Windows fatal exception: access violation".

I had to modify the build somewhat to get it to run in VS2019, so it might be a problem on my end.

I don't get this error, it works for me on both Linux and Windows up to the point that it complains about System.Reflection.Emit from Python.NET's CodeGenerator class. I'll update the gist to the latest state.

Sure, with the net40 target platform, but I am running this on nanoserver with .net core2.

I need the ReflectionPolyfill class to adapt the DefineDynamicAssembly method for .net standard. If I remove the net40 target framework and build the resulting assembly gives me an access violation.

Work is ongoing in https://github.com/filmor/clr-loader.

I'm able to load up .NET Core (>=3 due to hostfxr changes) and get a callable object in Python for something that has the signature int Func(IntPtr data, int size). I'll implement the same interface for Mono now and incorporate the clrmodule hack from Python.NET.

We are getting there, Mono is implemented as well now, I'll try to add .NET framework support tonight.

.NET framework support as well as a simple interface are also implemented now. I will create a PR that drops clrmodule in favour of a pure-python clr.py that uses clr_loader and calls a function with the correct signature (int32 (void*, int32)) in Python.NET to run the initialisation.

Hello, I'm interested on the status of this ! Is there some way to use a dotnet core DLL with pythonnet right now ? And on Linux (both x86 and arm), how do we proceed, as we could install mono and/or MS dotnet core sdk: which one is needed ? How to import, AddReferences, use the assemblies ?
Right now I only errors such as the one below:

System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
   at System.Reflection.RuntimeAssembly.GetExportedTypes(RuntimeAssembly assembly, ObjectHandleOnStack retTypes)
   at System.Reflection.RuntimeAssembly.GetExportedTypes()
   at System.Reflection.Assembly.get_ExportedTypes()

@kpoman

Hi, I also tried loading a .NET Core 2.0, 2.1 or 3.1 dll using pythonnet. It failed just like @kpoman 's attempt. So I dug a little deeper and it seems, without knowing any details, that pythonnet is loading "some" Runtime when you call import clr in my case it adds C:\Windows\Microsoft.NET\Framework64\v4.0.30319

So @filmor: Is there a way I can tell pythonnet to use another directory? Because I can build (rather publish) my dotnetcore application as self contained. So in my mind it would make sense to tell pythonnet to just use this directory where I have my application DLLs + the Runtime to load all of it.

Here some print out where it shows that the System.Runtime 4.0.0.0 is loaded from my default installation and the attempt to load a different Runtime afterwards fails (of course)

System.Runtime, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
demo, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.Assembly.GetTypes()
## Could not load file or assembly 'System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e' or one of its dependencies. An attempt was made to load a program with an incorrect format.
System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.Assembly.GetTypes()

This is much more complicated than you make it out to be :). The reason this works in the way it currently does is because of some magic with regard to how Windows deals with .NET DLLs. There is startup code in there that loads mscorlib that starts up the .NET (Framework) runtime. As far as I can tell you can't use the same mechanism to load .NET Core. You can definitely not use it on Linux or OS X where the runtime linker deals with ELF or Mach-O format libraries, respectively.

I have already started work on this, but don't expect anything usable before January. The loader as such already works (see https://github.com/filmor/clr-loader) for all dynamic .NET runtimes (Framework, Core, Mono), now I'm in the process of refactoring Python.NET to make use of this in #1015.

Yeah I thought it might be. I'll check out your loader and look around. Maybe I can help out with something (doubtful but I'll try)

Appreciate all your hard work and brains on this 🙂. What's the answer to the question about self-contained exe/dlls, will those be loadable under the new mechanism such that users won't need .net core 3 installed? Happy to raise a separate issue about it if it needs something different doing

@mungojam

@mungojam Contributions to clr-loader on this topic are welcome, I didn't look into this yet. Essentially, all you need to do is locating the hostfxr library, this is currently done in the constructor of HostFxr in https://github.com/filmor/clr-loader/blob/master/clr_loader/hostfxr.py.

A mild bump, what's the state of this issue? When would it be possible to load a .net core assembly?

@hillin

@hillin Development happens completely in the open. If you don't see any progress on this issue in form of comments or commits, then there is no progress.

@filmor

@filmor have you been working on this lately? What is the status?

I've given this some thought, and now think this should ideally be a separate Python module to be published for pip, that the main repository should depend on.

e.g. pip install hostfxr followed by

import hostfxr

hostfxr.init()
hostfxr.get_delegate()

BTW, the linked notebook uses the "old" way to load CLR. The "new" way is through hostfxr + optional nethost to locate hostfxr.

That's pretty much exaxtly what https://github.com/filmor/clr-loader does, with the added advantage that it also implements loaders for Mono and .Net Framework. It also uses hostfxr instead of the old style and works with .NET 5.

Do you want to migrate it under pythonnet (as a separate repo) and publish a package? Does it need polishing?

All of those things, yes.

I'll move it under the project now. There are plenty of TODOs, in particular CI and testing. We can publish a package once it's actually used in pythonnet.

I think it makes sense to publish package right away as 0.x "alpha", then add CI + testing.

Done: https://pypi.org/project/clr-loader/0.1.0/

Navigation Menu

Repository navigation

Implement CLR loading in pure Python #857

4 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Implement CLR loading in pure Python #857

Description

Activity

spinwards commented on May 8, 2019

spinwards commented on May 8, 2019

filmor commented on May 8, 2019

spinwards commented on May 9, 2019

filmor commented on Oct 27, 2019

filmor commented on Oct 29, 2019

filmor commented on Oct 31, 2019

kpoman commented on Dec 5, 2019

bneumann commented on Dec 18, 2019

filmor commented on Dec 18, 2019

bneumann commented on Dec 18, 2019

mungojam commented on Dec 18, 2019

filmor commented on Dec 18, 2019

4 remaining items

hillin commented on Jun 2, 2020

filmor commented on Jun 9, 2020

lostmsu commented on Oct 30, 2020

lostmsu commented on Nov 20, 2020

filmor commented on Nov 20, 2020

lostmsu commented on Nov 20, 2020

filmor commented on Nov 20, 2020

lostmsu commented on Nov 20, 2020

filmor commented on Nov 20, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions