Repository navigation
Memory Leak in python when using a C# DLL #577
Description
Environment
- Pythonnet version: 2.3.0
- Python version: 3.6.1
- Operating System: Windows 7 Professional 64bits.
- .NET Framework 4.6
Details
- I experience memory leaks when trying to use a C# dll in python to run a simulation and retrieve results using python. When I try to run multiple time the calculation I get leaks
Here is a template of code I run to experience this memory leak (replaced the name of the dll I use by generic names)
import clr
import os
import sys
import psutil
import gc
dll_path = r''the path to my dll"
sys.path.append(os.path.join(dll_path, 'MyDll.dll'))
clr.AddReference(os.path.join(dll_path, "MyDll"))
while psutil.virtual_memory().percent < 80:
print('psutil_virtual_memory:', psutil.virtual_memory().percent)
obj = MyDll.MyObject()
res = obj.Run() # long simulation is run and I get the results.
gc.collect()
clr.System.GC.Collect()Output :
psutil_virtual_memory: 41.8
psutil_virtual_memory: 44.4
psutil_virtual_memory: 46.6
psutil_virtual_memory: 48.9
.....
until my memory reach 80%
The issue is that running the equivalent code in C# does not give any memory leak but it's when calling it from python we get issues of memory leaks. the python garbage collector doesn't help at all with this. I tried also to call the System GC Collect but without success too.
My question is how can I investigate and try to fix this issue.
Activity
den-run-ai commentedon Nov 24, 2017
Please try out the fix according to this issue:
#541
BreitA commentedon Nov 24, 2017
@denfromufa
If I read well the issue #541 it seems you were saying in this issue
My issue is that gc.collect() from python doesn't release my memory.
I saw that you have a branch fixing the memory issue.
Should I try to install the pythonnet version of your branch?
den-run-ai commentedon Nov 24, 2017
Yes, try this branch developed by @dmitriyse:
#532
dmitriyse commentedon Nov 24, 2017
I also created branch https://github.com/pythonnet/pythonnet/tree/featured
With all PRs applied. I would recommend to test this branch if something goes wrong.
BreitA commentedon Nov 24, 2017
got this warning and this error when doing python setup.py bdist_wheel with your branch. Is there any quick fix I can try?
den-run-ai commentedon Nov 25, 2017
den-run-ai commentedon Nov 25, 2017
den-run-ai commentedon Nov 25, 2017
BreitA commentedon Nov 25, 2017
Sorry I didn't restarted my computer after installing VS I must be tired. So now I fixed this issue I got the following error
I added to path the winsdk link to this exe but this change nothing. I am on my home machine so the operating system is now :
Operating System: Windows 10 Home 64bits.
Visual studio and .NET :
Microsoft Visual Studio Community 2017
Version 15.4.4
VisualStudio.15.Release/15.4.4+27004.2009
Microsoft .NET Framework
Version 4.7.02046
Installed Version: Community
when using --xplat option ihave the same error about ildasm.exe after it runs through .NET core
I have 3 ildasm.exe :
C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\x64\ildasm.exe
C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.7 Tools\x64\ildasm.exe
C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\ildasm.exe
it looks like this
#216
I checked the file in question and it seems it already calls the 1.2.7 of unmanagedexports
is there anything I can do to fix this in my environment or is this in the code of the module ?
BreitA commentedon Nov 25, 2017
I manage to install it from the following binary :
https://ci.appveyor.com/project/pythonnet/pythonnet/branch/featured/job/7daumkda8h36k89t/artifacts
this branch didn't fix my memory leak
den-run-ai commentedon Nov 25, 2017
You need to provide minimum reproducible example like I did in the linked issue above.
BreitA commentedon Nov 25, 2017
Ok I'll try to do that but it make take a while for me to identify one leaking element of the C# code library I use because I know very few thing about this code yet or about C# in general to begin with and this code is a simulator using multiple nested structure of C# objects.
Anyway your simple example of instancing a double in a for loop doesn't cause memory leak in my case even on the release 2.3.0 version of pythonnet. I will try to instanciate a simple example of a nested structure of C# object to see if it leaks.
I also used a temporary solution to work with until i fix the issue, I encapsulate my call to the C# library in a multiprocessing.Process.
when the process end the memory leak is cleaned with the process and doesn't propagate to the main process.
using multiprocessing.Pool, the memory leak is shared to the pool meaning if you have a lot of elements in your pool the memory leak will skyrocket pretty fast but when the whole pool end the memory leak is also cleaned
Thanks for your time.
den-run-ai commentedon Nov 26, 2017
You can also use .NET memory profiler such as one of these, but it may not work due to some unmanaged memory:
https://stackoverflow.com/a/100490/2230844
The easiest is commenting out code until the leak is narrowed down to few lines of code.
den-run-ai commentedon Apr 18, 2018
closing due to inactivity and no reproducible example
BreitA commentedon Apr 22, 2018
@denfromufa Since i didn't have access to the C# code i couldn't do what was suggested. Anyway i circumvent the issue migrating my code to linux with mono and dont have memory leak issues there.