virtual environment: either crashes or cannot find modules #1478

Activity

1. Felk commented on Jun 26, 2021

   

   Felk

   on Jun 26, 2021

   Last edited by Felk

   Author

   More actions

   Some more random tinkering led me to a solution. The following order of operations works, deviating from the wiki:

   1. Set PythonEngine.PythonPath
   2. Set PythonEngine.PythonHome
   3. PythonEngine.Initialize();

   Doing things in this order does not crash and also causes the new python path to be visible in sys.path. My complete example looks something like this:

   Runtime.PythonDLL = "C:/Program Files/Python39/python39.dll";
   string pathToVirtualEnv = "C:/Users/felk/venv39";
   
   string path = Environment.GetEnvironmentVariable("PATH")!.TrimEnd(Path.PathSeparator);
   path = string.IsNullOrEmpty(path) ? pathToVirtualEnv : path + Path.PathSeparator + pathToVirtualEnv;
   Environment.SetEnvironmentVariable("PATH", path, EnvironmentVariableTarget.Process);
   Environment.SetEnvironmentVariable("PYTHONHOME", pathToVirtualEnv, EnvironmentVariableTarget.Process);
   Environment.SetEnvironmentVariable("PYTHONPATH",
       $"{pathToVirtualEnv}/Lib/site-packages{Path.PathSeparator}" + 
       $"{pathToVirtualEnv}/Lib{Path.PathSeparator}", EnvironmentVariableTarget.Process);
   
   PythonEngine.PythonPath = PythonEngine.PythonPath + Path.PathSeparator +
                             Environment.GetEnvironmentVariable("PYTHONPATH", EnvironmentVariableTarget.Process);
   PythonEngine.PythonHome = pathToVirtualEnv;
   
   PythonEngine.Initialize();
   
   dynamic sys = Py.Import("sys");
   Console.WriteLine(sys.path);
   Console.WriteLine(PythonEngine.PythonPath);
   
   PythonEngine.BeginAllowThreads();
   // ... any further code wrapped in using(Py.GIL()) { ... }

   console output:

   ['C:\\Program Files\\Python39\\python39.zip', 'C:\\Program Files\\Python39\\Lib', 'C:\\Program Files\\Python39\\DLLs', 'S:\\projects\\myproject\\bin\\Debug\\net5.0', 'C:\\Users\\felk\\venv3
   9\\Lib\\site-packages', 'C:\\Users\\felk\\venv39\\Lib', 'C:\\Users\\felk\\AppData\\Roaming\\Python\\Python39\\site-packages', 'C:\\Program Files\\dotnet\\shared\\Microsoft.NETCore.App\\5.0.7\\']
   C:\Program Files\Python39\python39.zip;C:\Program Files\Python39\Lib\;C:\Program Files\Python39\DLLs\;S:\projects\myproject\bin\Debug\net5.0;C:/Users/felk/venv39/Lib/site-packages;C:/Users/fe
   lk/venv39/Lib;
   

   This seems to work for me, but I'll leave this issue open for now in case anyone wants to adjust the wiki.
2. bpdavis86 commented on Aug 12, 2021

   

   bpdavis86

   on Aug 12, 2021

   More actions

   I am appending my own troubles with virtual environments to this open issue rather than create a new one.

   Environment

   • Pythonnet version: Self-built from commit ec65efe 8/11/2021
   • Python version: 3.8.10
   • Operating System: Windows 10 Pro 20H2 19042.1110
   • .NET Runtime: .NET Core 3.1

   Issue

   I spent better part of 2 days trying to get virtual environments working with pythonnet. Part of the problem is that there are a number of various methods to create virtual environments and they may end up creating slightly different "flavored" environments. For example, see:

   https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe

   This does not even cover conda environments which are totally separate.

   In order to better understand how virtual environments can be made to work with pythonnet, you have to dig into the details of how Python sets up sys.path and how this behaves with virtual environments, in particular, PEP 405 "lightweight" environments such as those created with the venv module.

   Key Reference Material:

   • https://stackoverflow.com/questions/897792/where-is-pythons-sys-path-initialized-from
   • https://www.python.org/dev/peps/pep-0405/
   • https://docs.python.org/3/library/site.html

   If one reads through these documents, one finds that all things begin with sys.executable. Python walks up the path looking for its system libraries based on this location except in the case where it finds the file "pyvenv.cfg" one level above the executable. This is the extension provided by PEP 405 to enable lightweight virtual environments. In this case, the pyvenv.cfg "home" key tells Python where the base install is.

   This already presents a problem, because let's see how sys.executable gets set from a C# program using pythonnet to embed Python. Here is a test program to expose some details of the resulting Python environment.

   using System;
   using System.Collections.Generic;
   using Python.Runtime;
   namespace TestPythonnet
   {
       class Program
       {
           static void Main(string[] args)
           {
               // using my base python install, not a venv            
               var pathToBaseEnv = @"C:\Users\myuser\AppData\Local\Programs\Python\Python38";
               Environment.SetEnvironmentVariable("PYTHONHOME", pathToBaseEnv, EnvironmentVariableTarget.Process);
               Runtime.PythonDLL = pathToBaseEnv + @"\python38.dll";
   
               PythonEngine.Initialize();
               using (Py.GIL())
               {
                   dynamic sys = Py.Import("sys");
                   dynamic os = Py.Import("os");
                   Console.WriteLine($"PYTHONHOME: {os.getenv("PYTHONHOME")}");
                   Console.WriteLine($"PYTHONPATH: {os.getenv("PYTHONPATH")}");
                   Console.WriteLine($"sys.executable: {sys.executable}");
                   Console.WriteLine($"sys.prefix: {sys.prefix}");
                   Console.WriteLine($"sys.base_prefix: {sys.base_prefix}");
                   Console.WriteLine($"sys.exec_prefix: {sys.exec_prefix}");
                   Console.WriteLine($"sys.base_exec_prefix: {sys.base_exec_prefix}");
                   Console.WriteLine("sys.path:");
                   foreach (var p in sys.path)
                   {
                       Console.WriteLine(p);
                   }
                   Console.WriteLine();
               }
   
               PythonEngine.Shutdown();
   			
           }
       }
   }

   PYTHONHOME:
   PYTHONPATH: C:\Users\myuser\OneDrive\Documents\Python Scripts
   sys.executable: C:\Users\myuser\source\repos\TestPythonnet\TestPythonnet\bin\Debug\netcoreapp3.1\TestPythonnet.exe
   sys.prefix: C:\Users\myuser\AppData\Local\Programs\Python\Python38
   sys.base_prefix: C:\Users\myuser\AppData\Local\Programs\Python\Python38
   sys.exec_prefix: C:\Users\myuser\AppData\Local\Programs\Python\Python38
   sys.base_exec_prefix: C:\Users\myuser\AppData\Local\Programs\Python\Python38
   sys.path:
   C:\Users\myuser\OneDrive\Documents\Python Scripts
   C:\Users\myuser\AppData\Local\Programs\Python\Python38\python38.zip
   C:\Users\myuser\AppData\Local\Programs\Python\Python38\Lib
   C:\Users\myuser\AppData\Local\Programs\Python\Python38\DLLs
   C:\Users\myuser\source\repos\TestPythonnet\TestPythonnet\bin\Debug\netcoreapp3.1
   C:\Users\myuser\AppData\Local\Programs\Python\Python38
   C:\Users\myuser\AppData\Local\Programs\Python\Python38\lib\site-packages
   C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.1.18\
   
   

   There are two major points to notice here:

   1. The setting of the PYTHONHOME environment variable did nothing for the environment variables inside Python.
   2. The executable detected is the embedding .NET program rather than the Python interpreter.

   The first issue may be related to this post regarding SetEnvironmentVariable and setenv/getenv. I have not confirmed.
   https://stackoverflow.com/questions/4788398/changes-via-setenvironmentvariable-do-not-take-effect-in-library-that-uses-geten

   Regardless, it seems that whatever environment settings one makes in code do not end up in the Python environment variables. Therefore, the environment manipulation in the example code is a red herring, except for the PATH variable, which would allow one to set the Python DLL name without a full absolute path if the PATH includes the Python base folder (with pythonXX.dll).

   One should instead do such environment manipulations through the Run/Test environment, i.e. Visual Studio project setup for the debugging environment.

   Regarding the second issue, one may well ask how this example succeeded if Python does not see the PYTHONHOME variable and the system libraries are not to be found along the path to sys.executable. Examining the Stack Overflow we find:

   If it can't find these "landmark" files or sys.prefix hasn't been found yet, then python sets sys.prefix to a "fallback" value. Linux and Mac, for example, use pre-compiled defaults as the values of sys.prefix and sys.exec_prefix. Windows waits until sys.path is fully figured out to set a fallback value for sys.prefix.

   For sys.path

   If on Windows and no applocal = true was set in pyvenv.cfg, then the contents of the subkeys of the registry key HK_CURRENT_USER\Software\Python\PythonCore<DLLVersion>\PythonPath\ are added, if any.

   On Mac and Linux, the value of sys.exec_prefix is added. On Windows, the directory which was used (or would have been used) to search dynamically for sys.prefix is added.

   Finally

   At this stage on Windows, if no prefix was found, then python will try to determine it by searching all the directories in sys.path for the landmark files, as it tried to do with the directory of sys.executable previously, until it finds something. If it doesn't, sys.prefix is left blank.

   Examining my registry shows that HKEY_CURRENT_USER\SOFTWARE\Python\PythonCore\3.8\PythonPath contains

   C:\Users\myuser\AppData\Local\Programs\Python\Python38\Lib\;C:\Users\myuser\AppData\Local\Programs\Python\Python38\DLLs\
   

   So it seems Windows was able to resolve the path issue using the registry and then applies this path to sys.prefix despite it having no resemblance to sys.executable.

   What about the rest of the sys.path entries?

   • C:\Users\myuser\OneDrive\Documents\Python Scripts comes from PYTHONPATH (already in my user environment)

   • C:\Users\myuser\source\repos\TestPythonnet\TestPythonnet\bin\Debug\netcoreapp3.1 comes from rule 10 in the Stack Overflow regarding "the directory which was used (or would have been used) to search dynamically for sys.prefix is added"

   • C:\Users\myuser\AppData\Local\Programs\Python\Python38 and
     C:\Users\myuser\AppData\Local\Programs\Python\Python38\lib\site-packages are the result of the "site" package run (see https://docs.python.org/3/library/site.html).

   • Finally, C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.1.18\ is added by pythonnet itself in Python.Runtime.Runtime.Initialize

   // Need to add the runtime directory to sys.path so that we
   // can find built-in assemblies like System.Data, et. al.
   string rtdir = RuntimeEnvironment.GetRuntimeDirectory();
   IntPtr path = PySys_GetObject("path").DangerousGetAddress();
   IntPtr item = PyString_FromString(rtdir);
   if (PySequence_Contains(path, item) == 0)
   {
       PyList_Append(new BorrowedReference(path), new BorrowedReference(item));
   }
   XDecref(item);
   AssemblyManager.UpdatePath();

   So, now that we've figured out what happened, how do we fix this to work with virtual environments properly, specifically PEP 405 "lightweight" environments? Well, unfortunately, it seems we can't replicate the setup exactly without lots of manual intervention. A PEP 405 environment looks something like this (on my system):

   C:\Users\myuser\py38
       Include
           <empty>
       Lib
           site-packages
               <all your installed packages for the environment>
       Scripts
           activate
           activate.bat
           Activate.ps1
           ...
           pip.exe
           pip3.8.exe
           pip3.exe
           python.exe
           pythonw.exe
           <and others depending on what packages you've installed>
       share
       pyvenv.cfg
   

   What is not included is pythonXX.dll, which is exactly the file needed by pythonnet to function.

   In "normal" operations, I would put C:\Users\myuser\py38\Scripts first in my path (using the activation script), which would launch Python using the python.exe contained there. Therefore, sys.executable would point to C:\Users\myuser\py38\Scripts\python.exe, Python would "walk back" through the filesystem and find pyvenv.cfg, which contains home = C:\Users\myuser\AppData\Local\Programs\Python\Python38. Therefore, we get this environment

   # From PYTHONPATH
   C:\Users\myuser\OneDrive\Documents\Python Scripts
   # From base environment
   C:\Users\myuser\AppData\Local\Programs\Python\Python38\python38.zip
   C:\Users\myuser\AppData\Local\Programs\Python\Python38\DLLs
   C:\Users\myuser\AppData\Local\Programs\Python\Python38\lib
   C:\Users\myuser\AppData\Local\Programs\Python\Python38
   # From venv site-packages
   C:\Users\myuser\py38
   C:\Users\myuser\py38\lib\site-packages
   C:\Users\myuser\py38\lib\site-packages\win32
   C:\Users\myuser\py38\lib\site-packages\win32\lib
   C:\Users\myuser\py38\lib\site-packages\Pythonwin
   

   Why don't we just set PYTHONHOME to the venv folder and be done with it? Unfortunately, this doesn't work because PYTHONHOME needs to point to the root of the Python install with the system libraries, which this lightweight venv does not have. (It seems PYTHONHOME is explicitly incompatible with venv because it is unset/cached in the activate script).

   Proposed Solution

   So, in order to replicate the venv setup, we need to follow a rather messy manual process.

   In the following steps, all environment variables should be set outside of code (i.e. in your shell or Visual Studio project setup) in order to avoid the problem mentioned before where values do not propagate down into Python after being changed in C#.

   1. Get pythonnet to be able to find your python3X.dll. This can be done in a number of ways.

      • Set PYTHONNET_PYDLL environment variable to point directly to the DLL (absolute path).
      • Add your base install Python home folder (containing python3X.dll) to the top of PATH, then set PYTHONNET_PYDLL to python3X.dll.
      • Add your base install Python home folder (containing python3X.dll) to the top of PATH, then set PYTHONNET_PYVER to 3.X.
      • Set Runtime.PythonDLL to point directly to the DLL (absolute path) in code.
      • Add your base install Python home folder (containing python3X.dll) to the top of PATH, then set Runtime.PythonDLL to python3X.dll in code.

   2. Set your Python home explicitly. This will avoid any shenanigans regarding autodetection of the base environment when sys.executable is not python.exe. Again, this can be done in a couple ways.

      • Set PYTHONHOME environment to your base install Python home folder.
      • Set PythonEngine.PythonHome your base install Python home folder in code.

   3. Specify the virtual environment path somehow. You could invent an environment variable for this, or supply directly in code.

   var pathToVirtualEnv = Environment.GetEnvironmentVariable("PYTHONNET_PYVENV"); // I invented this new enviroment variable
   // or
   var pathToVirtualEnv = @'C:\Users\myuser\py38'; // path to my virtualenv

   4. If the virtualenvironment is enabled, enable the "no site" flag on the Python interpreter. Normally one does this by calling the interpreter with the -S option. Pythonnet provides a way to set this flag through PythonEngine.SetNoSiteFlag. Unfortunately this has a bug currently associated with it on Windows which is documented in this Github issue. To workaround the bug, one must ensure to access some part of the PythonEngine API to load up the DLL before calling PythonEngine.SetNoSiteFlag. However, this API access must not cause the interpreter to initialize. (If you set PythonEngine.PythonHome in the previous step, this will suffice.) For example:

   // Access the API so the DLL gets loaded
   if (!String.IsNullOrEmpty(pathToVirtualEnv))
   {
       // Access the API so the DLL gets loaded
       string version = PythonEngine.Version;
       // Now we can set the flag and have it stick
       PythonEngine.SetNoSiteFlag();
   }

   5. Load the interpreter and run the site package manually with modified settings.

   PythonEngine.Initialize();
   using (Py.GIL())
   {
       if (!String.IsNullOrEmpty(pathToVirtualEnv))
       {
           // fix the prefixes to point to our venv
           // (This is for Windows, there may be some difference with sys.exec_prefix on other platforms)
           dynamic sys = Py.Import("sys");
           sys.prefix = pathToVirtualEnv;
           sys.exec_prefix = pathToVirtualEnv;
   
           dynamic site = Py.Import("site");
           // This has to be overwritten because site module may already have been loaded by the interpreter (but not run yet)
           site.PREFIXES = new List<PyObject> { sys.prefix, sys.exec_prefix };
           // Run site path modification with tweaked prefixes
           site.main();
       }
   
   }

   Here is a full code example. I used as much setup in the external environment as possible (because hardcoding paths into software is not a great way to go.)

   Environment variables (from launchSettings.json):

   {
     "profiles": {
       "TestPythonnet": {
         "commandName": "Project",
         "environmentVariables": {
           "PYTHONNET_PYVENV": "C:\\Users\\myuser\\py38",
           "PYTHONNET_PYVER": "3.8",
           "PYTHONHOME": "C:\\Users\\myuser\\AppData\\Local\\Programs\\Python\\Python38",
           "PATH": "C:\\Users\\myuser\\AppData\\Local\\Programs\\Python\\Python38; %PATH%"
         }
       }
     }
   }

   Program:

   using System;
   using System.Collections.Generic;
   using Python.Runtime;
   
   
   namespace TestPythonnet
   {
       class Program
       {
           static void Main(string[] args)
           {
   
               var pathToVirtualEnv = Environment.GetEnvironmentVariable("PYTHONNET_PYVENV");
   
               if (!String.IsNullOrEmpty(pathToVirtualEnv))
               {
                   // Access the API so the DLL gets loaded
                   string version = PythonEngine.Version;
                   // Now we can set the flag and have it stick
                   PythonEngine.SetNoSiteFlag();
               }
   
   
               PythonEngine.Initialize();
               using (Py.GIL())
               {
                   if (!String.IsNullOrEmpty(pathToVirtualEnv))
                   {
                       // fix the prefixes to point to our venv
                       // (This is for Windows, there may be some difference with sys.exec_prefix on other platforms)
                       dynamic sys = Py.Import("sys");
                       sys.prefix = pathToVirtualEnv;
                       sys.exec_prefix = pathToVirtualEnv;
   
                       dynamic site = Py.Import("site");
                       // This has to be overwritten because site module may already have 
                       // been loaded by the interpreter (but not run yet)
                       site.PREFIXES = new List<PyObject> { sys.prefix, sys.exec_prefix };
                       // Run site path modification with tweaked prefixes
                       site.main();
                   }
   
               }
   
               using (Py.GIL())
               {
                   dynamic sys = Py.Import("sys");
                   dynamic os = Py.Import("os");
                   Console.WriteLine($"PYTHONHOME: {os.getenv("PYTHONHOME")}");
                   Console.WriteLine($"PYTHONPATH: {os.getenv("PYTHONPATH")}");
                   Console.WriteLine($"sys.executable: {sys.executable}");
                   Console.WriteLine($"sys.prefix: {sys.prefix}");
                   Console.WriteLine($"sys.base_prefix: {sys.base_prefix}");
                   Console.WriteLine($"sys.exec_prefix: {sys.exec_prefix}");
                   Console.WriteLine($"sys.base_exec_prefix: {sys.base_exec_prefix}");
                   Console.WriteLine("sys.path:");
                   foreach (var p in sys.path)
                   {
                       Console.WriteLine(p);
                   }
                   Console.WriteLine();
               }
   
               PythonEngine.Shutdown();
           }
       }
   }

   Output:

   PYTHONHOME: C:\Users\myuser\AppData\Local\Programs\Python\Python38
   PYTHONPATH: C:\Users\myuser\OneDrive\Documents\Python Scripts
   sys.executable: C:\Users\myuser\source\repos\TestPythonnet\TestPythonnet\bin\Debug\netcoreapp3.1\TestPythonnet.exe
   sys.prefix: C:\Users\myuser\py38
   sys.base_prefix: C:\Users\myuser\AppData\Local\Programs\Python\Python38
   sys.exec_prefix: C:\Users\myuser\py38
   sys.base_exec_prefix: C:\Users\myuser\AppData\Local\Programs\Python\Python38
   sys.path:
   C:\Users\myuser\OneDrive\Documents\Python Scripts
   C:\Users\myuser\AppData\Local\Programs\Python\Python38\python38.zip
   C:\Users\myuser\AppData\Local\Programs\Python\Python38\DLLs
   C:\Users\myuser\AppData\Local\Programs\Python\Python38\lib
   C:\Users\myuser\source\repos\TestPythonnet\TestPythonnet\bin\Debug\netcoreapp3.1
   C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.1.18
   C:\Users\myuser\py38
   C:\Users\myuser\py38\lib\site-packages
   C:\Users\myuser\py38\lib\site-packages\win32
   C:\Users\myuser\py38\lib\site-packages\win32\lib
   C:\Users\myuser\py38\lib\site-packages\Pythonwin
   

   This is pretty close to the environment generated by running python normally through the venv, modulo the extra .NET folders.

   Proposed Enhancement Request

   Obviously, this is rather ugly and unfortunate. The root cause of the problem is that we cannot control the value of sys.executable so that Python can follow its standard process to load the venv. Fortunately, there is a new config API added to Python 3.8 with PEP 587. This API allows fine-grained control over the startup Python configuration, including sys.executable. Implementing access to this API through pythonnet would allow us to simply set sys.executable to the venv version of python.exe and have everything work from there.

   👍React with 👍18Reacted by Benedikt Reinartz, Felix König, Alex Helms, Dustin Wyatt, Farith, Hiroaki NATSUME, Oleg, Maged Helmy, Russel Webber, Roman Mishchuk and 8 more

   ❤️React with ❤️3Reacted by Trevor Herr, Max Kagamine and bart-soj
3. filmor commented on Aug 12, 2021

   

   filmor

   on Aug 12, 2021

   Member

   More actions

   Thank you for the detailed report. PRs for this are very welcome!

   A few comments:

   • The issue with PYTHONHOME is (I think) this Windows-specific issue: https://bugs.python.org/issue16633. You can instead use Python.Runtime.PythonHome = something to set the value correctly after setting the correct dll location and before running Initialize. This will use the Python API function Py_SetPythonHome which will work independent of the environment variable.
   • In any case it is (IMHO) always your responsibility to find the libpython. You can use (and maybe extend?) something like https://github.com/losttech/WhichPython for this task. This is out of scope for pythonnet.
4. lostmsu commented on Aug 14, 2021

   

   lostmsu

   on Aug 14, 2021

   Member

   More actions

   @Meisterrichter offtopic idea: if you want your .NET app run within a virtualenv, can you maybe launch it from Python executable? E.g. do

   # TODO: setup clr first
   
   import clr
   clr.AddReference('MyApp')
   from MyApp import PythonEntryPoint
   PythonEntryPoint.PublicMainFunction()

   ?
5. igblip commented on Nov 9, 2024

   

   igblip

   on Nov 9, 2024

   More actions

   I did a hack just to get past this problem while doing development in VSCode in a Debian devcontainer on MacOS. My production environment (AWS EC2 with Amazon Linux) will not need this hack, and I don't recommend using it anywhere except getting past the issue during development.

   First, the above post from @bpdavis86 was incredibly helpful and informative.

   In my case, I could not get a venv environment working through pythonnet. I needed a way to add site-packages system-wide, but packages were maintained by apt and not pip.

   Running python3 -m site resulted in:

   sys.path = [
       '/',
       '/usr/lib/python311.zip',
       '/usr/lib/python3.11',
       '/usr/lib/python3.11/lib-dynload',
       '/usr/local/lib/python3.11/dist-packages',
       '/usr/lib/python3/dist-packages',
       '/usr/lib/python3.11/dist-packages',
   ]
   USER_BASE: '/home/vscode/.local' (exists)
   USER_SITE: '/home/vscode/.local/lib/python3.11/site-packages' (doesn't exist)
   ENABLE_USER_SITE: True
   

   I have no idea why the zip file is in the path and have not investigated. It does not exist.

   The USER_SITE value led to the hack. Specifically, I first created a venv, activated it, and installed my required packages. I then created the USER_SITE folder and copied the packages to it. Running my C# app that runs a script via pythonnet worked.

   Again - this is just a hack.