Officially support Python 3.9

Environment

Pythonnet version: latest
Python version: python 3.9
Operating System: Windows
.NET Runtime:

Details

Describe what you were trying to get done.

My project EddingtonGUI is dependant on Pythonnet. I wanted for quite some time to update it to use Python 3.9 as default, but I noticed that you don't support it yet.

When are you planning on officially supporting Python 3.9? What prevents you from currently supporting it?
I'm asking so I could update my release scheduling accordingly.

Thank you!

The latest release 2.5 already supports Python 3.9 see [2.5.2] (https://github.com/pythonnet/pythonnet/releases/tag/v2.5.2). Version 3.0.0 is still in development.

Ye, but when I took a look at PyPi, Python3.9 is not written as supported.
I guess it's just a release error :)

After backporting the Python 3.9 support from master to the 2.5 branch, I noticed crashes in CI (I don't remember the details right now). That's why, even though the support is nominally there, I didn't publish the respective wheels and didn't update the trove specifier. You are welcome to give debugging this a go, development happens on master, though, we haven't done any release planning for 3.0 yet.

@filmor

@filmor considering our current master also has crashes on 3.9, random crashes in v2.5 on Python 3.9 could be related.

We should consider 3.9 support in v2.5 as-is, e.g. support installation, but not necessarily fix all crashes.

@lostmsu

@lostmsu, is there a recipe for reproducing the current master 3.9 crashes?

They happen in CI occasionally in both Ubuntu and MacOS with the normal shutdown mode.
Here's a sample failure: https://github.com/pythonnet/pythonnet/runs/1944796763
The CI script is this: https://github.com/pythonnet/pythonnet/blob/master/.github/workflows/main.yml

Weird, dotnet test --runtime any-${{ matrix.platform }} src/embed_tests/ crashed with a "Fatal Python error: PyGILState_Release: auto-releasing thread-state, but no thread-state for this thread" error. That error (plus the fact that it only happens occasionally) sounds like a race condition, but it looks like it was running PythonEngine.RunSimpleString("import 1234"), which is only a single import statement, so I wonder what could be causing a race condition.

Is there a stack trace? Also, which platform?

I haven't tried to reproduce it yet. I was just looking at the CI run you linked to.

The CI crash shows this stack trace:

	0x7fff677da808 - /usr/lib/system/libsystem_c.dylib : abort
	0x109521abd - /Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/libpython3.9.dylib : fatal_error_exit
	0x109521a7a - /Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/libpython3.9.dylib : fatal_error
	0x109526ca4 - /Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/libpython3.9.dylib : _Py_FatalErrorFunc
	0x1095295e5 - /Users/runner/hostedtoolcache/Python/3.9.1/x64/lib/libpython3.9.dylib : PyGILState_Release
	0x10ac5164b - Unknown
	0x10abb1c0c - Unknown
	0x1048bc9a9 - /Library/Frameworks/Mono.framework/Versions/6.12.0/bin/mono : mono_gc_run_finalize
	0x1048dbadc - /Library/Frameworks/Mono.framework/Versions/6.12.0/bin/mono : sgen_gc_invoke_finalizers
	0x1048be6b3 - /Library/Frameworks/Mono.framework/Versions/6.12.0/bin/mono : finalizer_thread
	0x10487cdad - /Library/Frameworks/Mono.framework/Versions/6.12.0/bin/mono : start_wrapper
	0x7fff67914109 - /usr/lib/system/libsystem_pthread.dylib : _pthread_start
	0x7fff6790fb8b - /usr/lib/system/libsystem_pthread.dylib : thread_start

I noticed in pyobject.cs, for example, the the gc finalizer is meant to be suppressed.

        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

Does Py.GILState suppress finalization? If so, what code is calling `PyGILState_Release in a magical gc finalizer thread?!

That's a very good observation. GILState must not have a finalizer doing PyGILState_Release (via ReleaseLock), because you must call PyGILState_Release in the same thread as PyGILState_Ensure. I believe we should throw a fatal exception instead.

https://docs.python.org/3/c-api/init.html#c.PyGILState_Release

allow GIL state debugging; require GILState to be properly disposed

@Jeff17Robbins

@Jeff17Robbins unfortunately, this is not the culprit: https://github.com/pythonnet/pythonnet/pull/1397/checks?check_run_id=1967913124#step:13:144

@lostmsu

@lostmsu I see a completely different stack trace of a crash on Ubuntu and Macos. What's the difference between the "Soft" tests and the "Normal" tests? And...how hard would it be to upgrade to Python3.9.2 to see if it is still a problem with that minor release of CPython?

@Jeff17Robbins

@Jeff17Robbins it's ShutdownMode here.

The issue reproduces with different Python minor versions.

@lostmsu

@lostmsu is there a writeup of ShutdownMode? What does "Soft" soften? Since the crashes seem only to occur due to "Normal" ShutDownMode, it would help to know what the differences are between "Soft" and "Normal"?

@Jeff17Robbins

@Jeff17Robbins soft shutdown mode tries to preserve live CLR objects referenced by Python in order to be able to restore them upon the reinitialization of PythonEngine (the main goal is to support AppDomain reloading).

The code in Runtime.Initialize and Runtime.Shutdown are pretty much the only difference there.

allow GIL state debugging; require GILState to be properly disposed

On my Windows 10 machine, v2.5.2 built by Christoph Gohlke always fails to import in Python 3.9:

Python 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "C:\Tools\Python3\lib\site.py", line 73, in <module>
    import os
  File "C:\Tools\Python3\lib\os.py", line 29, in <module>
    from _collections_abc import _check_methods
  File "C:\Tools\Python3\lib\_collections_abc.py", line 416, in <module>
    class _CallableGenericAlias(GenericAlias):
TypeError: type 'types.GenericAlias' is not an acceptable base type

Hello, one of our internal frameworks at the company I work at depends on pythonnet. But pythonnet not having wheels for Python 3.9 prevents our users from using Python 3.9. We are on Windows 10 platform. Is there anything I can do to help for the Python 3.9 release of pythonnet? I am not familiar with pythonnet or CPython internals but I can maybe try compiling something using Visual Studio and report logs or run a test suit?

@canol

@canol there's now a preview pip package for pythonnet 3.0, that supports Python 3.9 and 3.10

@lostmsu

@lostmsu, the preview pip package for pythonnet 3.0 (I have 3.0.0a2) won't run the demo scripts (helloform.py, splitter.py, or wordpad.py). There are failures with generics and not recognizing 1 and 0 as Boolean. Converting 1 and 10 to True and False helps with helloform.py and splitter.py, but I haven't been able to fix wordpad.py because the error messages are too obscure for me:


Unhandled Exception: Python.Runtime.PythonException: 'int' value cannot be converted to System.Boolean
   at Python.Runtime.PythonException.ThrowLastAsClrException()
   at Python.Runtime.Dispatcher.TrueDispatch(Object[] args)
   at Python.Runtime.Dispatcher.Dispatch(Object[] args)
   at __System_Threading_ThreadStartDispatcher.Invoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Obviously this is a Boolean-related error message but I have no clue where to start looking.
I would humbly request that either a) Boolean and generic related errors be addressed or b) support be added for 2.5.2. Thank you.

Navigation Menu

Repository navigation

Officially support Python 3.9 #1389

Environment

Details

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Officially support Python 3.9 #1389

Description

Environment

Details

Activity

sschukat commented on Feb 19, 2021

saroad2 commented on Feb 19, 2021

filmor commented on Feb 19, 2021

lostmsu commented on Feb 19, 2021

Jeff17Robbins commented on Feb 22, 2021

lostmsu commented on Feb 22, 2021

ppena-LiveData commented on Feb 22, 2021

lostmsu commented on Feb 22, 2021

ppena-LiveData commented on Feb 22, 2021

Jeff17Robbins commented on Feb 24, 2021

lostmsu commented on Feb 24, 2021

lostmsu commented on Feb 24, 2021

Jeff17Robbins commented on Feb 24, 2021

lostmsu commented on Feb 25, 2021

Jeff17Robbins commented on Feb 25, 2021

lostmsu commented on Feb 25, 2021

wdscxsj commented on Mar 29, 2021

canol commented on Jun 14, 2021

lostmsu commented on Jan 4, 2022

burque505 commented on May 29, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions