Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Python 3.9 #44485

Open
park-junha opened this issue Oct 31, 2020 · 14 comments
Open

Support Python 3.9 #44485

park-junha opened this issue Oct 31, 2020 · 14 comments

Comments

@park-junha
Copy link

@park-junha park-junha commented Oct 31, 2020

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template

System information

  • TensorFlow version (you are using): 2.3.1
  • Are you willing to contribute it (Yes/No): No (willing to do trivial patches)

Describe the feature and the current behavior/state.
TensorFlow should be supported on Python 3.9.

Will this change the current api? How? Significant changes will likely need to be made.

Who will benefit with this feature?
Anyone on Python 3.9

Any Other info.

@mihaimaruseac
Copy link
Collaborator

@mihaimaruseac mihaimaruseac commented Nov 2, 2020

We have not released anything for python3.9.

Release process is as follows:

  1. Ensure all of our dependencies support python3.9 (not yet there)
  2. Ensure our code works with all these dependencies on all supported versions of python (at the moment, at least gast needs additional support work).
  3. Release nightly version supporting the new python version
  4. Release a full release.

As we are already doing the 2.4 release, python3.9 support for TF will come in 2.5 release at the earliest.

@ravikyram, @amahendrakar, @rmothukuru let's deduplicate to this issue all py39 requests/issues. I'll monitor this one and provide updates as we start adding support.

sesheta pushed a commit to thoth-station/adviser that referenced this issue Nov 3, 2020
…#1528)

* Introduce a sieve for filtering out incompatible TensorFlow for Py3.9

* Python 3.9 support will come in 2.5 release at the earliest

tensorflow/tensorflow#44485 (comment)
@rmothukuru
Copy link
Contributor

@rmothukuru rmothukuru commented Nov 17, 2020

@mihaimaruseac
Copy link
Collaborator

@mihaimaruseac mihaimaruseac commented Nov 18, 2020

I may be wrong, but I believe this is the case:

python3.8 -m pipdeptree -p tensorflow | grep "^  -" | sed "s/  - //" | sed "s/ \[.*]//" | xargs python3.9 -m pip install

which is essentially

python3.9 -m pip install absl-py astunparse flatbuffers gast google-pasta grpcio h5py keras-preprocessing numpy opt-einsum protobuf six tensorboard tensorflow-estimator termcolor typing-extensions wheel wrapt

works fine for me. Am I missing anything?

(from #40840 (comment))

The main issue is that we also have C++ code and we need the needed dependencies to also upgrade for their C++ code. By this time, likely most are solved, so 2.5 release will have py3.9 support.

We should get py3.9 in nightly soon.

@mihaimaruseac
Copy link
Collaborator

@mihaimaruseac mihaimaruseac commented Nov 25, 2020

So, all of our depdencies have a release for py3.9 but our setup.py upper bounds to versions which are below that release. We will need to increase these bounds while making sure we don't break existing usecases/tests. Will slowly happen over December.

@willemavjc
Copy link

@willemavjc willemavjc commented Dec 11, 2020

So, all of our depdencies have a release for py3.9 but our setup.py upper bounds to versions which are below that release. We will need to increase these bounds while making sure we don't break existing usecases/tests. Will slowly happen over December.

@mihaimaruseac Any update? (Quite concerned here as we did an upgrade to 3.9 for some various reasons such Python's core enhancements but totally zapped out the TensorFlow's upper bound T_T)

Oh, and why limiting on upper bound nightly releases if they are for testing?
I tried pip install tf-nightly 2.5.0.dev20201211 but it ended up as the stable version:

ERROR: Could not find a version that satisfies the requirement tf-nightly
ERROR: No matching distribution found for tf-nightly
@alanpurple
Copy link
Contributor

@alanpurple alanpurple commented Dec 27, 2020

I've built tf 2.4 for python 3.9.1 on windows 10 and ubuntu 18.04 and they work fine, no problem at all

@willemavjc
Copy link

@willemavjc willemavjc commented Dec 27, 2020

I've built tf 2.4 for python 3.9.1 on windows 10 and ubuntu 18.04 and they work fine, no problem at all

Curious to know how you did that since you cannot use it because of this “stupid” dependency check. Oh you mean you built it from source? If that’s so, would you mind sharing the commands/options you used?

thanks!

@ahtik
Copy link
Contributor

@ahtik ahtik commented Dec 27, 2020

@willemavjc I can share my build from source setup. Win 10 and works with python 3.9.1, TF 2.4.
Python 3.9 installed in c:\python39\ dir.

# Copy https://github.com/bazelbuild/bazel/releases/download/3.7.2/bazel-3.7.2-windows-x86_64.exe to somewhere in PATH and rename to bazel.exe 
# Make sure you have Visual Studio 2019 Build Tools installed (located under C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC)
# Make sure cudnn-8.0.5.39 is unpacked into C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0 respective dirs.
# Ensure your env vars contain something like this:
# CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0
# CUDA_TOOLKIT_PATH=%CUDA_PATH%
# PATH=%CUDA_PATH%\bin;%PATH%
# PATH=%CUDA_PATH%\extras\CUPTI\libx64;%PATH%

# Using PowerShell 7.1.0
git clone https://github.com/tensorflow/tensorflow.git tf24build
cd tf24build
git checkout v2.4.0
\python39\python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -U pip

# Now, depending on how fresh is your Windows installation, you need either numpy==1.19.3 or or 1.19.4.
# See https://github.com/numpy/numpy/issues/16744 for details
pip install six numpy==1.19.4 wheel keras_applications keras_preprocessing --no-deps

$Env:BAZEL_VC = "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC"
# Adjust the Compute Capabilities to match your target GPU!
$Env:TF_CUDA_COMPUTE_CAPABILITIES = "7.5"
$Env:CC_OPT_FLAGS = "/arch:AVX2"
$Env:TF_NEED_CUDA = "1"

# Check the prompts but should be just enter for everything
.\configure.cmd

bazel build --config=opt --define=no_tensorflow_py_deps=true //tensorflow/tools/pip_package:build_pip_package

# Generates tensorflow_2_4_py39.whl
bazel-bin\tensorflow\tools\pip_package\build_pip_package tensorflow_2_4_py39

# Create a new venv and install the new wheel, test it's health with
# python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])));print(tf.config.list_physical_devices('GPU'))" 
@byronyi
Copy link
Contributor

@byronyi byronyi commented Dec 27, 2020

Sorry but AFAIK protobuf does not support Python 3.9 properly with their Linux wheels. See protocolbuffers/protobuf#7978.

@ahtik
Copy link
Contributor

@ahtik ahtik commented Dec 27, 2020

Ah, I think I also left out a few bits here...
Because grpcio version that tensorflow depends on is not having the python 3.9 wheels (but newer version has), my setup to install the wheels is twofold:

# Requirements from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/pip_package/setup.py
pip install "grpcio==1.34.0" "absl-py~=0.10" "astunparse~=1.6.3" "flatbuffers~=1.12.0" "google_pasta~=0.2" "h5py~=3.1.0" "keras_preprocessing~=1.1.2" "numpy~=1.19.2" "opt_einsum~=3.3.0" "protobuf>=3.9.2" "six~=1.15.0" "termcolor~=1.1.0" "typing_extensions~=3.7.4" "wheel~=0.35" "wrapt~=1.12.1" "gast==0.3.3"

pip install --no-deps C:\tmp\tensorflow-2.4.0-cp39-cp39-win_amd64.whl

This makes it easier to also manage the protobuf installation separately. At least for Windows there is an unofficial Python 3.9 protobuf wheel at https://www.lfd.uci.edu/~gohlke/pythonlibs/#protobuf (not the latest but good enough, 3.12.3, ).

@mihaimaruseac
Copy link
Collaborator

@mihaimaruseac mihaimaruseac commented Dec 27, 2020

protobuf and gast are current blockers to have 100% tests passing

@byronyi
Copy link
Contributor

@byronyi byronyi commented Dec 28, 2020

protobuf and gast are current blockers to have 100% tests passing

gast 0.4.0 seems to support Python 3.9. Haven't tested it yet though.

@alanpurple
Copy link
Contributor

@alanpurple alanpurple commented Dec 28, 2020

@willemavjc
only problem blocked me was grpcio 1.32.0, which I ignored using --no-dependency option

I've built tensorflow from source of course

cuda 11.2
cudnn 8.0.5
branch - r2.4

@mihaimaruseac
Copy link
Collaborator

@mihaimaruseac mihaimaruseac commented Dec 28, 2020

protobuf and gast are current blockers to have 100% tests passing

gast 0.4.0 seems to support Python 3.9. Haven't tested it yet though.

It does, but I think @mdanatg has a change to update and there are a few things that need to change to make all tests pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.