Update and diversify TensorFlow optional installations.#3292
Conversation
91026e2 to
33126b8
Compare
There was a problem hiding this comment.
Looks nice so far; thanks a lot for handling this! t
Two questions regarding the scope :
- How should we handle documentation?
- Can we have minimal CI testing that covers each of these paths ? What do you think would be the best method there ? Maybe a workflow only on main pushes that installs each and runs one of the lighter TF test script ?
|
yess working at it! good suggestions |
|
For the docs happy to coordinate and add this to my ongoing updates! |
e198b38 to
42b2813
Compare
|
Note: adding some smoke tests in CI, which I will probably move to a separate PR later to keep the discussions separated. But first want to test for compatible versioning. I will also update the docs once finished and coordinate with you @C-Achard & help with the new docs to avoid parallel work. |
285ba05 to
3f71887
Compare
b7e27a1 to
527cf2f
Compare
|
@C-Achard, it's ready for review now. Let me know what you think about the CI changes: for now I added a simple smoke test directly in the intelligent testing workflow, which means we can avoid (downloading and) installing ffmpeg for every run, and also doesn't affect the concurrency with the full-matrix tests. If you think it should make use of the python-package workflow, like the other test lanes, I can see arguments for that as well. |
2a29eb6 to
527cf2f
Compare
There was a problem hiding this comment.
Can we cherry-pick/adapt changes to the latest docs audit updates ?
|
Seems tensorflow-macos is starting to fail in CI, perhaps we should make this high priority |
There was a problem hiding this comment.
Note: this has already been replaced by #3280 and can be safely discarded I think
- revise tf extra - add tf-cu11, tf-cu12, tf-latest This brings in line the CI testing with the actual extras provided, simplifies the installation, and allows for targeted installs for different users.
e5cea11 to
8c880cd
Compare
Xet is Hugging Face’s new storage backend designed to replace the aging Git LFS (Large File Storage). While Git LFS treats every file as a single "blob," Xet breaks files into small "chunks." In restricted CI environments (like CircleCI, GCP, or Kaggle), the Xet "chunked" download often stalls. Because Xet opens many simultaneous connections to different data chunks, CI firewalls or NAT gateways sometimes flag this as suspicious activity or simply drop the connections when they hit a certain threshold.
8c880cd to
a9f3d51
Compare
|
Cleaned up this PR and fixed failing tests:
|
Summary
The current tensorflow installation via the optional extra
tfis designed as a one-solution-for-all, instead of providing a tailored fit for multiple users with different machines. Also, there is divergence between the recommendations in the docs, CI and the current pyproject.toml. This PR implements a more diverse set of tensorflow installations, and aligns the CI so that these versions are actually tested. The documentation is also updated accordingly.Addresses #3271
Optional dependencies
The new configuration is as follows:
tf: tensorflow>=2.12,<2.18 with all related dependencies -> recommended range that fits the documented last-supported versions and was tested on cuda 11.8 and cuda 12.0. Takes into account the installed python version.tf-cu11tensorflow==2.14 with all related dependencies -> for cuda 11.x machines.tf-cu12tensorflow==2.18 with all related dependences -> for cuda 12.x machinestf-latesttensorflow>=2.18): 'best effort' support for users that prefer the latest installation (not tested).CI
tfextra.tests/test_tf_install_smokeintelligent-testing.yamltesting a matrix of os<>python<>extras. The job is purposely implemented in parallel to the existing lanes (e.g. full testing lane), as it doesn't need downloading ffmpeg etc.DocumentationThe documentation changes are now moved to a separate PR #3317.