Skip to content

Update and diversify TensorFlow optional installations. #3292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MMathisLab merged 8 commits into from
May 11, 2026
Merged

Update and diversify TensorFlow optional installations. #3292

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f9a9579
update pyproject.toml and uv.lock TF-extras
deruyter92 Apr 22, 2026
db857b6
add basic smoke test for tensorflow installation
deruyter92 Apr 22, 2026
9a78331
update CI: add "extras" field to matrix_json
deruyter92 Apr 22, 2026
77085d7
update CI: add TF install smoke test (runs on full test lane)
deruyter92 Apr 22, 2026
b5662c4
skip functional tests for tensorflow and windows with python >=3.11
deruyter92 Apr 22, 2026
9e7ed58
Update tests that rely on TF: add import-or-skip.
deruyter92 Apr 23, 2026
f7b4e37
Update CI: ensure pip with `dev` dependency groups
deruyter92 May 7, 2026
a9f3d51
CircleCI: disable hugginface xet
deruyter92 May 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 29 additions & 5 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,43 +8,67 @@ jobs:
executor:
name: python/default
tag: '3.10' # or '3.12'
environment:
# HuggingFace: disable xet and use cache directory
# NOTE @deruyter92 2026-05-07: "xet" opens many simultaneous connections
# to different data chunks. Currently doesn't work well with CircleCI.
# See: https://github.com/huggingface/xet-core/issues/800
HF_HUB_DISABLE_XET: 1
HF_HOME: ~/.cache/huggingface

steps:
- checkout

# 1) Restore uv cache
# Restore uv cache
- restore_cache:
name: Restore uv cache
keys:
- v2-uv-pip-{{ checksum "pyproject.toml" }}
- v2-uv-pip-
# 2) Install uv

# Restore HuggingFace weights cache
- restore_cache:
name: Restore Hugging Face cache
keys:
- hf-weights-v1-{{ checksum "pyproject.toml" }}
- hf-weights-v1-

# Install uv
- run:
name: Install uv
command: |
pip install uv

# Install DeepLabCut runtime deps only
- run:
name: Install DeepLabCut runtime deps only
command: |
uv pip install --system -e .

# 3) (Optional) Trim the cache for CI so uploads stay small and fast
# (Optional) Trim the cache for CI so uploads stay small and fast
- run:
name: Prune uv cache for CI
command: uv cache prune --ci || true

# 4) Save the cache for next runs
# Save the uv cache for next runs
- save_cache:
name: Save uv cache
key: v2-uv-pip-{{ checksum "pyproject.toml" }}
paths:
- ~/.cache/uv

# Test DLC
- run:
name: TestDLC
# uv handles the installation of the dependencies
command: python testscript_cli.py

# Save the HF weights cache for next runs
- save_cache:
name: Save huggingface cache
key: hf-weights-v1-{{ checksum "pyproject.toml" }}
paths:
- ~/.cache/huggingface

workflows:
main:
jobs:
Expand Down
63 changes: 53 additions & 10 deletions .github/workflows/intelligent-testing.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ jobs:
with:
concurrency_key: ${{ github.event.pull_request.number && format('pr-{0}', github.event.pull_request.number) || '' }}
matrix_json: >-
{"include":[{"os":"ubuntu-latest","python-version":"3.12"}]}
{"include":[{"os":"ubuntu-latest","python-version":"3.12", "extras": "[tf]"}]}
pytest_paths_json: ${{ needs.intelligent-test-selection.outputs.pytest_paths }}
functional_scripts_json: ${{ needs.intelligent-test-selection.outputs.functional_scripts }}
full_suite: false
Expand All @@ -110,15 +110,58 @@ jobs:
matrix_json: >-
{
"include": [
{"os":"ubuntu-latest","python-version":"3.10"},
{"os":"ubuntu-latest","python-version":"3.11"},
{"os":"ubuntu-latest","python-version":"3.12"},
{"os":"macos-latest","python-version":"3.10"},
{"os":"macos-latest","python-version":"3.11"},
{"os":"macos-latest","python-version":"3.12"},
{"os":"windows-latest","python-version":"3.10"},
{"os":"windows-latest","python-version":"3.11"},
{"os":"windows-latest","python-version":"3.12"}
{"os":"ubuntu-latest","python-version":"3.10", "extras": "[tf]"},
{"os":"ubuntu-latest","python-version":"3.11", "extras": "[tf]"},
{"os":"ubuntu-latest","python-version":"3.12", "extras": "[tf]"},
{"os":"macos-latest","python-version":"3.10", "extras": "[tf]"},
{"os":"macos-latest","python-version":"3.11", "extras": "[tf]"},
{"os":"macos-latest","python-version":"3.12", "extras": "[tf]"},
{"os":"windows-latest","python-version":"3.10", "extras": "[tf]"},
{"os":"windows-latest","python-version":"3.11", "extras": "[tf]"},
{"os":"windows-latest","python-version":"3.12", "extras": "[tf]"}
]
}
full_suite: true

tf-install-smoke-test:
name: TensorFlow install smoke test
needs: intelligent-test-selection
if: needs.intelligent-test-selection.outputs.run_full == 'true'
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ['3.10', '3.11', '3.12']
# Run smoke test on the extras that are not tested in the full matrix tests
extras: ["[tf-cu11]", "[tf-cu12]"]
exclude:
- os: windows-latest
python-version: '3.11'
- os: windows-latest
python-version: '3.12'
- extras: "[tf-cu11]"
python-version: '3.12'

steps:
- name: Checkout code
uses: actions/checkout@v6

- name: Set up Python
uses: conda-incubator/setup-miniconda@v3
with:
channels: conda-forge
channel-priority: strict
python-version: ${{ matrix.python-version }}

- name: Install dependencies
shell: bash -el {0}
run: |
python -m ensurepip --upgrade
python -m pip install --upgrade pip setuptools wheel
python -m pip install dependency-groups
python -m pip install --no-cache-dir -e ".${{ matrix.extras }}" --group dev

- name: Run TensorFlow install smoke test
shell: bash -el {0}
run: |
pytest tests/test_tf_install_smoke.py
48 changes: 14 additions & 34 deletions .github/workflows/python-package.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ on:
description: "JSON matrix definition"
required: true
type: string
default: '{"include":[{"os":"ubuntu-latest","python-version":"3.12"}]}'
default: '{"include":[{"os":"ubuntu-latest","python-version":"3.12","extras": ""}]}'
pytest_paths_json:
description: "JSON array of pytest paths"
required: false
Expand Down Expand Up @@ -96,23 +96,13 @@ jobs:
channel-priority: strict
python-version: ${{ matrix.python-version }}

- name: Install TensorFlow
if: runner.os != 'macOS'
shell: bash -el {0} # Important to enable conda env
run: |
pip install --no-cache-dir tensorflow tensorpack==0.11 tf_slim==1.1.0 tf-keras

- name: Install TensorFlow on macOS
if: runner.os == 'macOS'
shell: bash -el {0}
run: |
pip install --no-cache-dir tensorflow-macos tensorpack==0.11 tf_slim==1.1.0 tf-keras

- name: Install dependencies
shell: bash -el {0}
shell: bash -el {0} # Important to enable conda env
run: |
python -m ensurepip --upgrade
python -m pip install --upgrade pip setuptools wheel
pip install --no-cache-dir -e . --group dev
python -m pip install dependency-groups
python -m pip install --no-cache-dir -e ".${{ matrix.extras }}" --group dev

- name: Install ffmpeg (Linux/macOS)
if: runner.os != 'Windows'
Expand Down Expand Up @@ -221,19 +211,18 @@ jobs:
env:
FULL_SUITE: ${{ inputs.full_suite }}
FUNCTIONAL_SCRIPTS_JSON: ${{ inputs.functional_scripts_json }}
RUNNER_OS: ${{ runner.os }}
PYTHON_VERSION: ${{ matrix.python-version }}
run: |
python - << 'PY'
import json
import os
import subprocess
import sys

full_suite = os.environ["FULL_SUITE"].lower() == "true"
runner_os = os.environ["RUNNER_OS"]
python_version = os.environ["PYTHON_VERSION"]
def is_windows_python_3_11_or_greater() -> bool:
"""Aligned with pyproject: .[tf*] omits TensorFlow on Windows for Python 3.11+."""
return sys.platform == "win32" and sys.version_info >= (3, 11)

full_suite = os.environ["FULL_SUITE"].lower() == "true"
if full_suite:
scripts = [
"examples/testscript_tensorflow_single_animal.py",
Expand All @@ -247,22 +236,13 @@ jobs:
print("No functional scripts selected; skipping functional tests.")
raise SystemExit(0)

# Respect current TensorFlow support expectations on Windows
tf_allowed = not (runner_os == "Windows" and python_version in {"3.11", "3.12"})

filtered = []
for script in scripts:
is_tf = "tensorflow" in script
if is_tf and not tf_allowed:
print(f"Skipping unsupported TensorFlow script on {runner_os} Python {python_version}: {script}")
if "tensorflow" in script and is_windows_python_3_11_or_greater():
ver = f"{sys.version_info.major}.{sys.version_info.minor}"
print(
f"Skipping TensorFlow example on Windows {ver} (no TF in .[tf*] for 3.11+): {script}"
)
continue
filtered.append(script)

if not filtered:
print("No functional scripts remain after platform filtering; skipping.")
raise SystemExit(0)

for script in filtered:
print("Running:", script)
rc = subprocess.call([sys.executable, script])
if rc != 0:
Expand Down
71 changes: 60 additions & 11 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,23 +80,55 @@ docs = [
"sphinxcontrib-mermaid",
]
fmpose3d = [ "fmpose3d>=0.0.8" ]
# Use only one of [tf, tf-cu11, tf-cu12, tf-latest]. Do not combine extras.
tf = [
"protobuf<7",
# !!! TensorFlow is not supported on Windows with Python >= 3.11
"tensorflow>=2,<=2.10; platform_system=='Windows' and python_version<'3.11'",
"tensorflow>=2,<2.12; platform_system=='Linux' and python_version<'3.11'",
"tensorflow>=2.12; platform_system=='Linux' and python_version>='3.11'",
"tensorflow-io-gcs-filesystem==0.31; platform_system=='Windows' and python_version<'3.11'",
"tensorflow-io-gcs-filesystem>0.31; platform_system!='Windows'",
"tensorflow-macos>=2,<2.12; platform_system=='Darwin' and python_version<'3.11'",
"tensorflow-macos>=2.12; platform_system=='Darwin' and python_version>='3.11'",
"tensorflow>=2.12,<2.16; python_version<'3.12'",
"tensorflow>=2.16.1,<2.18; python_version>='3.12'",
"tensorflow-io-gcs-filesystem==0.31; platform_system=='Windows' and python_version<'3.12'",
"tensorflow-metal==1.2; platform_system=='Darwin' and python_version<'3.12'",
"tensorflow-metal>=1.2; platform_system=='Darwin' and python_version>='3.12'",
"tensorpack>=0.11",
"tf-keras; platform_system!='Windows'",
"tf-keras<2.15; python_version<'3.12'",
"tf-keras>=2.15,<2.18; python_version>='3.12'",
"tf-slim>=1.1",
]
tf-cu11 = [
"protobuf<7",
"tensorflow==2.14",
"tensorflow-io-gcs-filesystem==0.31; platform_system=='Windows'",
"tensorflow-metal==1.2; platform_system=='Darwin'",
"tensorpack==0.11",
"tf-keras==2.14.1",
"tf-slim==1.1",
"torch<2.1",
"torchvision<0.16",
]
tf-cu12 = [
"protobuf<7",
"tensorflow==2.18",
"tensorflow-metal==1.2; platform_system=='Darwin'",
"tensorpack==0.11",
"tf-keras==2.18",
"tf-slim==1.1",
"torch<2.11",
"torchvision<0.26",
]
tf-latest = [
"protobuf<7",
"tensorflow>=2.18",
"tensorflow-metal>=1.2; platform_system=='Darwin'",
"tensorpack>=0.11",
"tf-keras",
"tf-slim>=1.1",
]
# apple_mchips is kept for older systems, prefer [tf] in new projects.
apple_mchips = [
"tensorflow-macos; platform_system=='Darwin'",
"tensorflow-metal; platform_system=='Darwin'",
"protobuf<7; platform_system=='Darwin'",
"tensorflow>=2.12,<2.15; platform_system=='Darwin' and python_version<'3.12'",
"tensorflow>=2.15,<2.18; platform_system=='Darwin' and python_version>='3.12'",
"tensorflow-metal==1.2; platform_system=='Darwin' and python_version<'3.12'",
"tensorflow-metal>=1.2; platform_system=='Darwin' and python_version>='3.12'",
"tensorpack>=0.11; platform_system=='Darwin'",
"tf-keras; platform_system=='Darwin'",
"tf-slim>=1.1; platform_system=='Darwin'",
Expand Down Expand Up @@ -128,6 +160,23 @@ include-package-data = false
include = [ "deeplabcut*" ]
exclude = [ "tests*", "docs*", "examples*" ]

[tool.uv]
# One of tf / tf-cu12 / tf-latest. apple_mchips matches [tf] on macOS but conflicts with
# [tf-cu12] and [tf-latest] (overlapping tensorflow pins cannot be unified with uv's lock).
conflicts = [
[
{ extra = "tf" },
{ extra = "tf-cu11" },
{ extra = "tf-cu12" },
{ extra = "tf-latest" },
{ extra = "apple_mchips" },
],
[
{ extra = "tf-cu11" },
{ extra = "tf-cu12" },
{ extra = "fmpose3d" },
],
]
[[tool.uv.dependency-metadata]]
name = "openvino-dev"
version = "2022.1.0"
Expand Down
5 changes: 5 additions & 0 deletions tests/test_dataset_augmentation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@

from deeplabcut.pose_estimation_tensorflow.datasets import augmentation

tf = pytest.importorskip(
"tensorflow",
reason="TensorFlow not installed (use a project extra such as .[tf])",
)


@pytest.mark.parametrize(
"width, height",
Expand Down
5 changes: 5 additions & 0 deletions tests/test_evaluate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@
get_snapshots_by_index,
)

tf = pytest.importorskip(
"tensorflow",
reason="TensorFlow not installed (use a project extra such as .[tf])",
)


def make_single_animal_rmse_df(
bodyparts,
Expand Down
5 changes: 5 additions & 0 deletions tests/test_pose_multianimal_imgaug.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@
)
from deeplabcut.utils import read_plainconfig

tf = pytest.importorskip(
"tensorflow",
reason="TensorFlow not installed (use a project extra such as .[tf])",
)


def mock_imread(path, mode):
return (np.random.rand(400, 400, 3) * 255).astype(np.uint8)
Expand Down
7 changes: 6 additions & 1 deletion tests/test_predict_multianimal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,15 @@
# Licensed under GNU Lesser General Public License v3.0
#
import numpy as np
import tensorflow as tf
import pytest

from deeplabcut.pose_estimation_tensorflow.core import predict_multianimal

tf = pytest.importorskip(
"tensorflow",
reason="TensorFlow not installed (use a project extra such as .[tf])",
)

RADIUS = 5
THRESHOLD = 0.01
STRIDE = 8
Expand Down
Loading
Loading