## **ATTENTION: Our code base in this repo assumes you:**
-A. You have 2D videos and a DeepLabCut network to analyze them as described in the [main documentation](overview). This can be with multiple separate networks for each camera (less recommended), or one network trained on all views - recommended! (See [Nath*, Mathis* et al., 2019](https://www.biorxiv.org/content/10.1101/476531v1)). We also support multi-animal 3D with this code (please see [Lauer et al. 2022](https://doi.org/10.1038/s41592-022-01443-0)).
+A. You have 2D videos and a DeepLabCut network to analyze them as described in the
+[main documentation](overview). This can be with multiple
+separate networks for each camera (less recommended), or one network trained on all views - recommended! (See
+[Nath*, Mathis* et al., 2019](https://www.biorxiv.org/content/10.1101/476531v1)). We also support multi-animal 3D with this code (please see
+[Lauer et al. 2022](https://doi.org/10.1038/s41592-022-01443-0)).
B. You are using 2 cameras, in a [stereo configuration](https://github.com/DeepLabCut/DeepLabCut/blob/5ac4c8cb6bcf2314a3abfcf979b8dd170608e094/deeplabcut/pose_estimation_3d/camera_calibration.py#L223), for 3D*.
@@ -20,12 +27,16 @@ Here are other excellent options for you to use that extend DeepLabCut:
-- **[AcinoSet](https://github.com/African-Robotics-Unit/AcinoSet)**; **n**-camera support with triangulation, extended Kalman filtering, and trajectory optimization code (see video to the right for a min demo, courtesy of Prof. Patel), plus a GUI to visualize 3D data. It is built to work directly with DeepLabCut (but currently tailored to cheetah's, thus some coding skills are required at this time).
+- **[AcinoSet](https://github.com/African-Robotics-Unit/AcinoSet)**; **n**-camera support with triangulation, extended Kalman filtering, and trajectory optimization
+code (see video to the right for a min demo, courtesy of Prof. Patel), plus a GUI to visualize 3D data. It is built to
+work directly with DeepLabCut (but currently tailored to cheetah's, thus some coding skills are required at this time).
-- **[anipose.org](https://anipose.readthedocs.io/en/latest/)**; a wrapper for 3D deeplabcut that provides >3 camera support and is built to work directly with DeepLabCut. You can `pip install anipose` into your DLC conda environment.
+- **[anipose.org](https://anipose.readthedocs.io/en/latest/)**; a wrapper for 3D deeplabcut that provides >3 camera support and is built to work directly with
+DeepLabCut. You can `pip install anipose` into your DLC conda environment.
-- **Argus, easywand or DLTdv** w/DeepLabCut see https://github.com/haliaetus13/DLCconverterDLT; this can be used with the the highly popular Argus or DLTdv tools for wand calibration.
+- **Argus, easywand or DLTdv** w/DeepLabCut see https://github.com/haliaetus13/DLCconverterDLT; this can be used with
+the the highly popular Argus or DLTdv tools for wand calibration.
## Jump in with direct DeepLabCut 2-camera support:
@@ -39,83 +50,118 @@ Here are other excellent options for you to use that extend DeepLabCut:
### (1) Create a New 3D Project:
-Watch a [DEMO VIDEO](https://youtu.be/Eh6oIGE4dwI) on how to use this code, and check out the Notebook [here](https://github.com/DeepLabCut/DeepLabCut/blob/master/examples/JUPYTER/Demo_3D_DeepLabCut.ipynb)!
+Watch a [DEMO VIDEO](https://youtu.be/Eh6oIGE4dwI) on how to use this code, and check out the Notebook [here](https://github.com/DeepLabCut/DeepLabCut/blob/main/examples/JUPYTER/Demo_3D_DeepLabCut.ipynb)!
-You will run this function **one** time per project; a project is defined as a given set of cameras and calibration images. You can always analyze new videos within this project.
+You will run this function **one** time per project; a project is defined as a given set of cameras and calibration
+images. You can always analyze new videos within this project.
-The function **create\_new\_project\_3d** creates a new project directory specifically for converting the 2D pose to 3D pose, required subdirectories, and a basic 3D project configuration file. Each project is identified by the name of the project (e.g. Task1), name of the experimenter (e.g. YourName), as well as the date at creation.
+The function **create\_new\_project\_3d** creates a new project directory specifically for converting the 2D pose to 3D
+pose, required subdirectories, and a basic 3D project configuration file. Each project is identified by the name of the
+project (e.g. Task1), name of the experimenter (e.g. YourName), as well as the date at creation.
-Thus, this function requires the user to input the enter the name of the project, the name of the experimenter and number of cameras to be used. Currently, DeepLabCut supports triangulation using 2 cameras, but will expand to more than 2 cameras in a future version.
+Thus, this function requires the user to enter the name of the project, the name of the experimenter and number of
+cameras to be used. Currently, DeepLabCut supports triangulation using 2 cameras, but will expand to more than 2 cameras
+in a future version.
To start a 3D project type the following in ipython:
```python
-deeplabcut.create_new_project_3d('ProjectName','NameofLabeler',num_cameras = 2)
+deeplabcut.create_new_project_3d("ProjectName", "NameofLabeler", num_cameras=2)
```
-TIP 1: you can also pass ``working_directory=`Full path of the working directory'`` if you want to place this folder somewhere beside the current directory you are working in. If the optional argument ``working_directory`` is unspecified, the project directory is created in the current working directory.
+TIP 1: you can also pass `working_directory="Full path of the working directory"` if you want to place this folder
+somewhere beside the current directory you are working in. If the optional argument `working_directory` is unspecified,
+the project directory is created in the current working directory.
-TIP 2: you can also place ``config_path3d`` in front of ``deeplabcut.create_new_project_3d`` to create a variable that holds the path to the config.yaml file, i.e. ``config_path3d=deeplabcut.create_new_project_3d(...`` Or, set this variable for easy use. Please note that ``config_path3d='Full path of the 3D project configuration file'``.
+TIP 2: you can also place `config_path3d` in front of `deeplabcut.create_new_project_3d` to create a variable that holds
+the path to the config.yaml file, i.e. `config_path3d=deeplabcut.create_new_project_3d(...` Or, set this variable for
+easy use. Please note that `config_path3d='Full path of the 3D project configuration file'`.
- This function will create a project directory with the name **Name of the project+name of the experimenter+date of creation of the project+3d** in the **Working directory**. The project directory will have subdirectories: **calibration_images**, **camera_matrix**, **corners**, and **undistortion**. All the outputs generated during the course of a project will be stored in one of these subdirectories, thus allowing each project to be curated in separation from other projects.
+This function will create a project directory with the name **Name of the project+name of the experimenter+date of
+creation of the project+3d** in the **Working directory**. The project directory will have subdirectories:
+**calibration_images**, **camera_matrix**, **corners**, and **undistortion**. All the outputs generated during the
+course of a project will be stored in one of these subdirectories, thus allowing each project to be curated in
+separation from other projects.
- The purpose of the subdirectories is as follows:
+The purpose of the subdirectories is as follows:
- **calibration_images:** This directory will contain a set of calibration images acquired from the two cameras. A calibration image can be acquired using a printed checkerboard and its pair wise images are taken from both the cameras to consider as a set of calibration images. These pair of images are saved as ``.jpg`` with camera names as the prefix. e.g. ``camera-1-01.jpg`` and ``camera-2-01.jpg`` for the first pair of images. While taking the images:
-- Keep the orientation of the chessboard same and do not rotate more than 30 degrees. Rotating the chessboard circular will change the origin across the frames and may result in incorrect order of detected corners.
-- Cover several distances, and within each distance, cover all parts of the image view (all corners and center).
-Use a chessboard as big as possible, ideally a chessboard with of at least 8x6 squares.
-- Aim for taking at least 70 pair of images as after corner detection, some of the images might need to be discarded due to either incorrect corner detection or incorrect order of detected corners.
+**calibration_images:** This directory will contain a set of calibration images acquired from the two cameras. A
+calibration image can be acquired using a printed checkerboard and its pair wise images are taken from both the cameras
+to consider as a set of calibration images.
- **camera_matrix:** This directory will store the parameter for both the cameras as a pickle file. Specifically, these pickle files contain the intrinsic and extrinsic camera parameters. While the intrinsic parameters represent a transformation from 3-D camera's coordinates into the image coordinates, the extrinsic parameters represent a rigid transformation from world coordinate system to the 3-D camera's coordinate system.
+**camera_matrix:** This directory will store the parameter for both the cameras as a pickle file. Specifically, these
+pickle files contain the intrinsic and extrinsic camera parameters. While the intrinsic parameters represent a
+transformation from 3-D camera's coordinates into the image coordinates, the extrinsic parameters represent a rigid
+transformation from world coordinate system to the 3-D camera's coordinate system.
- **corners:** As a part of camera calibration, the checkerboard pattern is detected in the calibration images and these patterns will be stored in this directory. Each row of the checkerboard grid is marked with a unique color.
+**corners:** As a part of camera calibration, the checkerboard pattern is detected in the calibration images and these
+patterns will be stored in this directory. Each row of the checkerboard grid is marked with a unique color.
- **undistortion:** In order to check for calibration, the calibration images and the corresponding corner points are undistorted. These undistorted images are overlaid with undistorted points and will be stored in this directory.
+**undistortion:** In order to check for calibration, the calibration images and the corresponding corner points are
+undistorted. These undistorted images are overlaid with undistorted points and will be stored in this directory.
- Here is an overview of the calibration and triangulation workflow that follows:
+Here is an overview of the calibration and triangulation workflow that follows:
- +
@@ -213,22 +296,36 @@ The **triangulated file** is now saved under the same directory where the video
### (5) Visualize your 3D DeepLabCut Videos:
-In order to visualize both the 2D videos with tracked points plut the pose in 3D, the user can create a 3D video for certain frames (these are large files, so we advise just looking at a subset of frames). The user can specify the config file, the **path of the triangulated file folder**, and specify the start and end frame indices to create a 3D labeled video. Note that the ``triangulated_file_folder`` is where the newly created file that ends with ``yourDLC_3D_scorername.h5`` is located. This can be done using:
+In order to visualize both the 2D videos with tracked points plut the pose in 3D, the user can create a 3D video for
+certain frames (these are large files, so we advise just looking at a subset of frames). The user can specify the config
+file, the **path of the triangulated file folder**, and specify the start and end frame indices to create a 3D labeled
+video. Note that the `triangulated_file_folder` is where the newly created file that ends with
+`yourDLC_3D_scorername.h5` is located. This can be done using:
```python
-deeplabcut.create_labeled_video_3d(config_path, ['triangulated_file_folder'], start=50, end=250)
+deeplabcut.create_labeled_video_3d(
+ config_path,
+ ["triangulated_file_folder"],
+ start=50,
+ end=250
+)
```
-**TIP:** (see more parameters below) You can set how the axis of the 3D plot on the far right looks by changing the variables ``xlim``, ``ylim``, ``zlim`` and ``view``. Your checkerboard_3d.png image which was created above will show you the axis ranges. Here is an example:
+**TIP:** (see more parameters below) You can set how the axis of the 3D plot on the far right looks by changing the
+variables `xlim`, `ylim`, `zlim` and `view`. Your checkerboard_3d.png image which was created above will show you the
+axis ranges. Here is an example:
@@ -67,15 +67,19 @@ Getting Started: [a video tutorial on navigating the documentation!](https://www
- 
+
-
-
+
+
+
+
+ View in full screen
+
+
@@ -38,13 +39,13 @@ Now, we are going to install the core dependencies. The way this works is that t
`PyTorch` is the backend deep-learning language we wrote DLC3 in. To select the right version, head to the [“Install PyTorch”](https://pytorch.org/get-started/locally/) instructions in the official PyTorch Docs. Select your desired PyTorch build, operating system, select conda as your package manager and Python as the language. Select your compute platform (either a CUDA version or CPU only). Then, use the command to install the PyTorch package. Below are a few possible examples:
-- **GPU version of pytorch for CUDA 11.8**
+- **GPU version of pytorch for CUDA 12.4**
```
-conda install pytorch cudatoolkit=11.8 -c pytorch
+pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124
```
- **CPU only version of pytorch, using the latest version**
```
-conda install pytorch cpuonly -c pytorch
+pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
```
**(2) Install DeepLabCut**
@@ -61,7 +62,7 @@ pip install "git+https://github.com/DeepLabCut/DeepLabCut.git@pytorch_dlc#egg=de
```
- OR run for the **Stable release:**
```
-pip install "deeplabcut[gui,modelzoo,tf]"
+pip install "deeplabcut[gui,modelzoo,wandb]"
```
- This gives you DeepLabCut, the DLC GUI (gui), our latest neural networks (modelzoo) and a cool data logger (wandb) if you choose to use it later on!
diff --git a/docs/beginner-guides/labeling.md b/docs/beginner-guides/labeling.md
index 7d944137a6..e5c7492722 100644
--- a/docs/beginner-guides/labeling.md
+++ b/docs/beginner-guides/labeling.md
@@ -1,3 +1,4 @@
+(labeling)=
# Labeling GUI
## Selecting Frames to Label
@@ -45,7 +46,9 @@ Alright, you've got your extracted frames ready. Now comes the labeling!
- **Navigate Through Frames:** Use the slider to go from one frame to the next after you're done labeling.
- **Save Progress:** Remember to save your work as you go with **`Command and S`** (or **`Ctrl and S`** on Windows).
-> 💡 **Note:** For a detailed walkthrough on using the Napari labeling GUI, have a look at the [DeepLabCut Napari Guide](https://deeplabcut.github.io/DeepLabCut/docs/napari_GUI.html). Additionally, you can watch our instructional [YouTube video](https://www.youtube.com/watch?v=hsA9IB5r73E) for more insights and tips.
+> 💡 **Note:** For a detailed walkthrough on using the Napari labeling GUI, have a look at the
+[DeepLabCut Napari Guide](napari-gui). Additionally, you can watch our instructional
+[YouTube video](https://www.youtube.com/watch?v=hsA9IB5r73E) for more insights and tips.
### Completing the Set
diff --git a/docs/beginner-guides/manage-project.md b/docs/beginner-guides/manage-project.md
index a53789360f..3f26589ae2 100644
--- a/docs/beginner-guides/manage-project.md
+++ b/docs/beginner-guides/manage-project.md
@@ -41,4 +41,4 @@ A **`Configuration Editor`** window will open, displaying all the configuration
- **Save the Configuration:** Once you're satisfied with the modifications, click **`Save`**. This will store your changes and return you to the main GUI window.
-## Next, head over the beginner guide for [Labeling your data](https://deeplabcut.github.io/DeepLabCut/docs/labelling)
+## Next, head over the beginner guide for [Labeling your data](labeling)
diff --git a/docs/beginner-guides/video-analysis.md b/docs/beginner-guides/video-analysis.md
index 59c976360d..849d8b6638 100644
--- a/docs/beginner-guides/video-analysis.md
+++ b/docs/beginner-guides/video-analysis.md
@@ -16,7 +16,7 @@ After training and evaluating your model, the next step is to apply it to your v
- **Find Results in Your Project Folder:** After analysis, go to your project's video folder.
- **Analysis Files:** Look also for a `.metapickle`, an `.h5`, and possibly a `.csv` file for detailed analysis data.
-- **Review the Plot Poses Subfolder:** This contains visual outputs of the video analysis.
+- **Review the "plot-poses" subfolder:** This contains visual outputs of the video analysis.
diff --git a/docs/benchmark.md b/docs/benchmark.md
index 114568decd..612e307d0f 100644
--- a/docs/benchmark.md
+++ b/docs/benchmark.md
@@ -32,4 +32,4 @@ benchmarks. For an example of how to implement a benchmark submission, refer to
.. automodule:: deeplabcut.benchmark.metrics
:members:
:show-inheritance:
-```
\ No newline at end of file
+```
diff --git a/docs/convert_maDLC.md b/docs/convert_maDLC.md
index 6671afb1ef..19dd017692 100644
--- a/docs/convert_maDLC.md
+++ b/docs/convert_maDLC.md
@@ -1,10 +1,12 @@
(convert-maDLC)=
-# How to convert a pre-2.2 project for use with DeepLabCut 2.2
+# How to convert a pre-2.2 project for use with DeepLabCut 2.2 or later
-If you have a pre-2.2 project (`labeled-data`) with a **single animal** that you want to use with a multianimal project in DLC 2.2, i.e. use your older data to now train the new multi-task deep neural network, here is what you need to do.
+If you have a pre-2.2 project (`labeled-data`) with a **single animal** that you want to use with a multianimal project
+in DLC 2.2 or later, i.e. use your older data to now train the new multi-task deep neural network, here is what you
+need to do.
(1) We recommend you make a back-up of your project folder.
@@ -14,7 +16,8 @@ If you have a pre-2.2 project (`labeled-data`) with a **single animal** that you
-- After `task, scorer, date, project_path` please add the following (i.e. in the image above, you would start adding below line 6) Note, the ordering isn't important but useful to keep consistent with the template:
+- After `task, scorer, date, project_path` please add the following (i.e. in the image above, you would start adding
+below line 6) Note, the ordering isn't important but useful to keep consistent with the template:
```python
multianimalproject: true
@@ -29,9 +32,12 @@ individuals:
- mouse1
```
-- `"uniquebodyparts: []` can stay blank, unless you have other items labeled you want to estimate (consider these as similar to bodyparts in pre-2.2); i.e. corners of a box, etc. All unique bodyparts should not be connected to the multianimal bodyparts in the skeleton you will eventually make. But see "advanced option" below.
+- `"uniquebodyparts: []` can stay blank, unless you have other items labeled you want to estimate (consider these as
+similar to bodyparts in pre-2.2); i.e. corners of a box, etc. All unique bodyparts should not be connected to the
+multianimal bodyparts in the skeleton you will eventually make. See "advanced option" below.
-- Please move your "bodyparts:" to "multianimalbodyparts:" (bodypart names must stay the same!) These are the parts that will always be interconnected fully!
+- Please move your "bodyparts:" to "multianimalbodyparts:" (bodypart names must stay the same!) These are the parts
+that will always be interconnected fully!
```python
multianimalbodyparts:
- snout
@@ -46,20 +52,25 @@ then you can set `bodyparts: MULTI!`
deeplabcut.convert2_maDLC(path_config_file, userfeedback=True)
```
-Now you will see that your data within `labeled-data` are converted to a new format, and the single animal format was saved for you under a new file named `CollectedData_ ...singleanimal.h5` and `.csv` as a back-up!
+Now you will see that your data within `labeled-data` are converted to a new format, and the single animal format was
+saved for you under a new file named `CollectedData_ ...singleanimal.h5` and `.csv` as a back-up!
-(4) We strongly recommend to first run check_labels and verify that the conversion was as expected before creating a multianimal training dataset. For instance, you can load this project `config.yaml` in the Project Manager GUI and check labels then create a multi-animal training set with
+(4) We strongly recommend to first run check_labels and verify that the conversion was as expected before creating a
+multianimal training dataset. For instance, you can load this project `config.yaml` in the Project Manager GUI and
+check labels then create a multi-animal training set with
```python
deeplabcut.create_multianimaltraining_dataset(path_config_file)
```
to begin training.
-**Advanced option:** You can also assign former `bodyparts` to either `uniquebodyparts` or `multianimalbodyparts` (you can even leave some unassigned, which means they will be dropped in the conversion).
+**Advanced option:** You can also assign former `bodyparts` to either `uniquebodyparts` or `multianimalbodyparts`
+(you can even leave some unassigned, which means they will be dropped in the conversion).
Example: Imagine you had a project with the moon and a rocket with two parts labeled:
`bodyparts: [moon, rocket_tip,rocket_bottom]`
-Now you want to use this former project (labeled-data) and work on a new dataset (videos) with one moon but multiple (3) rockets. Then convert it as follows:
+Now you want to use this former project (labeled-data) and work on a new dataset (videos) with one moon but multiple
+(3) rockets. Then convert it as follows:
```
individuals: [rocket1, rocket2, rocket3]
uniquebodyparts: [moon]
diff --git a/docs/course.md b/docs/course.md
index 2ad096995f..b71438cb4f 100644
--- a/docs/course.md
+++ b/docs/course.md
@@ -1,5 +1,10 @@
## DeepLabCut Self-paced Course
+::::{warning}
+This course was designed for DLC 2.
+An updated version for DLC 3 is in the works.
+::::
+
Do you have video of animal behaviors? Step 1: Get Poses ...
@@ -17,7 +22,7 @@ We expect it to take *roughly* 1-2 weeks to get through if you do it rigorously.
## Installation:
You need Python and DeepLabCut installed!
-- [See these "beginner docs" for help!](https://deeplabcut.github.io/DeepLabCut/docs/beginners-guide.html)
+- [See these "beginner docs" for help!](beginners-guide)
- **WATCH:** overview of conda: [Python Tutorial: Anaconda - Installation and Using Conda](https://www.youtube.com/watch?v=YJC6ldI3hWk)
@@ -32,10 +37,8 @@ You need Python and DeepLabCut installed!
- **Learning:** learning and teaching signal processing, and overview from Prof. Demba Ba [talk at JupyterCon](https://www.youtube.com/watch?v=ywz-LLYwkQQ)
-- **Learning:** Watch a talk from Alexander Mathis (a lead DeepLabCut developer) [talk about DeepLabCut!](https://www.youtube.com/watch?v=ZjWPHM0sL4E)
-
- **DEMO:** Can I DEMO DEEPLABCUT (DLC) quickly?
- - Yes: [you can click through this DEMO notebook](https://github.com/DeepLabCut/DeepLabCut/blob/master/examples/COLAB/COLAB_DEMO_mouse_openfield.ipynb)
+ - Yes: [you can click through this DEMO notebook](https://github.com/DeepLabCut/DeepLabCut/blob/main/examples/COLAB/COLAB_DEMO_mouse_openfield.ipynb)
- AND follow along with me: [Video Tutorial!](https://www.youtube.com/watch?v=DRT-Cq2vdWs)
@@ -55,9 +58,9 @@ You need Python and DeepLabCut installed!
**What you need:** any videos where you can see the animals/objects, etc.
You can use our demo videos, grab some from the internet, or use whatever older data you have. Any camera, color/monochrome, etc will work. Find diverse videos, and label what you want to track well :)
-- IF YOU ARE PART OF THE COURSE: you will be contributing to the DLC Model Zoo :smile:
+- IF YOU ARE PART OF THE COURSE: you will be contributing to the DLC Model Zoo 😊
- - **Slides:** [Overview of starting new projects](https://github.com/DeepLabCut/DeepLabCut-Workshop-Materials/blob/master/part1-labeling.pdf)
+ - **Slides:** [Overview of starting new projects](https://github.com/DeepLabCut/DeepLabCut-Workshop-Materials/blob/main/part1-labeling.pdf)
- **READ ME PLEASE:** [DeepLabCut, the science](https://rdcu.be/4Rep)
- **READ ME PLEASE:** [DeepLabCut, the user guide](https://rdcu.be/bHpHN)
- **WATCH:** Video tutorial 1: [using the Project Manager GUI](https://www.youtube.com/watch?v=KcXogR-p5Ak)
@@ -71,12 +74,12 @@ You can use our demo videos, grab some from the internet, or use whatever older
### **Module 2: Neural Networks**
- - **Slides:** [Overview of creating training and test data, and training networks](https://github.com/DeepLabCut/DeepLabCut-Workshop-Materials/blob/master/part2-network.pdf)
+ - **Slides:** [Overview of creating training and test data, and training networks](https://github.com/DeepLabCut/DeepLabCut-Workshop-Materials/blob/main/part2-network.pdf)
- **READ ME PLEASE:** [What are convolutional neural networks?](https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53)
- **READ ME PLEASE:** Here is a new paper from us describing challenges in robust pose estimation, why PRE-TRAINING really matters - which was our major scientific contribution to low-data input pose-estimation - and it describes new networks that are available to you. [Pretraining boosts out-of-domain robustness for pose estimation](https://paperswithcode.com/paper/pretraining-boosts-out-of-domain-robustness)
- - **MORE DETAILS:** ImageNet: check out the original paper and dataset: http://www.image-net.org/ (link to [ppt from Dr. Fei-Fei Li](http://www.image-net.org/papers/ImageNet_2010.ppt))
+ - **MORE DETAILS:** ImageNet: check out the original paper and dataset: http://www.image-net.org/
- **REVIEW PAPER:** [A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives](https://www.sciencedirect.com/science/article/pii/S0896627320307170)
@@ -84,18 +87,18 @@ You can use our demo videos, grab some from the internet, or use whatever older
Before you create a training/test set, please read/watch:
- - **More information:** [Which types neural networks are available, and what should I use?](https://github.com/AlexEMG/DeepLabCut/wiki/What-neural-network-should-I-use%3F)
+ - **More information:** [Which types neural networks are available, and what should I use?](https://github.com/DeepLabCut/DeepLabCut/wiki/What-neural-network-should-I-use%3F-(Trade-offs,-speed-performance,-and-considerations))
- **WATCH:** Video tutorial 1: [How to test different networks in a controlled way](https://www.youtube.com/watch?v=WXCVr6xAcCA)
- Now, decide what model(s) you want to test.
- IF you want to train on your CPU, then run the step `create_training_dataset`, in the GUI etc. on your own computer.
- - IF you want to use GPUs on google colab, [**(1)** watch this FIRST/follow along here!](https://www.youtube.com/watch?v=qJGs8nxx80A) **(2)** move your whole project folder to Google Drive, and then [**use this notebook**](https://github.com/DeepLabCut/DeepLabCut/blob/master/examples/COLAB/COLAB_YOURDATA_TrainNetwork_VideoAnalysis.ipynb)
+ - IF you want to use GPUs on google colab, [**(1)** watch this FIRST/follow along here!](https://www.youtube.com/watch?v=qJGs8nxx80A) **(2)** move your whole project folder to Google Drive, and then [**use this notebook**](https://github.com/DeepLabCut/DeepLabCut/blob/main/examples/COLAB/COLAB_YOURDATA_TrainNetwork_VideoAnalysis.ipynb)
**MODULE 2 webinar**: https://youtu.be/ILsuC4icBU0
### **Module 3: Evalution of network performance**
- - **Slides** [Evalute your network](https://github.com/DeepLabCut/DeepLabCut-Workshop-Materials/blob/master/part3-analysis.pdf)
+ - **Slides** [Evalute your network](https://github.com/DeepLabCut/DeepLabCut-Workshop-Materials/blob/main/part3-analysis.pdf)
- **WATCH:** [Evaluate the network in ipython](https://www.youtube.com/watch?v=bgfnz1wtlpo)
- why evaluation matters; how to benchmark; analyzing a video and using scoremaps, conf. readouts, etc.
diff --git a/docs/deeplabcutlive.md b/docs/deeplabcutlive.md
index f8784b651d..f1edbff3cc 100644
--- a/docs/deeplabcutlive.md
+++ b/docs/deeplabcutlive.md
@@ -1,3 +1,4 @@
+(deeplabcut-live)=
# DeepLabCut-Live!
We provide two additional pip packages that allow you to record and stream camera data and run DeeplabCut models in real-time.
diff --git a/docs/gui/PROJECT_GUI.md b/docs/gui/PROJECT_GUI.md
index 106e7eca5c..e0883c324e 100644
--- a/docs/gui/PROJECT_GUI.md
+++ b/docs/gui/PROJECT_GUI.md
@@ -10,7 +10,7 @@ As some users may be more comfortable working with an interactive interface, we
(1) Install DeepLabCut using the simple-install with Anaconda found [here!](how-to-install)*.
Now you have DeepLabCut installed, but if you want to update it, either follow the prompt in the GUI which will ask you to upgrade when a new version is available, or just go into your env (activate DEEPLABCUT) then run:
-` pip install 'deeplabcut[gui,modelzoo]'` *but please see [full install guide](https://deeplabcut.github.io/DeepLabCut/docs/installation.html)!
+` pip install 'deeplabcut[gui,modelzoo]'` *but please see [full install guide](how-to-install)!
(2) Open the terminal and run: `python -m deeplabcut`
@@ -23,15 +23,14 @@ Now you have DeepLabCut installed, but if you want to update it, either follow t
Start at the Project Management Tab and work your way through the tabs to built your customized model and deploy it on new data.
We recommend to keep the terminal visible (as well as the GUI) so you can see the ongoing processes as you step through your project, or any errors that might arise.
-- For specific napari-based labeling features, see the ["napari gui" docs](https://deeplabcut.github.io/DeepLabCut/docs/napari_GUI.html#usage).
+- For specific napari-based labeling features, see the ["napari gui" docs](napari-gui-usage).
- To change from dark to light mode, set appearance at the top:
-Next, open the **config.yaml** file, which was created during **create\_new\_project**. You can edit this file in any text editor. Familiarize yourself with the meaning of the parameters (Box 1). You can edit various parameters, in particular you **must add the list of *bodyparts* (or points of interest)** that you want to track. You can also set the *colormap* here that is used for all downstream steps (can also be edited at anytime), like labeling GUIs, videos, etc. Here any [matplotlib colormaps](https://matplotlib.org/tutorials/colors/colormaps.html) will do!
+You can also set the *colormap* here that is used for all downstream steps (can also be edited at anytime), like labeling GUIs, videos, etc. Here any [matplotlib colormaps](https://matplotlib.org/tutorials/colors/colormaps.html) will do!
An easy way to programmatically edit the config file at any time is to use the function **edit\_config**, which takes the full path of the config file to edit and a dictionary of key–value pairs to overwrite.
-````python
-edits = {'colormap': 'summer',
- 'individuals': ['mickey', 'minnie', 'bianca'],
- 'skeleton': [['snout', 'tailbase'], ['snout', 'rightear']]}
+```python
+import deeplabcut
+
+config_path = "/path/to/project-dlc-2025-01-01/config.yaml"
+edits = {
+ "colormap": "summer",
+ "individuals": ["mickey", "minnie", "bianca"],
+ "skeleton": [["snout", "tailbase"], ["snout", "rightear"]]
+}
deeplabcut.auxiliaryfunctions.edit_config(config_path, edits)
-````
+```
Please DO NOT have spaces in the names of bodyparts, uniquebodyparts, individuals, etc.
-**ATTENTIONt:** You need to edit the config.yaml file to **modify the following items** which specify the animal ID, body parts, and any unique labels. Note, we also highly recommend that you use **more bodypoints** that you might be interested in for your experiment, i.e., labeling along the spine/tail for 8 bodypoints would be better than four. This will help the performance.
+**ATTENTION:** You need to edit the config.yaml file to **modify the following items** which specify the animal ID, bodyparts, and any unique labels. Note, we also highly recommend that you use **more bodyparts** that you might be interested in for your experiment, i.e., labeling along the spine/tail for 8 bodyparts would be better than four. This will help the performance.
Modifying the `config.yaml` is crucial:
@@ -123,6 +168,7 @@ multianimalbodyparts:
identity: True/False
```
+
**Individuals:** are names of "individuals" in the annotation dataset. These should/can be generic (e.g. mouse1, mouse2, etc.). These individuals are comprised of the same bodyparts defined by `multianimalbodyparts`. For annotation in the GUI and training, it is important that all individuals in each frame are labeled. Thus, keep in mind that you need to set individuals to the maximum number in your labeled-data set, .i.e., if there is (even just one frame) with 17 animals then the list should be `- indv1` to `- indv17`. Note, once trained if you have a video with more or less animals, that is fine - you can have more or less animals during video analysis!
**Identity:** If you can tell the animals apart, i.e., one might have a collar, or a black marker on the tail of a mouse, then you should label these individuals consistently (i.e., always label the mouse with the black marker as "indv1", etc). If you have this scenario, please set `identity: True` in your `config.yaml` file. If you have 4 black mice, and you truly cannot tell them apart, then leave this as `false`.
@@ -131,14 +177,22 @@ identity: True/False
**Uniquebodyparts:** are points that you want to track, but that appear only once within each frame, i.e. they are "unique". Typically these are things like unique objects, landmarks, tools, etc. They can also be animals, e.g. in the case where one German shepherd is attending to many sheep the sheep bodyparts would be multianimalbodyparts, the shepherd parts would be uniquebodyparts and the individuals would be the list of sheep (e.g. Polly, Molly, Dolly, ...).
-### Select Frames to Label:
+### (C) Select Frames to Label
**CRITICAL:** A good training dataset should consist of a sufficient number of frames that capture the breadth of the behavior. This ideally implies to select the frames from different (behavioral) sessions, different lighting and different animals, if those vary substantially (to train an invariant, robust feature detector). Thus for creating a robust network that you can reuse in the laboratory, a good training dataset should reflect the diversity of the behavior with respect to postures, luminance conditions, background conditions, animal identities, etc. of the data that will be analyzed. For the simple lab behaviors comprising mouse reaching, open-field behavior and fly behavior, 100−200 frames gave good results [Mathis et al, 2018](https://www.nature.com/articles/s41593-018-0209-y). However, depending on the required accuracy, the nature of behavior, the video quality (e.g. motion blur, bad lighting) and the context, more or less frames might be necessary to create a good network. Ultimately, in order to scale up the analysis to large collections of videos with perhaps unexpected conditions, one can also refine the data set in an adaptive way (see refinement below). **For maDLC, be sure you have labeled frames with closely interacting animals!**
The function `extract_frames` extracts frames from all the videos in the project configuration file in order to create a training dataset. The extracted frames from all the videos are stored in a separate subdirectory named after the video file’s name under the ‘labeled-data’. This function also has various parameters that might be useful based on the user’s need.
+
```python
-deeplabcut.extract_frames(config_path, mode='automatic/manual', algo='uniform/kmeans', userfeedback=False, crop=True/False)
+deeplabcut.extract_frames(
+ config_path,
+ mode='automatic/manual',
+ algo='uniform/kmeans',
+ userfeedback=False,
+ crop=True/False,
+)
```
+
**CRITICAL POINT:** It is advisable to keep the frame size small, as large frames increase the training and
inference time, or you might not have a large enough GPU for this.
When running the function `extract_frames`, if the parameter crop=True, then you will be asked to draw a box within the GUI (and this is written to the config.yaml file).
@@ -160,63 +214,99 @@ behaviors, and not extract the frames across the whole video. This can be achiev
parameters in the config.yaml file. Also, the user can change the number of frames to extract from each video using
the numframes2extract in the config.yaml file.
- **For maDLC, be sure you have labeled frames with closely interacting animals!** Therefore, manually selecting some frames is a good idea if interactions are not highly frequent in the video.
+```{TIP}
+For maDLC, **be sure you have labeled frames with closely interacting animals**!
+Therefore, manually selecting some frames is a good idea if interactions are not highly
+frequent in the video.
+```
-However, picking frames is highly dependent on the data and the behavior being studied. Therefore, it is hard to
-provide all purpose code that extracts frames to create a good training dataset for every behavior and animal. If the user feels specific frames are lacking, they can extract hand selected frames of interest using the interactive GUI
+However, picking frames is highly dependent on the data and the behavior being studied.
+Therefore, it is hard to provide all purpose code that extracts frames to create a good
+training dataset for every behavior and animal. If the user feels specific frames are
+lacking, they can extract hand selected frames of interest using the interactive GUI
provided along with the toolbox. This can be launched by using:
+
```python
deeplabcut.extract_frames(config_path, 'manual')
```
-The user can use the *Load Video* button to load one of the videos in the project configuration file, use the scroll
-bar to navigate across the video and *Grab a Frame* (or a range of frames, as of version 2.0.5) to extract the frame(s). The user can also look at the extracted frames and e.g. delete frames (from the directory) that are too similar before reloading the set and then manually annotating them.
+// FIXME(niels) - add a napari frame extractor description.
+The user can use the *Load Video* button to load one of the videos in the project
+configuration file, use the scroll bar to navigate across the video and *Grab a Frame*.
+The user can also look at the extracted frames and e.g. delete frames (from the
+directory) that are too similar before reloading the set and then manually annotating
+them.
+
+````{admonition} Click the button to see API Docs
+:class: dropdown
+```{eval-rst}
+.. include:: ./api/deeplabcut.extract_frames.rst
+```
+````
-### Label Frames:
+### (D) Label Frames
```python
deeplabcut.label_frames(config_path)
```
-As of 2.2 there is a new multi-animal labeling GUI (as long as in your `config.yaml` says `multianimalproject: true` at the top, this will automatically launch).
+The toolbox provides a function **label_frames** which helps the user to easily label
+all the extracted frames using an interactive graphical user interface (GUI). The user
+should have already named the bodyparts to label (points of interest) in the
+project’s configuration file by providing a list. The following command invokes the
+napari-deeplabcut labelling GUI.
-The toolbox provides a function **label_frames** which helps the user to easily label all the extracted frames using
-an interactive graphical user interface (GUI). The user should have already named the body parts to label (points of
-interest) in the project’s configuration file by providing a list. The following command invokes the labeling toolbox.
+[🎥 DEMO](https://youtu.be/hsA9IB5r73E)
-The user needs to use the *Load Frames* button to select the directory which stores the extracted frames from one of
-the videos. Subsequently, the user can use one of the radio buttons (top right) to select a body part to label. **RIGHT** click to add the label. Left click to drag the label, if needed. If you label a part accidentally, you can use the middle button on your mouse to delete (or hit the delete key while you hover over the point)! If you cannot see a body part in the frame, skip over the label! Please see the ``HELP`` button for more user instructions. This auto-advances once you labeled the first body part. You can also advance to the next frame by clicking on the RIGHT arrow on your keyboard (and go to a previous frame with LEFT arrow).
-Each label will be plotted as a dot in a unique color.
+HOT KEYS IN THE Labeling GUI (also see "help" in GUI):
-The user is free to move around the body part and once satisfied with its position, can select another radio button
-(in the top right) to switch to the respective body part (it otherwise auto-advances). The user can skip a body part if it is not visible. Once all the visible body parts are labeled, then the user can use ‘Next Frame’ to load the following frame. The user needs to save the labels after all the frames from one of the videos are labeled by clicking the save button at the bottom right. Saving the labels will create a labeled dataset for each video in a hierarchical data file format (HDF) in the
-subdirectory corresponding to the particular video in **labeled-data**. You can save at any intermediate step (even without closing the GUI, just hit save) and you return to labeling a dataset by reloading it!
+```
+Ctrl + C: Copy labels from previous frame.
+Keyboard arrows: advance frames.
+Delete key: delete label.
+```
-**CRITICAL POINT:** It is advisable to **consistently label similar spots** (e.g., on a wrist that is very large, try
-to label the same location). In general, invisible or occluded points should not be labeled by the user, unless you want to teach the network to "guess" - this is possible, but could affect accuracy. If you don't want/or don't see a bodypart, they can simply be skipped by not applying the label anywhere on the frame.
+
-OPTIONAL: In the event of adding more labels to the existing labeled dataset, the user need to append the new
-labels to the bodyparts in the config.yaml file. Thereafter, the user can call the function **label_frames**. A box will pop up and ask the user if they wish to display all parts, or only add in the new labels. Saving the labels after all the images are labelled will append the new labels to the existing labeled dataset.
+**CRITICAL POINT:** It is advisable to **consistently label similar spots** (e.g., on a
+wrist that is very large, try to label the same location). In general, invisible or
+occluded points should not be labeled by the user, unless you want to teach the network
+to "guess" - this is possible, but could affect accuracy. If you don't want/or don't see
+a bodypart, they can simply be skipped by not applying the label anywhere on the frame.
-**maDeepLabCut CRITICAL POINT:** For multi-animal labeling, unless you can tell apart the animals, you do not need to worry about the "ID" of each animal. For example: if you have a white and black mouse label the white mouse as animal 1, and black as animal 2 across all frames. If two black mice, then the ID label 1 or 2 can switch between frames - no need for you to try to identify them (but always label consistently within a frame). If you have 2 black mice but one always has an optical fiber (for example), then DO label them consistently as animal1 and animal_fiber (for example). The point of multi-animal DLC is to train models that can first group the correct bodyparts to individuals, then associate those points in a given video to a specific individual, which then also uses temporal information to link across the video frames.
+OPTIONAL: In the event of adding more labels to the existing labeled dataset, the user
+needs to append the new labels to the bodyparts in the config.yaml file. Thereafter, the
+user can call the function **label_frames**. A box will pop up and ask the user if they
+wish to display all parts, or only add in the new labels. Saving the labels after all
+the images are labelled will append the new labels to the existing labeled dataset.
-Note, we also highly recommend that you use more bodypoints that you might otherwise have (see the example below).
+**maDeepLabCut CRITICAL POINT:** For multi-animal labeling, unless you can tell apart
+the animals, you do not need to worry about the "ID" of each animal. For example: if you
+have a white and black mouse label the white mouse as animal 1, and black as animal 2
+across all frames. If two black mice, then the ID label 1 or 2 can switch between
+frames - no need for you to try to identify them (but always label consistently within a
+frame). If you have 2 black mice but one always has an optical fiber (for example), then
+DO label them consistently as animal1 and animal_fiber (for example). The point of
+multi-animal DLC is to train models that can first group the correct bodyparts to
+individuals, then associate those points in a given video to a specific individual,
+which then also uses temporal information to link across the video frames.
-**Example Labeling with maDeepLabCut:**
-- note you should within an animal be consistent, i.e., all bodyparts on mouse1 should be on mouse1, but across frames "mouse1" can be any of the black mice (as here it is nearly impossible to tell them apart visually). IF you can tell them apart, do label consistently!
+Note, we also highly recommend that you use more bodyparts that you might otherwise have
+(see the example below).
-
-
-
@@ -225,13 +315,33 @@ deeplabcut.check_labels(config_path, visualizeindividuals=True/False)
For each video directory in labeled-data this function creates a subdirectory with **labeled** as a suffix. Those directories contain the frames plotted with the annotated body parts. The user can double check if the body parts are labeled correctly. If they are not correct, the user can reload the frames (i.e. `deeplabcut.label_frames`), move them around, and click save again.
+````{admonition} Click the button to see API Docs
+:class: dropdown
+```{eval-rst}
+.. include:: ./api/deeplabcut.check_labels.rst
+```
+````
+
+### (F) Create Training Dataset
+
+At this point, you'll need to select your neural network type.
+
+For the **PyTorch engine**, please see [the PyTorch Model Architectures](
+dlc3-architectures) for options.
-### Create Training Dataset:
+For the **TensorFlow engine**, please see Lauer et al. 2021 for options. Multi-animal
+models will use `imgaug`, ADAM optimization, our new DLCRNet, and batch training. We
+suggest keeping these defaults at this time. At this step, the ImageNet pre-trained
+networks (i.e. ResNet-50) weights will be downloaded. If they do not download (you will
+see this downloading in the terminal, then you may not have permission to do so (
+something we have seen with some Windows users - see the **[
+WIKI troubleshooting for more help!](
+https://github.com/DeepLabCut/DeepLabCut/wiki/Troubleshooting-Tips)**).
-At this point you also select your neural network type. Please see Lauer et al. 2021 for options. For **create_multianimaltraining_dataset** we already changed this such that by default you will use imgaug, ADAM optimization, our new DLCRNet, and batch training. We suggest these defaults at this time. Then run:
+Then run:
```python
-deeplabcut.create_multianimaltraining_dataset(config_path)
+deeplabcut.create_training_dataset(config_path)
```
- The set of arguments in the function will shuffle the combined labeled dataset and split it to create train and test
@@ -242,47 +352,33 @@ keeps track of how often the dataset was refined).
- OPTIONAL: If the user wishes to benchmark the performance of the DeepLabCut, they can create multiple
training datasets by specifying an integer value to the `num_shuffles`; see the docstring for more details.
-- Each iteration of the creation of a training dataset will create several files, which is used by the feature detectors,
-and a ``.pickle`` file that contains the meta information about the training dataset. This also creates two subdirectories
-within **dlc-models** called ``test`` and ``train``, and these each have a configuration file called pose_cfg.yaml.
-Specifically, the user can edit the **pose_cfg.yaml** within the **train** subdirectory before starting the training. These
-configuration files contain meta information with regard to the parameters of the feature detectors. Key parameters
-are listed in Box 2.
-
-- At this step, the ImageNet pre-trained networks (i.e. ResNet-50) weights will be downloaded. If they do not download (you will see this downloading in the terminal, then you may not have permission to do so (something we have seen with some Windows users - see the **[WIKI troubleshooting for more help!](https://github.com/DeepLabCut/DeepLabCut/wiki/Troubleshooting-Tips)**).
-
-**OPTIONAL POINTS:**
-
-With the data-driven skeleton selection introduced in 2.2rc1+, DLC networks are trained by default
-on complete skeletons (i.e., they learn all possible redundant connections), before being optimally pruned
-at model evaluation. Although this procedure is by far superior to manually defining a graph,
-we leave manually-defining a skeleton as an option for the advanced user:
-
-```python
-my_better_graph = [[0, 1], [1, 2], [2, 3]] # These are indices in the list of multianimalbodyparts
-deeplabcut.create_multianimaltraining_dataset(config_path, paf_graph=my_better_graph)
-```
-
-Alternatively, the `skeleton` defined in the `config.yaml` file can also be used:
-
-```python
-deeplabcut.create_multianimaltraining_dataset(config_path, paf_graph='config')
-```
-
-Importantly, if a user-defined graph is used it still is required to cover all multianimalbodyparts at least once.
-
-**DATA AUGMENTATION:** At this stage you can also decide what type of augmentation to use. The default loaders work well for most all tasks (as shown on www.deeplabcut.org), but there are many options, more data augmentation, intermediate supervision, etc. Please look at the [**pose_cfg.yaml**](https://github.com/DeepLabCut/DeepLabCut/blob/master/deeplabcut/pose_cfg.yaml) file for a full list of parameters **you might want to change before running this step.** There are several data loaders that can be used. For example, you can use the default loader (introduced and described in the Nature Protocols paper), [TensorPack](https://github.com/tensorpack/tensorpack) for data augmentation (currently this is easiest on Linux only), or [imgaug](https://imgaug.readthedocs.io/en/latest/). We recommend `imgaug` (which is default now!). You can set this by passing:``` deeplabcut.create_training_dataset(config_path, augmenter_type='imgaug') ```
-
-The differences of the loaders are as follows:
-- `default`: our standard DLC 2.0 introduced in Nature Protocols variant (scaling, auto-crop augmentation) *will be renamed to `crop_scale` in a future release!*
-- `imgaug`: a lot of augmentation possibilities, efficient code for target map creation & batch sizes >1 supported. You can set the parameters such as the `batch_size` in the `pose_cfg.yaml` file for the model you are training.
-- `tensorpack`: a lot of augmentation possibilities, multi CPU support for fast processing, target maps are created less efficiently than in imgaug, does not allow batch size>1
-- `deterministic`: only useful for testing, freezes numpy seed; otherwise like default.
-
-Our recent [A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives](https://www.cell.com/neuron/pdf/S0896-6273(20)30717-0.pdf), details the advantage of augmentation for a worked example (see Fig 7). TL;DR: use imgaug and use the symmetries of your data!
-
-
-Alternatively, you can set the loader (as well as other training parameters) in the **pose_cfg.yaml** file of the model that you want to train. Note, to get details on the options, look at the default file: [**pose_cfg.yaml**](https://github.com/DeepLabCut/DeepLabCut/blob/master/deeplabcut/pose_cfg.yaml).
+- Each iteration of the creation of a training dataset will create several files, which
+is used by the feature detectors, and a ``.pickle`` file that contains the meta
+information about the training dataset. This also creates two subdirectories within
+**dlc-models-pytorch** (**dlc-models** for the TensorFlow engine) called ``test`` and
+``train``, and these each have a configuration file called pose_cfg.yaml. Specifically,
+the user can edit the **pytorch_config.yaml** (**pose_cfg.yaml** for TensorFlow engine)
+within the **train** subdirectory before starting the training. These configuration
+files contain meta information with regard to the parameters of the feature detectors.
+Key parameters are listed in Box 2.
+
+**DATA AUGMENTATION:** At this stage you can also decide what type of augmentation to
+use. Once you've called `create_training_dataset`, you can edit the
+[**pytorch_config.yaml**](dlc3-pytorch-config) file that was created (or for the
+TensorFlow engine, the [**pose_cfg.yaml**](
+https://github.com/DeepLabCut/DeepLabCut/blob/master/deeplabcut/pose_cfg.yaml) file).
+
+- PyTorch Engine: [Albumentations](https://albumentations.ai/docs/) is used for data
+augmentation. Look at the [**pytorch_config.yaml**](dlc3-pytorch-config) for more
+information about image augmentation options.
+- TensorFlow Engine: The default augmentation works well for most tasks (as shown on
+www.deeplabcut.org), but there are many options, more data augmentation, intermediate
+supervision, etc. Only `imgaug` augmentation is available for multi-animal projects.
+
+[A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives](
+https://www.cell.com/neuron/pdf/S0896-6273(20)30717-0.pdf), details the advantage of
+augmentation for a worked example (see Fig 8). TL;DR: use imgaug and use the symmetries
+of your data!
Importantly, image cropping as previously done with `deeplabcut.cropimagesandlabels` in multi-animal projects
is now part of the augmentation pipeline. In other words, image crops are no longer stored in labeled-data/..._cropped
@@ -292,118 +388,254 @@ In addition, one can specify a crop sampling strategy: crop centers can either b
As a reminder, cropping images into smaller patches is a form of data augmentation that simultaneously
allows the use of batch processing even on small GPUs that could not otherwise accommodate larger images + larger batchsizes (this usually increases performance and decreasing training time).
+**MODEL COMPARISON**: You can also test several models by creating the same train/test
+split for different networks.
+You can easily do this in the Project Manager GUI (by selecting the "Use an existing
+data split" option), which also lets you compare PyTorch and TensorFlow models.
-### Train The Network:
-
-```python
-deeplabcut.train_network(config_path, allow_growth=True)
-```
+````{versionadded} 3.0.0
+You can now create new shuffles using the same train/test split as
+existing shuffles with `create_training_dataset_from_existing_split`. This allows you to
+compare model performance (between different architectures or when using different
+training hyper-parameters) as the shuffles were trained on the same data, and evaluated
+on the same test data!
-The set of arguments in the function starts training the network for the dataset created for one specific shuffle. Note that you can change the loader (imgaug/default/etc) as well as other training parameters in the **pose_cfg.yaml** file of the model that you want to train (before you start training).
+Example usage - creating 3 new shuffles (with indices 10, 11 and 12) for a ResNet 50
+pose estimation model, using the same data split as was used for shuffle 0:
-Example parameters that one can call:
```python
-deeplabcut.train_network(config_path, shuffle=1, trainingsetindex=0, gputouse=None, max_snapshots_to_keep=5, autotune=False, displayiters=100, saveiters=15000, maxiters=30000, allow_growth=True)
+deeplabcut.create_training_dataset_from_existing_split(
+ config_path,
+ from_shuffle=0,
+ shuffles=[10, 11, 12],
+ net_type="resnet_50",
+)
```
+````
-By default, the pretrained networks are not in the DeepLabCut toolbox (as they are around 100MB each), but they get downloaded before you train. However, if not previously downloaded from the TensorFlow model weights, it will be downloaded and stored in a subdirectory *pre-trained* under the subdirectory *models* in *Pose_Estimation_Tensorflow*.
-At user specified iterations during training checkpoints are stored in the subdirectory *train* under the respective iteration directory.
-
-If the user wishes to restart the training at a specific checkpoint they can specify the full path of the checkpoint to
-the variable ``init_weights`` in the **pose_cfg.yaml** file under the *train* subdirectory (see Box 2).
-
-**CRITICAL POINT:** It is recommended to train the networks for thousands of iterations until the loss plateaus (typically around **500,000**) if you use batch size 1, and **50-100K** if you use batchsize 8 (the default).
+````{admonition} Click the button to see API Docs for deeplabcut.create_training_dataset
+:class: dropdown
+```{eval-rst}
+.. include:: ./api/deeplabcut.create_training_dataset.rst
+```
+````
-If you use **maDeepLabCut** the recommended training iterations is **20K-100K** (it automatically stops at 200K!), as we use Adam and batchsize 8; if you have to reduce the batchsize for memory reasons then the number of iterations needs to be increased.
+````{admonition} Click the button to see API Docs for deeplabcut.create_training_model_comparison
+:class: dropdown
+```{eval-rst}
+.. include:: ./api/deeplabcut.create_training_model_comparison.rst
+```
+````
-The variables ``display_iters`` and ``save_iters`` in the **pose_cfg.yaml** file allows the user to alter how often the loss is displayed and how often the weights are stored.
+````{admonition} Click the button to see API Docs for deeplabcut.create_training_dataset_from_existing_split
+:class: dropdown
+```{eval-rst}
+.. include:: ./api/deeplabcut.create_training_dataset_from_existing_split.rst
+```
+````
-**maDeepLabCut CRITICAL POINT:** For multi-animal projects we are using not only different and new output layers, but also new data augmentation, optimization, learning rates, and batch training defaults. Thus, please use a lower ``save_iters`` and ``maxiters``. I.e. we suggest saving every 10K-15K iterations, and only training until 50K-100K iterations. We recommend you look closely at the loss to not overfit on your data. The bonus, training time is much less!!!
+### (G) Train The Network
-**Parameters:**
+```python
+deeplabcut.train_network(config_path, shuffle=1)
```
-config : string
- Full path of the config.yaml file as a string.
-shuffle: int, optional
- Integer value specifying the shuffle index to select for training. Default is set to 1
+The set of arguments in the function starts training the network for the dataset created
+for one specific shuffle. Note that you can change training parameters in the
+[**pytorch_config.yaml**](dlc3-pytorch-config) file (or **pose_cfg.yaml** for TensorFlow
+models) of the model that you want to train (before you start training).
+
+At user specified iterations during training checkpoints are stored in the subdirectory
+*train* under the respective iteration & shuffle directory.
-trainingsetindex: int, optional
- Integer specifying which TrainingsetFraction to use. By default the first (note that TrainingFraction is a list in config.yaml).
+````{admonition} Tips on training models with the PyTorch Engine
+:class: dropdown
-gputouse: int, optional. Natural number indicating the number of your GPU (see number in nvidia-smi). If you do not have a GPU, put None.
-See: https://nvidia.custhelp.com/app/answers/detail/a_id/3751/~/useful-nvidia-smi-queries
+Example parameters that one can call:
-max_snapshots_to_keep: int, or None. Sets how many snapshots are kept, i.e. states of the trained network. For every saving iteration a snapshot is stored, however, only the last max_snapshots_to_keep many are kept! If you change this to None, then all are kept.
-See: https://github.com/DeepLabCut/DeepLabCut/issues/8#issuecomment-387404835
+```python
+deeplabcut.train_network(
+ config_path,
+ shuffle=1,
+ trainingsetindex=0,
+ device="cuda:0",
+ max_snapshots_to_keep=5,
+ displayiters=100,
+ save_epochs=5,
+ epochs=200,
+)
+```
+
+Pytorch models in DeepLabCut 3.0 are trained for a set number of epochs, instead of a
+maximum number of iterations (which is what was used for TensorFlow models). An epoch
+is a single pass through the training dataset, which means your model has seen each
+training image exactly once. So if you have 64 training images for your network, an
+epoch is 64 iterations with batch size 1 (or 32 iterations with batch size 2, 16 with
+batch size 4, etc.).
+
+By default, the pretrained networks are not in the DeepLabCut toolbox (as they can be
+more than 100MB), but they get downloaded automatically before you train.
+
+If the user wishes to restart the training at a specific checkpoint they can specify the
+full path of the checkpoint to the variable ``resume_training_from`` in the [
+**pytorch_config.yaml**](
+dlc3-pytorch-config) file (checkout the "Restarting Training at a Specific Checkpoint"
+section of the docs) under the *train* subdirectory.
+
+**CRITICAL POINT:** It is recommended to train the networks **until the loss plateaus**
+(depending on the dataset, model architecture and training hyper-parameters this happens
+after 100 to 250 epochs of training).
+
+The variables ``display_iters`` and ``save_epochs`` in the [**pytorch_config.yaml**](
+dlc3-pytorch-config) file allows the user to alter how often the loss is displayed
+and how often the weights are stored. We suggest saving every 5 to 25 epochs.
+````
-autotune: property of TensorFlow, somehow faster if 'false' (as Eldar found out, see https://github.com/tensorflow/tensorflow/issues/13317). Default: False
+````{admonition} Tips on training models with the TensorFlow Engine
+:class: dropdown
-displayiters: this variable is actually set in pose_config.yaml. However, you can overwrite it with this hack. Don't use this regularly, just if you are too lazy to dig out
-the pose_config.yaml file for the corresponding project. If None, the value from there is used, otherwise it is overwritten! Default: None
+Example parameters that one can call:
-saveiters: this variable is actually set in pose_config.yaml. However, you can overwrite it with this hack. Don't use this regularly, just if you are too lazy to dig out
-the pose_config.yaml file for the corresponding project. If None, the value from there is used, otherwise it is overwritten! Default: None
+```python
+deeplabcut.train_network(
+ config_path,
+ shuffle=1,
+ trainingsetindex=0,
+ gputouse=None,
+ max_snapshots_to_keep=5,
+ autotune=False,
+ displayiters=100,
+ saveiters=15000,
+ maxiters=30000,
+ allow_growth=True,
+)
+```
+
+By default, the pretrained networks are not in the DeepLabCut toolbox (as they are
+around 100MB each), but they get downloaded before you train. However, if not previously
+downloaded from the TensorFlow model weights, it will be downloaded and stored in a
+subdirectory *pre-trained* under the subdirectory *models* in
+*Pose_Estimation_Tensorflow*. At user specified iterations during training checkpoints
+are stored in the subdirectory *train* under the respective iteration directory.
+
+If the user wishes to restart the training at a specific checkpoint they can specify the
+full path of the checkpoint to the variable ``init_weights`` in the **pose_cfg.yaml**
+file under the *train* subdirectory (see Box 2).
+
+**CRITICAL POINT:** It is recommended to train the networks for thousands of iterations
+until the loss plateaus (typically around **500,000**) if you use batch size 1, and
+**50-100K** if you use batchsize 8 (the default).
+
+If you use **maDeepLabCut** the recommended training iterations is **20K-100K**
+(it automatically stops at 200K!), as we use Adam and batchsize 8; if you have to reduce
+ the batchsize for memory reasons then the number of iterations needs to be increased.
+
+The variables ``display_iters`` and ``save_iters`` in the **pose_cfg.yaml** file allows
+the user to alter how often the loss is displayed and how often the weights are stored.
+
+**maDeepLabCut CRITICAL POINT:** For multi-animal projects we are using not only
+different and new output layers, but also new data augmentation, optimization, learning
+rates, and batch training defaults. Thus, please use a lower ``save_iters`` and
+``maxiters``. I.e. we suggest saving every 10K-15K iterations, and only training until
+50K-100K iterations. We recommend you look closely at the loss to not overfit on your
+data. The bonus, training time is much less!!!
+````
-maxiters: This sets how many iterations to train. This variable is set in pose_config.yaml. However, you can overwrite it with this. If None, the value from there is used, otherwise it is overwritten! Default: None
+````{admonition} Click the button to see API Docs for train_network
+:class: dropdown
+```{eval-rst}
+.. include:: ./api/deeplabcut.train_network.rst
```
+````
-### Evaluate the Trained Network:
+### (H) Evaluate the Trained Network
+
+It is important to evaluate the performance of the trained network. This performance is
+measured by computing two metrics:
+
+- **Average root mean square error** (RMSE) between the manual labels and the ones
+predicted by your trained DeepLabCut model. The RMSE is proportional to the mean average
+Euclidean error (MAE) between the manual labels and the ones predicted by DeepLabCut.
+The MAE is displayed for all pairs and only likely pairs (>p-cutoff). This helps to
+exclude, for example, occluded body parts. One of the strengths of DeepLabCut is that
+due to the probabilistic output of the scoremap, it can, if sufficiently trained, also
+reliably report if a body part is visible in a given frame. (see discussions of finger
+tips in reaching and the Drosophila legs during 3D behavior in [Mathis et al, 2018]).
+- **Mean Average Precision** (mAP) and **Mean Average Recall** (mAR) for the individuals
+predicted by your trained DeepLabCut model. This metric describes the precision of your
+model, based on a considered definition of what a correct detection of an individual is.
+It isn't as useful for single-animal models, as RMSE does a great job of evaluating your
+model in that case.
+
+```{admonition} A more detailed description of mAP and mAR
+:class: dropdown
+
+For multi-animal pose estimation, multiple predictions can be made for each image.
+We want to get some idea of the proportion of correct predictions among all predictions
+that are made.
+However, the notion of "correct prediction" for pose estimation is not straightforward:
+is a prediction correct if all predicted keypoints are within 5 pixels of the ground
+truth? Within 2 pixels of the ground truth? What if all pixels but one match the ground
+truth perfectly, but the wrong prediction is 50 pixels away? Mean average precision (
+and mean average recall) estimate the precision/recall of your models by setting
+different "thresholds of correctness" and averaging results. How "correct" a
+prediction is can be evaluated through [object-keypoint similarity](
+https://cocodataset.org/#keypoints-eval).
+
+A good resource to get a deeper understanding of mAP is the [Stanford CS230 course](
+https://cs230.stanford.edu/section/8/#object-detection-iou-ap-and-map). While it
+describes mAP for object detection (where bounding boxes are predicted instead of
+keypoints), the same metric can be computed for pose estimation, where similarity
+between predictions and ground truth is computed through [object-keypoint similarity](
+https://cocodataset.org/#keypoints-eval) instead of intersection-over-union (IoU).
+```
+
+It's also important to visually inspect predictions on individual frames to assess the
+performance of your model. You can do this by setting `plotting=True` when you call
+`evaluate_network`. The evaluation results are computed by typing:
-Here, for traditional projects you will get a pixel distance metric and you should inspect the individual frames:
```python
-deeplabcut.evaluate_network(config_path, plotting=True)
+deeplabcut.evaluate_network(config_path, Shuffles=[1], plotting=True)
```
-:movie_camera:[VIDEO TUTORIAL AVAILABLE!](https://www.youtube.com/watch?v=bgfnz1wtlpo)
-It is important to evaluate the performance of the trained network. This performance is measured by computing
-the mean average Euclidean error (MAE; which is proportional to the average root mean square error) between the
-manual labels and the ones predicted by DeepLabCut. The MAE is saved as a comma separated file and displayed
-for all pairs and only likely pairs (>p-cutoff). This helps to exclude, for example, occluded body parts. One of the
-strengths of DeepLabCut is that due to the probabilistic output of the scoremap, it can, if sufficiently trained, also
-reliably report if a body part is visible in a given frame. (see discussions of finger tips in reaching and the Drosophila
-legs during 3D behavior in [Mathis et al, 2018]). The evaluation results are computed by typing:
+🎥 [VIDEO TUTORIAL AVAILABLE!](https://www.youtube.com/watch?v=bgfnz1wtlpo)
Setting ``plotting`` to True plots all the testing and training frames with the manual and predicted labels; these will
-be colored by body part type by default. They can alternatively be colored by individual by passing `plotting`=`individual`.
+be colored by body part type by default. They can alternatively be colored by individual by passing `plotting="individual"`.
The user should visually check the labeled test (and training) images that are created in the ‘evaluation-results’ directory.
Ideally, DeepLabCut labeled unseen (test images) according to the user’s required accuracy, and the average train
and test errors are comparable (good generalization). What (numerically) comprises an acceptable MAE depends on
many factors (including the size of the tracked body parts, the labeling variability, etc.). Note that the test error can
also be larger than the training error due to human variability (in labeling, see Figure 2 in Mathis et al, Nature Neuroscience 2018).
-**Optional parameters:**
-```
- Shuffles: list, optional -List of integers specifying the shuffle indices of the training dataset. The default is [1]
-
- plotting: bool, optional -Plots the predictions on the train and test images. The default is `False`; if provided it must be either `True` or `False`
-
- show_errors: bool, optional -Display train and test errors. The default is `True`
-
- comparisonbodyparts: list of bodyparts, Default is all -The average error will be computed for those body parts only (Has to be a subset of the body parts).
-
- gputouse: int, optional -Natural number indicating the number of your GPU (see number in nvidia-smi). If you do not have a GPU, put None. See: https://nvidia.custhelp.com/app/answers/detail/a_id/3751/~/useful-nvidia-smi-queries
-```
-
The plots can be customized by editing the **config.yaml** file (i.e., the colormap, scale, marker size (dotsize), and
-transparency of labels (alphavalue) can be modified). By default each body part is plotted in a different color
+transparency of labels (alpha-value) can be modified). By default each body part is plotted in a different color
(governed by the colormap) and the plot labels indicate their source. Note that by default the human labels are
-plotted as plus (‘+’), DeepLabCut’s predictions either as ‘.’ (for confident predictions with likelihood > p-cutoff) and
+plotted as plus (‘+’), DeepLabCut’s predictions either as ‘.’ (for confident predictions with likelihood > `pcutoff`) and
’x’ for (likelihood <= `pcutoff`).
-The evaluation results for each shuffle of the training dataset are stored in a unique subdirectory in a newly created
-directory ‘evaluation-results’ in the project directory. The user can visually inspect if the distance between the labeled
-and the predicted body parts are acceptable. In the event of benchmarking with different shuffles of same training
-dataset, the user can provide multiple shuffle indices to evaluate the corresponding network. If the generalization is
-not sufficient, the user might want to:
+The evaluation results for each shuffle of the training dataset are stored in a unique
+subdirectory in a newly created directory ‘evaluation-results-pytorch’ (or
+‘evaluation-results’ for TensorFlow models) in the project directory.
+The user can visually inspect if the distance between the labeled and the predicted body
+parts are acceptable. In the event of benchmarking with different shuffles of same training
+dataset, the user can provide multiple shuffle indices to evaluate the corresponding
+network. If the generalization is not sufficient, the user might want to:
-• check if the labels were imported correctly; i.e., invisible points are not labeled and the points of interest are
-labeled accurately
+• check if the labels were imported correctly; i.e., invisible points are not labeled
+and the points of interest are labeled accurately
• make sure that the loss has already converged
• consider labeling additional images and make another iteration of the training data set
+````{admonition} Click the button to see API Docs for evaluate_network
+:class: dropdown
+```{eval-rst}
+.. include:: ./api/deeplabcut.evaluate_network.rst
+```
+````
+
**maDeepLabCut: (or on normal projects!)**
In multi-animal projects, model evaluation is crucial as this is when
@@ -416,9 +648,10 @@ You should also plot the scoremaps, locref layers, and PAFs to assess performanc
```python
deeplabcut.extract_save_all_maps(config_path, shuffle=shuffle, Indices=[0, 5])
```
-you can drop "Indices" to run this on all training/testing images (this is very slow!)
-### Evaluating your network on videos
+You can drop "Indices" to run this on all training/testing images (this is very slow!)
+
+### (I) Analyze new Videos
**-------------------- DECISION POINT -------------------**
@@ -428,12 +661,28 @@ you can drop "Indices" to run this on all training/testing images (this is very
Please run:
```python
-scorername = deeplabcut.analyze_videos(config_path,['/fullpath/project/videos/testVideo.mp4'], videotype='.mp4')
-deeplabcut.create_video_with_all_detections(config_path, ['/fullpath/project/videos/testVideo.mp4'], videotype='.mp4')
+videos_to_analyze = ['/fullpath/project/videos/testVideo.mp4']
+scorername = deeplabcut.analyze_videos(config_path, videos_to_analyze, videotype='.mp4')
+deeplabcut.create_video_with_all_detections(config_path, videos_to_analyze, videotype='.mp4')
```
-Please note that you do **not** get the .h5/csv file you might be used to getting (this comes after tracking). You will get a `pickle` file that is used in `create_video_with_all_detections`.
-Another sanity check may be to examine the distributions of edge affinity costs using `deeplabcut.utils.plot_edge_affinity_distributions`. Easily separable distributions indicate that the model has learned strong links to group keypoints into distinct individuals — likely a necessary feature for the assembly stage (note that the amount of overlap will also depend on the amount of interactions between your animals in the daset).
-IF you have good clean out video, ending in `....full.mp4` (and the evaluation metrics look good, scoremaps look good, plotted evaluation images, and affinity distributions are far apart for most edges), then go forward!!!
+
+Please note that you do **not** get the .h5/csv file you might be used to getting (this
+comes after tracking). You will get a `pickle` file that is used in
+`create_video_with_all_detections`.
+
+For models predicting part-affinity fields, another sanity check may be to
+examine the distributions of edge affinity costs using `deeplabcut.utils.plot_edge_affinity_distributions`. Easily separable distributions
+indicate that the model has learned strong links to group keypoints into distinct
+individuals — likely a necessary feature for the assembly stage (note that the amount of
+overlap will also depend on the amount of interactions between your animals in the
+dataset). All TensorFlow multi-animal models use part-affinity fields and PyTorch models
+consisting of just a backbone name (e.g. `resnet_50`, `resnet_101`) use part-affinity
+fields. If you're unsure whether your PyTorch model has a one, check
+the **pytorch_config.yaml** for a `DLCRNetHead`.
+
+IF you have good clean out video, ending in `....full.mp4` (and the evaluation metrics
+look good, scoremaps look good, plotted evaluation images, and affinity distributions
+are far apart for most edges), then go forward!!!
If this does not look good, we recommend extracting and labeling more frames (even from more videos). Try to label close interactions of animals for best performance. Once you label more, you can create a new training set and train.
@@ -441,30 +690,53 @@ You can either:
1. extract more frames manually from existing or new videos and label as when initially building the training data set, or
2. let DeepLabCut find frames where keypoints were poorly detected and automatically extract those for you. All you need is
to run:
+
```python
deeplabcut.find_outliers_in_raw_data(config_path, pickle_file, video_file)
```
+
where pickle_file is the `_full.pickle` one obtains after video analysis.
Flagged frames will be added to your collection of images in the corresponding labeled-data folders for you to label.
-## Animal Assembly and Tracking across frames
+### Animal Assembly and Tracking across frames
-After pose estimation, now you perform assembly and tracking. *NEW* in 2.2 is a novel data-driven way to set the optimal skeleton and assembly metrics, so this no longer requires user input. The metrics, in case you do want to edit them, can be found in the `inference_cfg.yaml` file.
+After pose estimation, now you perform assembly and tracking.
+
+````{versionadded} v2.2.0
+*NEW* in 2.2 is a novel data-driven way to set the optimal skeleton and assembly
+metrics, so this no longer requires user input. The metrics, in case you do want to edit
+them, can be found in the `inference_cfg.yaml` file.
+````
### Optimized Animal Assembly + Video Analysis:
-- Please note that **novel videos DO NOT need to be added to the config.yaml file**. You can simply have a folder elsewhere on your computer and pass the video folder (then it will analyze all videos of the specified type (i.e. ``videotype='.mp4'``), or pass the path to the **folder** or exact video(s) you wish to analyze:
+Please note that **novel videos DO NOT need to be added to the config.yaml file**. You
+can simply have a folder elsewhere on your computer and pass the video folder (then it
+will analyze all videos of the specified type (i.e. ``videotype='.mp4'``), or pass the
+path to the **folder** or exact video(s) you wish to analyze:
```python
deeplabcut.analyze_videos(config_path, ['/fullpath/project/videos/'], videotype='.mp4', auto_track=True)
```
+
#### IF auto_track = True:
-- *NEW* in 2.2.0.3+: `deeplabcut.analyze_videos` has a new argument `auto_track=True`, chaining pose estimation, tracking, and stitching in a single function call with defaults we found to work well. Thus, you'll now get the `.h5` file you might be used to getting in standard DLC. If `auto_track=False`, one must run `convert_detections2tracklets` and `stitch_tracklets` manually (see below), granting more control over the last steps of the workflow (ideal for advanced users).
+```{versionadded} v2.2.0.3
+A new argument `auto_track=True`, was added to `deeplabcut.analyze_videos` chaining pose
+estimation, tracking, and stitching in a single function call with defaults we found to
+work well. Thus, you'll now get the `.h5` file you might be used to getting in standard
+DLC. If `auto_track=False`, one must run `convert_detections2tracklets` and
+`stitch_tracklets` manually (see below), granting more control over the last steps of
+the workflow (ideal for advanced users).
+```
#### IF auto_track = False:
- - You can validate the tracking parameters. Namely, you can iteratively change the parameters, run `convert_detections2tracklets` then load them in the GUI (`refine_tracklets`) if you want to look at the performance. If you want to edit these, you will need to open the `inference_cfg.yaml` file (or click button in GUI). The options are:
+You can validate the tracking parameters. Namely, you can iteratively change the
+parameters, run `convert_detections2tracklets` then load them in the GUI
+(`refine_tracklets`) if you want to look at the performance. If you want to edit these,
+you will need to open the `inference_cfg.yaml` file (or click button in GUI). The
+options are:
```python
# Tracking:
@@ -503,8 +775,13 @@ from the final h5 file as was customary in single animal projects.
**Next, tracklets are stitched to form complete tracks with:
```python
-deeplabcut.stitch_tracklets(config_path, ['videofile_path'], videotype='mp4',
- shuffle=1, trainingsetindex=0)
+deeplabcut.stitch_tracklets(
+ config_path,
+ ['videofile_path'],
+ videotype='mp4',
+ shuffle=1,
+ trainingsetindex=0,
+)
```
Note that the base signature of the function is identical to `analyze_videos` and `convert_detections2tracklets`.
@@ -515,15 +792,42 @@ can be directly specified as follows:
```python
deeplabcut.stitch_tracklets(..., n_tracks=n)
```
+
In such cases, file columns will default to dummy animal names (ind1, ind2, ..., up to indn).
+#### API Docs
+
+````{admonition} Click the button to see API Docs for analyze_videos
+:class: dropdown
+```{eval-rst}
+.. include:: ./api/deeplabcut.analyze_videos.rst
+```
+````
+
+````{admonition} Click the button to see API Docs for convert_detections2tracklets
+:class: dropdown
+```{eval-rst}
+.. include:: ./api/deeplabcut.convert_detections2tracklets.rst
+```
+````
+
+````{admonition} Click the button to see API Docs for stitch_tracklets
+:class: dropdown
+```{eval-rst}
+.. include:: ./api/deeplabcut.stitch_tracklets.rst
+```
+````
+
### Using Unsupervised Identity Tracking:
-In Lauer et al. 2022 we introduced a new method to do unsupervised reID of animals. Here, you can use the tracklets to learn the identity of animals to enhance your tracking performance. To use the code:
+In Lauer et al. 2022 we introduced a new method to do unsupervised reID of animals.
+Here, you can use the tracklets to learn the identity of animals to enhance your
+tracking performance. To use the code:
```python
deeplabcut.transformer_reID(config, videos_to_analyze, n_tracks=None, videotype="mp4")
```
+
Note you should pass the n_tracks (number of animals) you expect to see in the video.
### Refine Tracklets:
@@ -549,7 +853,7 @@ Short demo:
+
```{Hint}
**Labeling Pitfalls: How Corruptions Affect Performance**
-(A) Illustration of two types of labeling errors. Top is ground truth, middle is missing a label at the tailbase, and bottom is if the labeler swapped the ear identity (left to right, etc.). (B) Using a small training dataset of 106 frames, how do the corruptions in (A) affect the percent of correct keypoints (PCK) on the test set as the distance to ground truth increases from 0 pixels (perfect prediction) to 20 pixels (larger error)? The x axis denotes the difference in the ground truth to the predicted location (RMSE in pixels), whereas the y axis is the fraction of frames considered accurate (e.g., z80% of frames fall within 9 pixels, even on this small training dataset, for points that are not corrupted, whereas for swapped points this falls to z65%). The fraction of the dataset that is corrupted affects this value. Shown is when missing the tailbase label (top) or swapping the ears in 1%, 5%, 10%, and 20% of frames (of 106 labeled training images). Swapping versus missing labels has a more notable adverse effect on network performance.
+(A) Illustration of two types of labeling errors. Top is ground truth, middle is missing a label at the tailbase, and
+bottom is if the labeler swapped the ear identity (left to right, etc.). (B) Using a small training dataset of 106
+frames, how do the corruptions in (A) affect the percent of correct keypoints (PCK) on the test set as the distance
+to ground truth increases from 0 pixels (perfect prediction) to 20 pixels (larger error)? The x axis denotes the
+difference in the ground truth to the predicted location (RMSE in pixels), whereas the y axis is the fraction of
+frames considered accurate (e.g., z80% of frames fall within 9 pixels, even on this small training dataset, for
+points that are not corrupted, whereas for swapped points this falls to z65%). The fraction of the dataset that is
+corrupted affects this value. Shown is when missing the tailbase label (top) or swapping the ears in 1%, 5%, 10%,
+and 20% of frames (of 106 labeled training images). Swapping versus missing labels has a more notable adverse effect
+on network performance.
```
-The DeepLabCut toolbox supports **active learning** by extracting outlier frames be several methods and allowing the user to correct the frames, then retrain the model. See the [Nature Protocols paper](https://www.nature.com/articles/s41596-019-0176-0) for the detailed steps, or in the docs, [here](https://deeplabcut.github.io/DeepLabCut/docs/standardDeepLabCut_UserGuide.html#m-optional-active-learning-network-refinement-extract-outlier-frames).
+The DeepLabCut toolbox supports **active learning** by extracting outlier frames be several methods and allowing the
+user to correct the frames, then retrain the model. See the
+[Nature Protocols paper](https://www.nature.com/articles/s41596-019-0176-0) for the detailed steps, or in the docs,
+[here](active-learning).
-To facilitate this process, here we propose a new way to detect 'outlier frames', which is planned to be released in ~Sept 2022. Your contributions and suggestions are welcomed, so test the [PR](https://github.com/DeepLabCut/napari-deeplabcut/pull/38) and give us feedback!
+To facilitate this process, here we propose a new way to detect 'outlier frames'.
+Your contributions and suggestions are welcomed, so test the
+[PR](https://github.com/DeepLabCut/napari-deeplabcut/pull/38) and give us feedback!
-This #cookbook recipe aims to show a usecase of **clustering in napari** and is contributed by 2022 DLC AI Resident [Sabrina Benas](https://twitter.com/Sabrineiitor) 💜.
+This #cookbook recipe aims to show a usecase of **clustering in napari** and is contributed by 2022 DLC AI Resident
+[Sabrina Benas](https://twitter.com/Sabrineiitor) 💜.
## Detect Outliers to Refine Labels
### Open `napari` and the `DeepLabCut plugin`
- - Then open your `CollectedData_
+
### Clustering
-- Click on the button `cluster` and wait a few seconds until it displays a new layer with the cluster:
+Click on the button `cluster` and wait a few seconds until it displays a new layer with the cluster:
- 
+
You can click on a point and see the image on the right with the keypoints:
- 
+
### Visualize & refine
-If you decided to refine that frame (we moved the points to make outliers obvious), click `show img` and refine them using the plugin features and instructions:
+If you decided to refine that frame (we moved the points to make outliers obvious), click `show img` and refine them
+using the plugin features and instructions:
- 
+
- ```{Attention}
- When you're done, you need to click `ctl-s` to save it.
+```{Attention}
+When you're done, you need to click `ctl-s` to save it.
```
-- You can go back to the cluster layer by clicking on `close img` and refine another image. Reminder, when you're done editing you need to click `ctl-s` to save your work. And now you can take the updated `CollectedData` file, create and **new training shuffle**, and train the network! Read more about how to [create a training dataset](https://deeplabcut.github.io/DeepLabCut/docs/standardDeepLabCut_UserGuide.html#f-create-training-dataset-s).
+You can go back to the cluster layer by clicking on `close img` and refine another image. Reminder, when you're done
+editing you need to click `ctl-s` to save your work. And now you can take the updated `CollectedData` file, create
+and **new training shuffle**, and train the network! Read more about how to
+[create a training dataset](create-training-dataset).
```{hint}
-If you want to change the clustering method, you can modify the file [kmeans.py](https://github.com/DeepLabCutAIResidency/napari-deeplabcut/blob/cluster1/src/napari_deeplabcut/kmeans.py)
+If you want to change the clustering method, you can modify the file
+[kmeans.py](https://github.com/DeepLabCutAIResidency/napari-deeplabcut/blob/cluster1/src/napari_deeplabcut/kmeans.py)
+```
::::{important}
-You have to keep the way the file is opened (pandas dataframe) and the output has to be the cluster points, the points colors in the cluster colors and the frame names (in this order).
+You have to keep the way the file is opened (pandas dataframe) and the output has to be the cluster points, the points
+colors in the cluster colors and the frame names (in this order).
::::
```
diff --git a/docs/recipes/DLCMethods.md b/docs/recipes/DLCMethods.md
index b2f7a5a7af..74d5ec4c65 100644
--- a/docs/recipes/DLCMethods.md
+++ b/docs/recipes/DLCMethods.md
@@ -2,7 +2,13 @@
**Pose estimation using DeepLabCut**
-For body part tracking we used DeepLabCut (version 2.X.X) [Mathis et al, 2018, Nath et al, 2019]. Specifically, we labeled X number of frames taken from X videos/animals (then X% was used for training (default is 95%). We used a X-based neural network (i.e., X = ResNet-50, ResNet-101, MobileNetV2-0.35, MobileNetV2-0.5, MobileNetV2-0.75, MobileNetV2-1, EfficientNet ..X, dlcrnet_ms5, etc.)*** with default parameters* for X number of training iterations. We validated with X number of shuffles, and found the test error was: X pixels, train: X pixels (image size was X by X). We then used a p-cutoff of X (i.e. 0.9) to condition the X,Y coordinates for future analysis. This network was then used to analyze videos from similar experimental settings.
+For body part tracking we used DeepLabCut (version 3.X.X) [Mathis et al, 2018, Nath et al, 2019]. Specifically, we
+labeled X number of frames taken from X videos/animals (then X% was used for training (default is 95%). We used a
+X-based neural network (i.e., X = ResNet-50, ResNet-101, MobileNetV2-0.35, MobileNetV2-0.5, MobileNetV2-0.75,
+MobileNetV2-1, EfficientNet ..X, dlcrnet_ms5, cspnext_s, dekr_w32, rtmpose_s, etc.)*** with default parameters* for X
+number of training iterations. We validated with X number of shuffles, and found the test error was: X pixels, train:
+X pixels (image size was X by X). We then used a p-cutoff of X (i.e. 0.9) to condition the X,Y coordinates for future
+analysis. This network was then used to analyze videos from similar experimental settings.
*If any defaults were changed in *`pose_config.yaml`*, mention them.
@@ -43,4 +49,5 @@ If you use ResNets, consider citing Insafutdinov et al 2016 & He et al 2016. If
> 770–778 (2016). URL https://arxiv.org/abs/
> 1512.03385.
-We also have the network graphic freely available on SciDraw.io if you'd like to use it! https://scidraw.io/drawing/290. If you use our DLC logo, please include the TM symbol, thank you!
+We also have the network graphic freely available on SciDraw.io if you'd like to use it! https://scidraw.io/drawing/290.
+If you use our DLC logo, please include the TM symbol, thank you!
diff --git a/docs/recipes/MegaDetectorDLCLive.md b/docs/recipes/MegaDetectorDLCLive.md
index 20e35a38fc..8af6f59d1b 100644
--- a/docs/recipes/MegaDetectorDLCLive.md
+++ b/docs/recipes/MegaDetectorDLCLive.md
@@ -14,7 +14,9 @@ MegaDetector detects an animal and generates a bounding box around the animal. T
## DeepLabCut-Live
-DeepLabCut-Live! is a real-time package for running DeepLabCut. However, you can also use it as a lighter-weight package for running DeeplabCut even if you don't need real-time. It's very useful to use in HPC or servers, or in Apps, as we do here. To read more, check out the [docs](https://deeplabcut.github.io/DeepLabCut/docs/deeplabcutlive.html).
+DeepLabCut-Live! is a real-time package for running DeepLabCut. However, you can also use it as a lighter-weight
+package for running DeeplabCut even if you don't need real-time. It's very useful to use in HPC or servers, or in Apps,
+as we do here. To read more, check out the [docs](deeplabcut-live).
### MegaDetector meets DeepLabCut
diff --git a/docs/recipes/OpenVINO.md b/docs/recipes/OpenVINO.md
index 2033045ce8..78ea18e82f 100644
--- a/docs/recipes/OpenVINO.md
+++ b/docs/recipes/OpenVINO.md
@@ -1,10 +1,15 @@
# Intel OpenVINO backend
+::::{warning}
+This feature is currently implemented for TensorFlow-based models only.
+::::
+
DeepLabCut provides an option to run deep learning model with [OpenVINO](https://github.com/openvinotoolkit/openvino) backend.
-To enable OpenVINO in your pipeline, use `use_openvino` flag of `analyze_videos` method with one of string values indicating device:
-* "CPU" - Use CPU. This is a default value.
-* "GPU" - Use iGPU (requires OpenCL to be installed). First launch might take some time for kernels initialization.
-* "MULTI:CPU,GPU" - Use CPU and GPU simultaneously. In most cases this option provides the best efficiency.
+To enable OpenVINO in your pipeline, use `use_openvino` flag of `analyze_videos` method with one of string values
+indicating device:
+* ```"CPU"``` - Use CPU. This is a default value.
+* ```"GPU"``` - Use GPU (requires OpenCL to be installed). First launch might take some time for kernels initialization.
+* ```"MULTI:CPU,GPU"``` - Use CPU and GPU simultaneously. In most cases this option provides the best efficiency.
```python
def analyze_videos(
diff --git a/docs/recipes/OtherData.md b/docs/recipes/OtherData.md
index 1d9e648d70..73343284b3 100644
--- a/docs/recipes/OtherData.md
+++ b/docs/recipes/OtherData.md
@@ -9,12 +9,12 @@ Some users may have annotation data in different formats, yet want to use the DL
Here is a guide to do this via the ".csv" route: (the pandas array route is identical, just format the pandas array in the same way).
-**Step 1**: create a project as describe in the user guide: https://github.com/AlexEMG/DeepLabCut/blob/master/docs/UseOverviewGuide.md#create-a-new-project
+**Step 1**: create a project as describe in the user guide: https://github.com/DeepLabCut/DeepLabCut/blob/main/docs/UseOverviewGuide.md#create-a-new-project
**Step 2**: edit the ``config.yaml`` file to include the body part names, please take care that spelling, spacing, and capitalization are IDENTICAL to the "labeled data body part names".
-**Step 3**: Please inspect the excel formatted sheet (.csv) from our [demo project](https://github.com/AlexEMG/DeepLabCut/tree/master/examples/Reaching-Mackenzie-2018-08-30/labeled-data/reachingvideo1)
-- i.e. this file: https://github.com/AlexEMG/DeepLabCut/blob/master/examples/Reaching-Mackenzie-2018-08-30/labeled-data/reachingvideo1/CollectedData_Mackenzie.csv
+**Step 3**: Please inspect the excel formatted sheet (.csv) from our [demo project](https://github.com/DeepLabCut/DeepLabCut/tree/main/examples/Reaching-Mackenzie-2018-08-30/labeled-data/reachingvideo1)
+- i.e. this file: https://github.com/DeepLabCut/DeepLabCut/blob/main/examples/Reaching-Mackenzie-2018-08-30/labeled-data/reachingvideo1/CollectedData_Mackenzie.csv
**Step 4**: Edit the .csv file such that it contains the X, Y pixel coordinates, the body part names, the scorer name as well as the relative path to the image: e.g. /labeled-data/somefolder/img017.jpg
Then make sure the scorer name, and body parts are the same in the config.yaml file.
diff --git a/docs/recipes/TechHardware.md b/docs/recipes/TechHardware.md
index f4610f5442..879fc67dfb 100644
--- a/docs/recipes/TechHardware.md
+++ b/docs/recipes/TechHardware.md
@@ -23,10 +23,20 @@ The software is very robust to track data from any camera (cell phone cameras, g
**Anaconda/Python3:** Anaconda: a free and open source distribution of the Python programming language (download from https://www.anaconda.com/). DeepLabCut is written in Python 3 (https://www.python.org/) and not compatible with Python 2.
-
-**TensorFlow** You will need [TensorFlow](https://www.tensorflow.org/) (we used version 1.0 in the paper, later versions also work with the provided code (we tested **TensorFlow versions 1.0 to 1.15, and 2.0 to 2.5**; we recommend TF2.5 now) for Python 3.7, 3.8, or 3.9 with GPU support.
-
-To note, is it possible to run DeepLabCut on your CPU, but it will be VERY slow (see: [Mathis & Warren](https://www.biorxiv.org/content/early/2018/10/30/457242)). However, this is the preferred path if you want to test DeepLabCut on your own computer/data before purchasing a GPU, with the added benefit of a straightforward installation! Otherwise, use our COLAB notebooks for GPU access for testing.
-
-Docker: We highly recommend advaced users use the supplied [Docker container](https://github.com/MMathisLab/Docker4DeepLabCut2.0).
-NOTE: [this container does not work on windows hosts!](https://github.com/NVIDIA/nvidia-docker/issues/43)
+**For the TensorFlow Engine:** You will need [TensorFlow](https://www.tensorflow.org/).
+We used version 1.0 in the paper, later versions also work with the provided code (we
+tested **TensorFlow versions 1.0 to 1.15, and 2.0 to 2.12 (2.10 for Windows)**; we
+recommend TF2.12 for MacOS/Ubuntu and 2.10 for Windows) for Python 3.10 with GPU
+support.
+
+To note, is it possible to run DeepLabCut on your CPU, but it will be VERY slow (see:
+[Mathis & Warren](https://www.biorxiv.org/content/early/2018/10/30/457242)). However, this is the preferred path if you want to test
+DeepLabCut on your own computer/data before purchasing a GPU, with the added benefit of
+a straightforward installation! Otherwise, use our COLAB notebooks for GPU access for
+testing.
+
+Docker: We highly recommend advanced users use the supplied [Docker container](
+docker-containers).
+
+NOTE: [Currently GPU support in Docker Desktop is only available on Windows with the
+WSL2 backend.](https://docs.docker.com/desktop/features/gpu/)
diff --git a/docs/recipes/UsingModelZooPupil.md b/docs/recipes/UsingModelZooPupil.md
index 05d6a76d86..d73a1acbd7 100644
--- a/docs/recipes/UsingModelZooPupil.md
+++ b/docs/recipes/UsingModelZooPupil.md
@@ -1,17 +1,25 @@
# Using ModelZoo models on your own datasets
-Animal behavior has to be analyzed with painstaking accuracy. Therefore, animal pose estimation has been an important tool to study animal behavior precisely. +
Animal behavior has to be analyzed with painstaking accuracy. Therefore, animal pose estimation has been
+an important tool to study animal behavior precisely.
-Beside providing an open source toolbox for researchers to develop customized deep neural networks for markerless pose estimation, we at DeepLabCut also aim to build robust, generalizable models. Part of this effort is via the [DeeplabCut ModelZoo](http://www.mackenziemathislab.org/dlc-modelzoo).
+Beside providing an open source toolbox for researchers to develop customized deep neural networks for markerless pose
+estimation, we at DeepLabCut also aim to build robust, generalizable models. Part of this effort is via the
+[DeeplabCut ModelZoo](http://modelzoo.deeplabcut.org/).
-The Zoo hosts user-contributed and #teamDLC developed models that are trained on specific animals and scenarios. You can analyze your videos directly with these models without training. The models have strong zero-shot performance on unseen out-of-domain data which can be further improved via pseudo-labeling. Please check the first [ModelZoo manuscript](https://arxiv.org/abs/2203.07436v1) for further details.
+The Zoo hosts user-contributed and DLC-team developed models that are trained on specific animals and scenarios. You can
+analyze your videos directly with these models without training. The models have strong zero-shot performance on unseen
+out-of-domain data which can be further improved via pseudo-labeling. Please check the first
+[ModelZoo manuscript](https://arxiv.org/abs/2203.07436v1) for further details.
-This recipe aims to show a usecase of the **mouse_pupil_vclose** and is contributed by 2022 DLC AI Resident [Neslihan Wittek](https://github.com/neslihanedes) 💜.
+This recipe aims to show a usecase of the **mouse_pupil_vclose** and is contributed by 2022 DLC AI Resident
+[Neslihan Wittek](https://github.com/neslihanedes) 💜.
## `mouse_pupil_vclose` model
This model was contributed by Jim McBurney-Lin at University of California Riverside, USA.
-The model was trained on images of C57/B6J mice eyes, and also then augmented with mouse eye data from the Mathis Lab at EPFL.
+The model was trained on images of C57/B6J mice eyes, and also then augmented with mouse eye data from the Mathis Lab at
+EPFL.
@@ -30,23 +38,37 @@ The model was trained on images of C57/B6J mice eyes, and also then augmented wi
| 8 | VLpupil | Ventral/left aspect of pupil |
-Since we would like to evaluate the models performance on out-of-domain data, we will analyze pigeon pupils. For more discussions and work on so-called out-of-domain data, see [Mathis, Biasi 2020](http://www.mackenziemathislab.org/horse10).
+Since we would like to evaluate the models performance on out-of-domain data, we will analyze pigeon pupils. For more
+discussions and work on so-called out-of-domain data, see
+[Mathis, Biasi 2020](https://paperswithcode.com/dataset/horse-10).
## Pigeon Pupil
-The eye pupil admits and regulates the amount of light entering the retina in order to enable image perception. Beside this curicial role, the pupil also reflects the state of the brain. The systemic behavior of the pupil has not been vastly studied in birds, although researchers from Max Planck Institute for Ornithology in Seewiesen have shed light on pupil behaviors in pigeons.
+The eye pupil admits and regulates the amount of light entering the retina in order to enable image perception. Beside
+this curicial role, the pupil also reflects the state of the brain. The systemic behavior of the pupil has not been
+vastly studied in birds, although researchers from
+Max Planck Institute for Ornithology in Seewiesen
+have shed light on pupil behaviors in pigeons.
-The pupils of male pigeons get smaller during courtship behavior. This is in contrast to mammals, for which the pupil size dilates in response to an increase in arousal. In addition, the pupil size of pigeons dilates during non-REM sleep, while they rapidly constrict during REM sleep. Examining these differences and the reason behind them, might be helpful to understand the pupillary behavior in general.
+The pupils of male pigeons get smaller during courtship behavior. This is in contrast to mammals, for which the pupil
+size dilates in response to an increase in arousal. In addition, the pupil size of pigeons dilates during non-REM sleep,
+while they rapidly constrict during REM sleep. Examining these differences and the reason behind them, might be helpful
+to understand the pupillary behavior in general.
-In light of these findings, we wanted to show whether the **mouse_pupil_vclose** model give us an accurate tracking performance for the pigeon pupil as well.
+In light of these findings, we wanted to show whether the **mouse_pupil_vclose** model give us an accurate tracking
+performance for the pigeon pupil as well.
### Jupyter & Google Colab Notebook
-DeepLabCut provides a Google Colab Notebook to analyze your video with a pretrained networks from the ModelZoo. No need for local installation of DeepLabCut!
+DeepLabCut provides a Google Colab Notebook to analyze your video with a pretrained networks from the ModelZoo. No need
+for local installation of DeepLabCut!
-Since we are interested in the accuracy of the **mouse_pupil_vclose** on pigeon pupil data, we will use a video which consists of 7 recordings of pigeon pupils.
+Since we are interested in the accuracy of the **mouse_pupil_vclose** on pigeon pupil data, we will use a video which
+consists of 7 recordings of pigeon pupils.
-Check ModelZoo Colab page and a video tutorial on how to use the ModelZoo on Google Colab.
+Check the
+[ModelZoo Colab page](https://github.com/DeepLabCut/DeepLabCut/blob/main/examples/COLAB/COLAB_DLC_ModelZoo.ipynb)
+and a video tutorial on how to use the ModelZoo on Google Colab.
@@ -63,21 +85,23 @@ files.download("/content/file.zip")
### Analyze Videos at Your Local Machine
-DeepLabCut host models from the DeepLabCut ModelZoo Project.
+DeepLabCut host models from the [DeepLabCut ModelZoo Project](http://modelzoo.deeplabcut.org/).
The `create_pretrained_project` function will create a new project directory with the necessary sub-directories and a basic configuration file.
It will also initialize your project with a pre-trained model from the DeepLabCut ModelZoo.
The rest of the code should be run within your DeepLabCut environment.
-Check here for the instructions for the DeepLabCut installation.
+Check [here](how-to-install) for the instructions for the DeepLabCut installation.
+To initialize a new project directory with a pre-trained model from the DeepLabCut ModelZoo, run the code below.
+
+::::{warning}
+This method is currently implemented for Tensorflow only, Pytorch compatibility is coming soon.
+::::
```python
import deeplabcut
-```
-To initialize a new project directory with a pre-trained model from the DeepLabCut ModelZoo, run the code below.
-```python
deeplabcut.create_pretrained_project(
"projectname",
"experimenter",
@@ -93,6 +117,7 @@ deeplabcut.create_pretrained_project(
engine=deeplabcut.Engine.TF,
)
```
+
::::{important}
Your videos should be cropped around the eye for better model accuracy! 👁🐭
::::
@@ -101,13 +126,12 @@ Excitingly, 6 out of the 7 pigeon pupils were tracked nicely:
-
-When we further evaluate the model accuracy by checking the likelihood of tracked points, we see that the tracking is low confidience when the pigeons close their eyelid (which is of course expected, and can be leveraged to measure blinking 👁).
-
+When we further evaluate the model accuracy by checking the likelihood of tracked points, we see that the tracking is
+low confidience when the pigeons close their eyelid (which is of course expected, and can be leveraged to measure
+blinking 👁).
-
But you also might encounter larger problems than small tracking glitches:
@@ -118,12 +142,26 @@ The more problems you encounter, the higher the number of frames you might want
You should also add the path of the video(s) into the `config.yaml` file, or run the following command to add the videos to your project:
```python
-deeplabcut.add_new_videos('/pathofproject/config.yaml', ['/pathofvideos/pigeon.mp4'], copy_videos=False, coords=None, extract_frames=False)
+deeplabcut.add_new_videos(
+ "/pathofproject/config.yaml",
+ ["/pathofvideos/pigeon.mp4"],
+ copy_videos=False,
+ coords=None,
+ extract_frames=False
+)
```
The `deeplabcut.extract_outlier_frames` function will check for outliers and ask your feedback on whether to extract these outliers frames.
```python
-deeplabcut.extract_outlier_frames('/pathofproject/config.yaml', ['/pathofvideos/pigeon.mp4'], automatic=True)
+deeplabcut.analyze_videos(
+ "/pathofproject/config.yaml",
+ ["/pathofvideos/pigeon.mp4"]
+)
+deeplabcut.extract_outlier_frames(
+ "/pathofproject/config.yaml",
+ ["/pathofvideos/pigeon.mp4"],
+ automatic=True
+)
```
The `deeplabcut.refine_labels` function starts the GUI which allows you to refine the outlier frames manually.
You should load the outlier frames directory and corresponding `.h5` file from the previous model.
@@ -131,9 +169,9 @@ It will ask you to define the `likelihood` threshold: labels under the threshold
After refining, you should combine these data with your previous model's data set and create a new training data set.
```python
-deeplabcut.refine_labels('/pathofproject/config.yaml')
-deeplabcut.merge_datasets('/pathofproject/config.yaml')
-deeplabcut.create_training_dataset('/pathofproject/config.yaml')
+deeplabcut.refine_labels("/pathofproject/config.yaml")
+deeplabcut.merge_datasets("/pathofproject/config.yaml")
+deeplabcut.create_training_dataset("/pathofproject/config.yaml")
```
Before starting the training of your model, there is one last step left: editing the `init_weights` parameter in your `pose_cfg.yaml` file.
Go to your project and check the latest snapshot (e.g., `snapshot-610000`) of your model in `dlc-models/train` directory.
@@ -143,7 +181,7 @@ Edit the value of the `init_weights` key in the `pose_cfg.yaml` file and start t
`init_weights: pathofyourproject\dlc-models\iteration-0\DLCFeb31-trainset95shuffle1\train\snapshot-610000`
```python
-deeplabcut.train_network('/pathofproject/config.yaml', shuffle=1, saveiters=25000)
+deeplabcut.train_network("/pathofproject/config.yaml", shuffle=1, saveiters=25000)
```
```{hint}
Check this video for model refining!
diff --git a/docs/recipes/installTips.md b/docs/recipes/installTips.md
index 3864af7815..ab4565a880 100644
--- a/docs/recipes/installTips.md
+++ b/docs/recipes/installTips.md
@@ -3,24 +3,24 @@
## How to use the latest updates directly from GitHub
-We often update the master deeplabcut code base on github, and then ~1 a month we push out a stable release on pypi. This is what most users turn to on a daily basis (i.e. pypi is where you get your `pip install deeplabcut` code from! But, sometimes we add things to the repo that are not yet integrated, or you might want to edit the code yourself. Here, we show you how to do this.
+We often update the master deeplabcut code base on GitHub, and then ~1 a month we push out a stable release on pypi. This is what most users turn to on a daily basis (i.e. pypi is where you get your `pip install deeplabcut` code from! But, sometimes we add things to the repo that are not yet integrated, or you might want to edit the code yourself. Here, we show you how to do this.
### Method 1:
-If you want to *use* the latest, you can use pip and add the specific tags, such as `tf` or `gui`, etc. by modifying and running:
+If you want to *use* the latest, you can use pip and add the specific tags, such as `gui`, etc. by modifying and running:
```
-pip install --upgrade 'git+https://github.com/deeplabcut/deeplabcut.git#egg=deeplabcut[tf]'
+pip install --upgrade 'git+https://github.com/deeplabcut/deeplabcut.git#egg=deeplabcut[gui]'
```
which will download and update deeplabcut, and any dependencies that don't match the new version. If you want to force upgrade all of the dependencies to the latest available versions, too, then use the additional `--upgrade-strategy eager`, i.e.:
```
-pip install --upgrade --upgrade-strategy eager 'git+https://github.com/deeplabcut/deeplabcut.git#egg=deeplabcut[tf,gui]'
+pip install --upgrade --upgrade-strategy eager 'git+https://github.com/deeplabcut/deeplabcut.git#egg=deeplabcut[gui]'
```
### Method 2:
-If you want to be able to *edi* the source code of DeepLabCut, i.e., maybe add a feature or fix a 🐛, then you need to "clone" the source code:
+If you want to be able to *edit* the source code of DeepLabCut, i.e., maybe add a feature or fix a 🐛, then you need to "clone" the source code:
**Step 1:**
@@ -54,6 +54,10 @@ If you make changes, you can also then utilize our test scripts. Run the desired
i.e., for example:
```
+# Testing with the PyTorch engine
+python testscript_pytorch_multi_animal.py
+
+# Testing with the TensorFlow engine
python testscript_multianimal.py
```
@@ -243,6 +247,7 @@ Share images, automate workflows, and more with a free Docker ID:
For more examples and ideas, visit:
https://docs.docker.com/get-started/
```
+
### Next, Anaconda!
Click here to get the ubuntu/linux package: https://www.anaconda.com/products/individual#linux
@@ -283,6 +288,11 @@ Follow prompts!
## Troubleshooting: Note, if you get a failed build due to wxPython (note, this does not happen on Ubuntu 18, 16, etc), i.e.:
+```{warning}
+DeepLabCut no longer uses `wxpython` for its GUI - if you're getting such an error,
+you're likely installing an old version of DeepLabCut.
+```
+
```python
ERROR: Command errored out with exit status 1: /home/mackenzie/anaconda3/envs/DLC-GPU/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0jsmkrr1/wxpython_aeff462b2060421a9cf65df55f63a126/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0jsmkrr1/wxpython_aeff462b2060421a9cf65df55f63a126/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-pzy9q5u2/install-record.txt --single-version-externally-managed --compile --install-headers /home/mackenzie/anaconda3/envs/DLC-GPU/include/python3.7m/wxpython Check the logs for full command output.
@@ -313,11 +323,11 @@ Activate! `conda activate DEEPLABCUT` and then run: `conda install -c conda-forg
Then run `python -m deeplabcut` which launches the DLC GUI.
-## DeepLabCut MacOS M1 and M2 chip installation environment instructions:
-
-This only assumes you have anaconda installed.
+## DeepLabCut MacOS M-chip installation environment instructions:
-Use the `DEEPLABCUT_M1.yaml` conda file if you have an Macbok with an M1 or M2 chip, and follow these steps:
+This only assumes you have anaconda installed. Use the `DEEPLABCUT_M1.yaml` conda file
+if you have a newer MacBook (with an M1, M2, M3, M4 chip or more later), and follow
+these steps:
(1) git clone the deeplabcut cut repo:
@@ -330,17 +340,24 @@ git clone https://github.com/DeepLabCut/DeepLabCut.git
(3) Then, run:
```bash
-conda env create -f DEEPLABCUT_M1.yaml
+conda env create -f DEEPLABCUT.yaml
```
(4) Finally, activate your environment and to launch DLC with the GUI
```bash
-conda activate DEEPLABCUT_M1
+conda activate DEEPLABCUT
python -m deeplabcut
```
-The GUI will open. Of course, you can also run DeepLabCut in headless mode.
+The GUI will open. Of course, you can also run DeepLabCut in headless mode.
+
+If **you want to use the TensorFlow engine**, you'll need to install the `apple_mchips`
+extra with DeepLabCut. You can do so by running:
+
+```bash
+pip install deeplabcut[apple_mchips]
+```
## How to confirm that your GPU is being used by DeepLabCut
@@ -364,7 +381,8 @@ During training and analysis steps, DeepLabCut does not use the GPU processor he
(5) If you don't see activity there during training, then your GPU is likely not installed correctly for DeepLabCut. Return to the installation instructions, and be sure you installed CUDA 11+, and ran `conda install cudnn -c conda-forge` after installing DeepLabCut.
-## How to install DeepLabCut for Intel and AMD GPUs on Windows
+## How to install DeepLabCut for Intel and AMD GPUs on Windows for the TensorFlow engine
+
If you are on Windows 10/11 and have a DirectX 12 compatible GPU from any vendor (AMD, Intel, or Nvidia), you utilise GPU acceleration for inference, with an installation that is consistent between devices. This method uses [Tensorflow-directml](https://github.com/microsoft/tensorflow-directml) which uses DirectML instead of Cuda for ML training and inference.
To check the DirectX version of your installed GPU, type in dxdiag into windows search and select the run command. In system information, the bottom item of the list shows your DirectX version. In addition to this ensure your standard GPU drivers are up-to-date. Updating drivers by any official means (Nvidia Geforce experience, AMD radeon software, direct from the vendor website) is fine.
diff --git a/docs/recipes/nn.md b/docs/recipes/nn.md
index 75b44e315f..39ff354938 100644
--- a/docs/recipes/nn.md
+++ b/docs/recipes/nn.md
@@ -1,15 +1,16 @@
+(tf-training-tips-and-tricks)=
# Model training tips & tricks
-## Limiting a GPU's memory consumption
+## TensorFlow Engine: Limiting a GPU's memory consumption
-All GPU memory is allocated to training by default, preventing
+With TensorFlow, all GPU memory is allocated to training by default, preventing
other Tensorflow processes from being run on the same machine.
-A flexible solution to limiting memory usage is to call `deeplabcut.train(..., allow_growth=True)`,
-which dynamically grows the GPU memory region as it is needed.
-Another, stricter option is to explicitly cap GPU usage to only a fraction
-of the available memory. For example, allocating a maximum of 1/4 of the total
-memory could be done as follows:
+A flexible solution to limiting memory usage is to call
+`deeplabcut.train(..., allow_growth=True)`, which dynamically grows the GPU memory
+region as it is needed. Another, stricter option is to explicitly cap GPU usage to only
+a fraction of the available memory. For example, allocating a maximum of 1/4 of the
+total memory could be done as follows:
```python
import tensorflow as tf
@@ -18,6 +19,7 @@ gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.25)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))
```
+(tf-custom-image-augmentation)=
## Using custom image augmentation
Image augmentation is the process of artificially expanding the training set
@@ -25,89 +27,104 @@ by applying various transformations to images (e.g., rotation or rescaling)
in order to make models more robust and more accurate (read our
[primer](https://www.sciencedirect.com/science/article/pii/S0896627320307170) for
more information). Although data augmentation is automatically accomplished
-by DeepLabCut, default values (see the augmentation variables in the
-[default pose_cfg.yaml](https://github.com/DeepLabCut/DeepLabCut/blob/master/deeplabcut/pose_cfg.yaml#L23-L74) file)
-can be readily overwritten prior to training.
+by DeepLabCut, default values can be readily overwritten prior to training. See the
+augmentation variables defined in the:
-Another option we discuss is a different data-efficient approach based on a method called active learning. See this [this blog post](https://github.com/DeepLabCut/DeepLabCut/blob/master/docs/recipes/nn.md#using-custom-image-augmentation) for further details.
+- PyTorch Engine: [docs for the `pytorch_config.yaml` file](dlc3-pytorch-config)
+- TensorFlow Engine: [default pose_cfg.yaml file](
+https://github.com/DeepLabCut/DeepLabCut/blob/main/deeplabcut/pose_cfg.yaml#L23-L74)
-When you `create_training_dataset` [you have several options](https://github.com/DeepLabCut/DeepLabCut/wiki/DOCSTRINGS#create_training_dataset) on what types of augmentation to use.
-```python
-deeplabcut.create_training_dataset(configpath, augmenter_type='imgaug')
-```
-
-When you do this (i.e. pass `augmenter_type`) what underlying files you are calling are these:
-https://github.com/DeepLabCut/DeepLabCut/tree/master/deeplabcut/pose_estimation_tensorflow/datasets
-You can look at what types of augmentation are available to you (or edit those files to add more). Moreover, you can add more options to the pose_cfg.yaml file. Here is a simple script you can modify and run to automatically edit the correct pose_cfg.yaml to add more augmentation to the `imgaug` loader (or open it and edit yourself).
-
-But, you can add more:
+For the single-animal TensorFlow models, [you have several options](
+https://deeplabcut.github.io/DeepLabCut/docs/standardDeepLabCut_UserGuide.html#f-create-training-dataset-s-and-selection-of-your-neural-network)
+for image augmentation when calling `create_training_dataset`
-```python
-import deeplabcut
-
-train_pose_config, _ = deeplabcut.return_train_network_path(config_path)
-augs = {
- "gaussian_noise": True,
- "elastic_transform": True,
- "rotation": 180,
- "covering": True,
- "motion_blur": True,
-}
-deeplabcut.auxiliaryfunctions.edit_config(
- train_pose_config,
- augs,
-)
-```
-An in-depth tutorial on image augmentation and training hyperparameters can be found [here](https://deeplabcut.github.io/DeepLabCut/docs/recipes/pose_cfg_file_breakdown.html).
+An in-depth tutorial on image augmentation and training hyperparameters can be found [
+here](
+https://deeplabcut.github.io/DeepLabCut/docs/recipes/pose_cfg_file_breakdown.html).
## Evaluating intermediate (and all) snapshots
-The latest snapshot stored during training may not necessarily be the one that yields the highest performance. Therefore, you should analyze ALL snapshots, and select the best. Put 'all' in the snapshots section of the config.yaml to do this.
+The latest snapshot stored during training may not necessarily be the one that yields
+the highest performance. Therefore, you should analyze ALL snapshots, and select the
+best. Put 'all' in the snapshots section of the `config.yaml` to do this.
+(what-neural-network-should-i-use)=
## What neural network should I use? (Trade offs, speed performance, and considerations)
-### With the release of even more network options, you now have to decide what to use! This additionally flexibility is hopefully helpful, but we want to give you some guidance on where to start.
+You always select the network type when you create a training data set: i.e., standard
+dlc: `deeplabcut.create_training_dataset(config, net_type=resnet_50)` , or maDLC:
+`deeplabcut.create_multianimaltraining_dataset(config, net_type=dlcrnet_ms5)`. There is
+nothing else you should change.
+### PyTorch Engine
-**TL;DR - your best performance for most everything is ResNet-50; MobileNetV2-1 is much faster, needs less memory on your GPU to train and nearly as accurate.**
+The different architectures available are described in the [PyTorch model architectures
+](dlc3-architectures) page.
-You always select the network type when you create a training data set: i.e., standard dlc: `deeplabcut.create_training_dataset(config, net_type=resnet_50)` , or maDLC: `deeplabcut.create_multianimaltraining_dataset(config, net_type=dlcrnet_ms5)`. There is nothing else you should change.
+### TensorFlow Engine
+With the release of even more network options, you now have to decide what to use! This
+additionally flexibility is hopefully helpful, but we want to give you some guidance on
+where to start.
+
+**TL;DR - your best performance for most everything is ResNet-50; MobileNetV2-1 is much
+faster, needs less memory on your GPU to train and nearly as accurate.**
***
-## ResNets:
+#### ResNets:
-In Mathis et al. 2018 we benchmarked three networks: **ResNet-50, ResNet-101, and ResNet-101ws**. For ALL lab applications, ResNet-50 was enough. For all the demo videos on [www.deeplabcut.org](http://www.mousemotorlab.org/deeplabcut) the backbones are ResNet-50's. Thus, we recommend making this your go-to workhorse for data analysis. Here is a figure from the paper, see panel "B" (they are all within a few pixels of each other on the open-field dataset):
+In Mathis et al. 2018 we benchmarked three networks: **ResNet-50, ResNet-101, and
+ResNet-101ws**. For ALL lab applications, ResNet-50 was enough. For all the demo videos
+on [www.deeplabcut.org](http://www.mousemotorlab.org/deeplabcut) the backbones are
+ResNet-50's. Thus, we recommend making this your go-to workhorse for data analysis. Here
+is a figure from the paper, see panel "B" (they are all within a few pixels of each
+other on the open-field dataset):
@@ -117,14 +134,26 @@ read more here: https://arxiv.org/abs/1909.11229
-As a reminder, the core functions are described in our [Nature Protocols](https://www.nature.com/articles/s41596-019-0176-0) paper (published at the time of 2.0.6). Additional functions and features are continually added to the package. Thus, we recommend you read over the protocol and then please look at the following documentation and the doctrings. Thanks for using DeepLabCut!
+As a reminder, the core functions are described in our
+[Nature Protocols paper](https://www.nature.com/articles/s41596-019-0176-0) (published at the time of 2.0.6).
+Additional functions and features are continually added to the package. Thus, we recommend you read over the protocol
+and then please look at the following documentation and the doctrings. Thanks for using DeepLabCut!
## DeepLabCut in the Terminal/Command line interface:
-To begin, navigate to Aanaconda Prompt Terminal and right-click to "open as admin "(Windows), or simply launch "Terminal" (unix/MacOS) on your computer. We assume you have DeepLabCut installed (if not, see Install docs!). Next, launch your conda env (i.e., for example `conda activate DEEPLABCUT`) and then type `ipython`. Then type `import deeplabcut`.
+To begin, navigate to Anaconda Prompt Terminal and right-click to "open as admin "(Windows), or simply launch
+"Terminal" (unix/MacOS) on your computer. We assume you have DeepLabCut installed (if not, see Install docs!). Next,
+launch your conda env (i.e., for example `conda activate DEEPLABCUT`) and then type `ipython`. Then type:
+```python
+import deeplabcut
+```
```{Hint}
🚨 If you use Windows, please always open the terminal with administrator privileges! Right click, and "run as administrator".
@@ -32,48 +46,89 @@ To begin, navigate to Aanaconda Prompt Terminal and right-click to "open as admi
### (A) Create a New Project
-The function **create\_new\_project** creates a new project directory, required subdirectories, and a basic project configuration file. Each project is identified by the name of the project (e.g. Reaching), name of the experimenter (e.g. YourName), as well as the date at creation.
+The function `create_new_project` creates a new project directory, required subdirectories, and a basic project
+configuration file. Each project is identified by the name of the project (e.g. Reaching), name of the experimenter
+(e.g. YourName), as well as the date at creation.
-Thus, this function requires the user to input the name of the project, the name of the experimenter, and the full path of the videos that are (initially) used to create the training dataset.
+Thus, this function requires the user to input the name of the project, the name of the experimenter, and the full
+path of the videos that are (initially) used to create the training dataset.
-Optional arguments specify the working directory, where the project directory will be created, and if the user wants to copy the videos (to the project directory). If the optional argument working\_directory is unspecified, the project directory is created in the current working directory, and if copy\_videos is unspecified symbolic links for the videos are created in the videos directory. Each symbolic link creates a reference to a video and thus eliminates the need to copy the entire video to the video directory (if the videos remain at the original location).
+Optional arguments specify the working directory, where the project directory will be created, and if the user wants
+to copy the videos (to the project directory). If the optional argument `working_directory` is unspecified, the
+project directory is created in the current working directory, and if `copy_videos` is unspecified symbolic links
+for the videos are created in the videos directory. Each symbolic link creates a reference to a video and thus
+eliminates the need to copy the entire video to the video directory (if the videos remain at the original location).
```python
-deeplabcut.create_new_project('Name of the project', 'Name of the experimenter', ['Full path of video 1', 'Full path of video2', 'Full path of video3'], working_directory='Full path of the working directory', copy_videos=True/False, multianimal=True/False)
+deeplabcut.create_new_project(
+ "Name of the project",
+ "Name of the experimenter",
+ ["Full path of video 1", "Full path of video2", "Full path of video3"],
+ working_directory="Full path of the working directory",
+ copy_videos=True/False,
+ multianimal=False
+)
```
**Important path formatting note**
-Windows users, you must input paths as: ``r'C:\Users\computername\Videos\reachingvideo1.avi' `` or
-
-`` 'C:\\Users\\computername\\Videos\\reachingvideo1.avi'``
-
- (TIP: you can also place ``config_path`` in front of ``deeplabcut.create_new_project`` to create a variable that holds the path to the config.yaml file, i.e. ``config_path=deeplabcut.create_new_project(...)``)
-
-
-This set of arguments will create a project directory with the name **Name of the project+name of the experimenter+date of creation of the project** in the **Working directory** and creates the symbolic links to videos in the **videos** directory. The project directory will have subdirectories: **dlc-models**, **labeled-data**, **training-datasets**, and **videos**. All the outputs generated during the course of a project will be stored in one of these subdirectories, thus allowing each project to be curated in separation from other projects. The purpose of the subdirectories is as follows:
-
-**dlc-models:** This directory contains the subdirectories *test* and *train*, each of which holds the meta information with regard to the parameters of the feature detectors in configuration files. The configuration files are YAML files, a common human-readable data serialization language. These files can be opened and edited with standard text editors. The subdirectory *train* will store checkpoints (called snapshots in TensorFlow) during training of the model. These snapshots allow the user to reload the trained model without re-training it, or to pick-up training from a particular saved checkpoint, in case the training was interrupted.
-
-**labeled-data:** This directory will store the frames used to create the training dataset. Frames from different videos are stored in separate subdirectories. Each frame has a filename related to the temporal index within the corresponding video, which allows the user to trace every frame back to its origin.
-
-**training-datasets:** This directory will contain the training dataset used to train the network and metadata, which contains information about how the training dataset was created.
-
-**videos:** Directory of video links or videos. When **copy\_videos** is set to ``False``, this directory contains symbolic links to the videos. If it is set to ``True`` then the videos will be copied to this directory. The default is ``False``. Additionally, if the user wants to add new videos to the project at any stage, the function **add\_new\_videos** can be used. This will update the list of videos in the project's configuration file.
+Windows users, you must input paths as: `r'C:\Users\computername\Videos\reachingvideo1.avi'` or
+` 'C:\\Users\\computername\\Videos\\reachingvideo1.avi'`
+
+TIP: you can also place `config_path` in front of `deeplabcut.create_new_project` to create a variable that holds
+the path to the config.yaml file, i.e. `config_path=deeplabcut.create_new_project(...)`
+
+This set of arguments will create a project directory with the name
+**
-
-
-Next, open the **config.yaml** file, which was created during **create\_new\_project**. You can edit this file in any text editor. Familiarize yourself with the meaning of the parameters (Box 1). You can edit various parameters, in particular you **must add the list of *bodyparts* (or points of interest)** that you want to track. You can also set the *colormap* here that is used for all downstream steps (can also be edited at anytime), like labeling GUIs, videos, etc. Here any [matplotlib colormaps](https://matplotlib.org/tutorials/colors/colormaps.html) will do!
+Next, open the **config.yaml** file, which was created during **create\_new\_project**. You can edit this file in any
+text editor. Familiarize yourself with the meaning of the parameters (Box 1). You can edit various parameters, in
+particular you **must add the list of *bodyparts* (or points of interest)** that you want to track. You can also set the
+*colormap* here that is used for all downstream steps (can also be edited at anytime), like labeling GUIs, videos, etc.
+Here any [matplotlib colormaps](https://matplotlib.org/tutorials/colors/colormaps.html) will do!
Please DO NOT have spaces in the names of bodyparts.
**bodyparts:** are the bodyparts of each individual (in the above list).
- ### (C) Data Selection (extract frames)
-
-**CRITICAL:** A good training dataset should consist of a sufficient number of frames that capture the breadth of the behavior. This ideally implies to select the frames from different (behavioral) sessions, different lighting and different animals, if those vary substantially (to train an invariant, robust feature detector). Thus for creating a robust network that you can reuse in the laboratory, a good training dataset should reflect the diversity of the behavior with respect to postures, luminance conditions, background conditions, animal identities,etc. of the data that will be analyzed. For the simple lab behaviors comprising mouse reaching, open-field behavior and fly behavior, 100−200 frames gave good results [Mathis et al, 2018](https://www.nature.com/articles/s41593-018-0209-y). However, depending on the required accuracy, the nature of behavior, the video quality (e.g. motion blur, bad lighting) and the context, more or less frames might be necessary to create a good network. Ultimately, in order to scale up the analysis to large collections of videos with perhaps unexpected conditions, one can also refine the data set in an adaptive way (see refinement below).
-
-The function `extract_frames` extracts frames from all the videos in the project configuration file in order to create a training dataset. The extracted frames from all the videos are stored in a separate subdirectory named after the video file’s name under the ‘labeled-data’. This function also has various parameters that might be useful based on the user’s need.
+ ### (C) Select Frames to Label
+
+**CRITICAL:** A good training dataset should consist of a sufficient number of frames that capture the breadth of the
+behavior. This ideally implies to select the frames from different (behavioral) sessions, different lighting and
+different animals, if those vary substantially (to train an invariant, robust feature detector). Thus for creating a
+robust network that you can reuse in the laboratory, a good training dataset should reflect the diversity of the
+behavior with respect to postures, luminance conditions, background conditions, animal identities,etc. of the data that
+will be analyzed. For the simple lab behaviors comprising mouse reaching, open-field behavior and fly behavior, 100−200
+frames gave good results [Mathis et al, 2018](https://www.nature.com/articles/s41593-018-0209-y). However, depending on
+the required accuracy, the nature of behavior, the video quality (e.g. motion blur, bad lighting) and the context, more
+or less frames might be necessary to create a good network. Ultimately, in order to scale up the analysis to large
+collections of videos with perhaps unexpected conditions, one can also refine the data set in an adaptive way (see
+refinement below).
+
+The function `extract_frames` extracts frames from all the videos in the project configuration file in order to create
+a training dataset. The extracted frames from all the videos are stored in a separate subdirectory named after the video
+file’s name under the ‘labeled-data’. This function also has various parameters that might be useful based on the user’s
+need.
```python
-deeplabcut.extract_frames(config_path, mode='automatic/manual', algo='uniform/kmeans', userfeedback=False, crop=True/False)
+deeplabcut.extract_frames(
+ config_path,
+ mode="automatic/manual",
+ algo="uniform/kmeans",
+ crop=True/False,
+ userfeedback=False
+)
```
**CRITICAL POINT:** It is advisable to keep the frame size small, as large frames increase the training and
inference time. The cropping parameters for each video can be provided in the config.yaml file (and see below).
-When running the function extract_frames, if the parameter crop=True, then you will be asked to draw a box within the GUI (and this is written to the config.yaml file).
-
-`userfeedback` allows the user to check which videos they wish to extract frames from. In this way, if you added more videos to the config.yaml file it does not, by default, extract frames (again) from every video. If you wish to disable this question, set `userfeedback = True`.
-
-The provided function either selects frames from the videos that are randomly sampled from a uniform distribution (uniform), by clustering based on visual appearance (k-means), or by manual selection. Random
-selection of frames works best for behaviors where the postures vary across the whole video. However, some behaviors
-might be sparse, as in the case of reaching where the reach and pull are very fast and the mouse is not moving much
-between trials (thus, we have the default set to True, as this is best for most use-cases we encounter). In such a case, the function that allows selecting frames based on k-means derived quantization would
-be useful. If the user chooses to use k-means as a method to cluster the frames, then this function downsamples the
-video and clusters the frames using k-means, where each frame is treated as a vector. Frames from different clusters
-are then selected. This procedure makes sure that the frames look different. However, on large and long videos, this
-code is slow due to computational complexity.
+When running the function extract_frames, if the parameter crop=True, then you will be asked to draw a box within the
+GUI (and this is written to the config.yaml file).
+
+`userfeedback` allows the user to specify which videos they wish to extract frames from. When set to `"True"`, a dialog
+will be initiated, where the user is asked for each video if (additional/any) frames from this video should be
+extracted. Use this, e.g. if you have already labeled some folders and want to extract data for new videos.
+
+The provided function either selects frames from the videos that are randomly sampled from a uniform distribution
+(uniform), by clustering based on visual appearance (k-means), or by manual selection. Random uniform selection of
+frames works best for behaviors where the postures vary across the whole video. However, some behaviors might be sparse,
+as in the case of reaching where the reach and pull are very fast and the mouse is not moving much between trials. In
+such a case, the function that allows selecting frames based on k-means derived quantization would be useful. If the
+user chooses to use k-means as a method to cluster the frames, then this function downsamples the video and clusters the
+frames using k-means, where each frame is treated as a vector. Frames from different clusters are then selected. This
+procedure makes sure that the frames look different. However, on large and long videos, this code is slow due to
+computational complexity.
**CRITICAL POINT:** It is advisable to extract frames from a period of the video that contains interesting
behaviors, and not extract the frames across the whole video. This can be achieved by using the start and stop
@@ -122,13 +204,16 @@ parameters in the config.yaml file. Also, the user can change the number of fram
the numframes2extract in the config.yaml file.
However, picking frames is highly dependent on the data and the behavior being studied. Therefore, it is hard to
-provide all purpose code that extracts frames to create a good training dataset for every behavior and animal. If the user feels specific frames are lacking, they can extract hand selected frames of interest using the interactive GUI
+provide all purpose code that extracts frames to create a good training dataset for every behavior and animal. If the
+user feels specific frames are lacking, they can extract hand selected frames of interest using the interactive GUI
provided along with the toolbox. This can be launched by using:
```python
-deeplabcut.extract_frames(config_path, 'manual')
+deeplabcut.extract_frames(config_path, "manual")
```
The user can use the *Load Video* button to load one of the videos in the project configuration file, use the scroll
-bar to navigate across the video and *Grab a Frame* (or a range of frames, as of version 2.0.5) to extract the frame(s). The user can also look at the extracted frames and e.g. delete frames (from the directory) that are too similar before reloading the set and then manually annotating them.
+bar to navigate across the video and *Grab a Frame* (or a range of frames, as of version 2.0.5) to extract the frame(s).
+The user can also look at the extracted frames and e.g. delete frames (from the directory) that are too similar before
+reloading the set and then manually annotating them.
@@ -144,38 +229,42 @@ bar to navigate across the video and *Grab a Frame* (or a range of frames, as of
### (D) Label Frames
-The toolbox provides a function **label_frames** which helps the user to easily label all the extracted frames using
-an interactive graphical user interface (GUI). The user should have already named the body parts to label (points of
-interest) in the project’s configuration file by providing a list. The following command invokes the labeling toolbox.
+The toolbox provides a function **label_frames** which helps the user to easily label
+all the extracted frames using an interactive graphical user interface (GUI). The user
+should have already named the bodyparts to label (points of interest) in the
+project’s configuration file by providing a list. The following command invokes the
+napari-deeplabcut labelling GUI. Checkout the [napari-deeplabcut docs](napari-gui) for
+more information about the labelling workflow.
+
```python
deeplabcut.label_frames(config_path)
```
-The user needs to use the *Load Frames* button to select the directory which stores the extracted frames from one of
-the videos. Subsequently, the user can use one of the radio buttons (top right) to select a body part to label. RIGHT click to add the label. Left click to drag the label, if needed. If you label a part accidentally, you can use the middle button on your mouse to delete! If you cannot see a body part in the frame, skip over the label! Please see the ``HELP`` button for more user instructions. This auto-advances once you labeled the first body part. You can also advance to the next frame by clicking on the RIGHT arrow on your keyboard (and go to a previous frame with LEFT arrow).
-Each label will be plotted as a dot in a unique color.
-
-The user is free to move around the body part and once satisfied with its position, can select another radio button
-(in the top right) to switch to the respective body part (it otherwise auto-advances). The user can skip a body part if it is not visible. Once all the visible body parts are labeled, then the user can use ‘Next Frame’ to load the following frame. The user needs to save the labels after all the frames from one of the videos are labeled by clicking the save button at the bottom right. Saving the labels will create a labeled dataset for each video in a hierarchical data file format (HDF) in the
-subdirectory corresponding to the particular video in **labeled-data**. You can save at any intermediate step (even without closing the GUI, just hit save) and you return to labeling a dataset by reloading it!
-
-**CRITICAL POINT:** It is advisable to **consistently label similar spots** (e.g., on a wrist that is very large, try
-to label the same location). In general, invisible or occluded points should not be labeled by the user. They can
-simply be skipped by not applying the label anywhere on the frame.
-OPTIONAL: In the event of adding more labels to the existing labeled dataset, the user need to append the new
-labels to the bodyparts in the config.yaml file. Thereafter, the user can call the function **label_frames**. As of 2.0.5+: then a box will pop up and ask the user if they wish to display all parts, or only add in the new labels. Saving the labels after all the images are labelled will append the new labels to the existing labeled dataset.
+[🎥 DEMO](https://youtu.be/hsA9IB5r73E)
HOT KEYS IN THE Labeling GUI (also see "help" in GUI):
+
```
Ctrl + C: Copy labels from previous frame.
Keyboard arrows: advance frames.
Delete key: delete label.
```
+
+**CRITICAL POINT:** It is advisable to **consistently label similar spots** (e.g., on a wrist that is very large, try
+to label the same location). In general, invisible or occluded points should not be labeled by the user. They can
+simply be skipped by not applying the label anywhere on the frame.
+OPTIONAL: In the event of adding more labels to the existing labeled dataset, the user need to append the new
+labels to the bodyparts in the config.yaml file. Thereafter, the user can call the function **label_frames**. As of
+2.0.5+: then a box will pop up and ask the user if they wish to display all parts, or only add in the new labels.
+Saving the labels after all the images are labelled will append the new labels to the existing labeled dataset.
-### (E) Check Annotated Frames
+For more information, checkout the [napari-deeplabcut docs](napari-gui) for
+more information about the labelling workflow.
+
+### (E) Check Annotated Frames
OPTIONAL: Checking if the labels were created and stored correctly is beneficial for training, since labeling
is one of the most critical parts for creating the training dataset. The DeepLabCut toolbox provides a function
@@ -184,7 +273,10 @@ is one of the most critical parts for creating the training dataset. The DeepLab
deeplabcut.check_labels(config_path, visualizeindividuals=True/False)
```
-For each video directory in labeled-data this function creates a subdirectory with **labeled** as a suffix. Those directories contain the frames plotted with the annotated body parts. The user can double check if the body parts are labeled correctly. If they are not correct, the user can reload the frames (i.e. `deeplabcut.label_frames`), move them around, and click save again.
+For each video directory in labeled-data this function creates a subdirectory with **labeled** as a suffix. Those
+directories contain the frames plotted with the annotated body parts. The user can double check if the body parts are
+labeled correctly. If they are not correct, the user can reload the frames (i.e. `deeplabcut.label_frames`), move them
+around, and click save again.
#### API Docs
````{admonition} Click the button to see API Docs
@@ -194,99 +286,228 @@ For each video directory in labeled-data this function creates a subdirectory wi
```
````
-### (F) Create Training Dataset(s) and selection of your neural network
+(create-training-dataset)=
+### (F) Create Training Dataset
-**CRITICAL POINT:** Only run this step **where** you are going to train the network. If you label on your laptop but move your project folder to Google Colab or AWS, lab server, etc, then run the step below on that platform! If you labeled on a Windows machine but train on Linux, this is fine as of 2.0.4 onwards it will be done automatically (it saves file sets as both Linux and Windows for you).
+**CRITICAL POINT:** Only run this step **where** you are going to train the network. If you label on your laptop but
+move your project folder to Google Colab or AWS, lab server, etc, then run the step below on that platform! If you
+labeled on a Windows machine but train on Linux, this is fine as of 2.0.4 onwards it will be done automatically (it
+saves file sets as both Linux and Windows for you).
-- If you move your project folder, you must only change the `project_path` (which is done automatically) in the main config.yaml file - that's it - no need to change the video paths, etc! Your project is fully portable.
+- If you move your project folder, you must only change the `project_path` (which is done automatically) in the main
+config.yaml file - that's it - no need to change the video paths, etc! Your project is fully portable.
-- Be aware you select your neural network backbone at this stage. As of DLC3+ we support PyTorch (and TensorFlow, but this will be phased out).
+- Be aware you select your neural network backbone at this stage. As of DLC3+ we support PyTorch (and TensorFlow, but
+this will be phased out).
-**OVERVIEW:** This function combines the labeled datasets from all the videos and splits them to create train and test datasets. The training data will be used to train the network, while the test data set will be used for evaluating the network. The function **create_training_dataset** performs those steps.
+**OVERVIEW:** This function combines the labeled datasets from all the videos and splits them to create train and test
+datasets. The training data will be used to train the network, while the test data set will be used for evaluating the
+network.
```python
deeplabcut.create_training_dataset(config_path)
```
-- OPTIONAL: If the user wishes to benchmark the performance of the DeepLabCut, they can create multiple training datasets by specifying an integer value to the `num_shuffles`; see the docstring for more details.
-
-within **dlc-models** called ``test`` and ``train``, and these each have a configuration file called pose_cfg.yaml.
-Specifically, the user can edit the **pose_cfg.yaml** within the **train** subdirectory before starting the training. These configuration files contain meta information with regard to the parameters of the feature detectors. Key parameters are listed in Box 2.
+- OPTIONAL: If the user wishes to benchmark the performance of the DeepLabCut, they can create multiple training
+datasets by specifying an integer value to the `num_shuffles`; see the docstring for more details.
-**CRITICAL POINT:** At this step, for **create_training_dataset** you select the network you want to use, and any additional data augmentation (beyond our defaults). You can set ``net_type`` and ``augmenter_type`` when you call the function.
+The function creates a new shuffle(s) directory in the **dlc-models-pytorch** directory
+(**dlc-models** if using Tensorflow), in the current "iteration" directory.
+The `train` and `test` directories each have a configuration file
+(**pytorch_config.yaml** in **train** and **pose_cfg.yaml** in **test** for Pytorch models,
+**pose_cfg.yaml** in **train** and **test** for Tensorflow models).
+Specifically, the user can edit the **pytorch_config.yaml** (or **pose_cfg.yaml**) within the **train** subdirectory
+before starting the training. These configuration files contain meta information with regard to the parameters
+of the feature detectors. For more information about the **pytorch_config.yaml** file, see [here](dlc3-pytorch-config)
+(for TensorFlow-based models, see key parameters
+[here](https://github.com/DeepLabCut/DeepLabCut/blob/main/deeplabcut/pose_cfg.yaml)).
-- Networks: ImageNet pre-trained networks OR SuperAnimal pre-trained networks weights will be downloaded, as you select. You can decide to do transfer-learning (recommended) or "fine-tune" both the backbone and the decoder head. We suggest seeing our [dedicated documentation on models](https://deeplabcut.github.io/DeepLabCut/docs/pytorch/architectures.html) for more information.
+**CRITICAL POINT:** At this step, for **create_training_dataset** you select the network you want to use, and any
+additional data augmentation (beyond our defaults). You can set `net_type`, `detector_type` (if using a detector)
+and `augmenter_type` when you call the function.
+- Networks: ImageNet pre-trained networks OR SuperAnimal pre-trained networks weights will be downloaded, as you
+select. You can decide to do transfer-learning (recommended) or "fine-tune" both the backbone and the decoder head. We
+suggest seeing our [dedicated documentation on models](dlc3-architectures) for more information (
+or the [this page on selecting models](what-neural-network-should-i-use) for the TensorFlow engine).
```{Hint}
-🚨 If they do not download (you will see this downloading in the terminal), then you may not have permission to do so - be sure to open your terminal "as an admin" (This is only something we have seen with some Windows users - see the **[docs for more help!](https://deeplabcut.github.io/DeepLabCut/docs/recipes/nn.html)**).
-```
-
-**DATA AUGMENTATION:** At this stage you can also decide what type of augmentation to use. The default loaders work well for most all tasks (as shown on www.deeplabcut.org), but there are many options, more data augmentation, intermediate supervision, etc. Please look at the [**pose_cfg.yaml**](https://github.com/DeepLabCut/DeepLabCut/blob/master/deeplabcut/pose_cfg.yaml) file for a full list of parameters **you might want to change before running this step.** There are several data loaders that can be used. For example, you can use the default loader (introduced and described in the Nature Protocols paper), [TensorPack](https://github.com/tensorpack/tensorpack) for data augmentation (currently this is easiest on Linux only), or [imgaug](https://imgaug.readthedocs.io/en/latest/). We recommend `imgaug`. You can set this by passing:``` deeplabcut.create_training_dataset(config_path, augmenter_type='imgaug') ```
-
-**For TensorFlow Models:** the differences of the loaders are as follows:
-- `imgaug`: a lot of augmentation possibilities, efficient code for target map creation & batch sizes >1 supported. You can set the parameters such as the `batch_size` in the `pose_cfg.yaml` file for the model you are training. This is the recommended DEFAULT!
-- `crop_scale`: our standard DLC 2.0 introduced in Nature Protocols variant (scaling, auto-crop augmentation)
-- `tensorpack`: a lot of augmentation possibilities, multi CPU support for fast processing, target maps are created less efficiently than in imgaug, does not allow batch size>1
-- `deterministic`: only useful for testing, freezes numpy seed; otherwise like default.
-
-**For PyTorch Models:**
-- #TODO: more information coming soon; in the meantime see the docstrings!
-
-
-**MODEL COMPARISON:** You can also test several models by creating the same test/train split for different networks. You can easily do this in the Project Manager GUI, which also lets you compare PyTorch and TensorFlow models.
-
-Please also consult the [following page on selecting models]( https://deeplabcut.github.io/DeepLabCut/docs/recipes/nn.html#what-neural-network-should-i-use-trade-offs-speed-performance-and-considerations)
+🚨 If they do not download (you will see this downloading in the terminal), then you may not have permission to do
+so - be sure to open your terminal "as an admin" (This is only something we have seen with some Windows users - see
+the **[docs for more help!](tf-training-tips-and-tricks)**).
+```
+
+**DATA AUGMENTATION:** At this stage you can also decide what type of augmentation to
+use. Once you've called `create_training_dataset`, you can edit the
+[**pytorch_config.yaml**](dlc3-pytorch-config) file that was created (or for the
+TensorFlow engine, the [**pose_cfg.yaml**](
+https://github.com/DeepLabCut/DeepLabCut/blob/main/deeplabcut/pose_cfg.yaml) file).
+
+- PyTorch Engine: [Albumentations](https://albumentations.ai/docs/) is used for data
+augmentation. Look at the [**pytorch_config.yaml**](dlc3-pytorch-config) for more
+information about image augmentation options.
+- TensorFlow Engine: The default augmentation works well for most tasks (as shown on
+www.deeplabcut.org), but there are many options, more data augmentation, intermediate
+supervision, etc. Here are the available loaders:
+ - `imgaug`: a lot of augmentation possibilities, efficient code for target map creation & batch sizes >1 supported.
+ You can set the parameters such as the `batch_size` in the `pose_cfg.yaml` file for the model you are training. This
+ is the recommended default!
+ - `crop_scale`: our standard DLC 2.0 introduced in Nature Protocols variant (scaling, auto-crop augmentation)
+ - `tensorpack`: a lot of augmentation possibilities, multi CPU support for fast processing, target maps are created
+ less efficiently than in imgaug, does not allow batch size>1
+ - `deterministic`: only useful for testing, freezes numpy seed; otherwise like default.
+
+**MODEL COMPARISON**: You can also test several models by creating the same train/test
+split for different networks.
+You can easily do this in the Project Manager GUI (by selecting the "Use an existing
+data split" option), which also lets you compare PyTorch and TensorFlow models.
+
+````{versionadded} 3.0.0
+You can now create new shuffles using the same train/test split as
+existing shuffles with `create_training_dataset_from_existing_split`. This allows you to
+compare model performance (between different architectures or when using different
+training hyper-parameters) as the shuffles were trained on the same data, and evaluated
+on the same test data!
+
+Example usage - creating 3 new shuffles (with indices 10, 11 and 12) for a ResNet 50
+pose estimation model, using the same data split as was used for shuffle 0:
- See Box 2 on how to specify **which network is loaded for training (including your own network, etc):**
-
-
-
-
@@ -454,51 +730,88 @@ It creates a folder called ``plot-poses`` (in the directory of the video). The p
```
````
-### (L) Create Labeled Videos:
+### (L) Create Labeled Videos
Additionally, the toolbox provides a function to create labeled videos based on the extracted poses by plotting the
-labels on top of the frame and creating a video. There are two modes to create videos: FAST and SLOW (but higher quality!). If you want to create high-quality videos, please add ``save_frames=True``. One can use the command as follows to create multiple labeled videos:
+labels on top of the frame and creating a video. There are two modes to create videos: FAST and SLOW (but higher
+quality!). One can use the command as follows to create multiple labeled videos:
```python
-deeplabcut.create_labeled_video(config_path, ['fullpath/analysis/project/videos/reachingvideo1.avi','fullpath/analysis/project/videos/reachingvideo2.avi'], save_frames = True/False)
-```
- Optionally, if you want to use the filtered data for a video or directory of filtered videos pass ``filtered=True``, i.e.:
+deeplabcut.create_labeled_video(
+ config_path,
+ ["fullpath/analysis/project/videos/reachingvideo1.avi",
+ "fullpath/analysis/project/videos/reachingvideo2.avi"],
+ save_frames = True/False
+)
+```
+ Optionally, if you want to use the filtered data for a video or directory of filtered videos pass `filtered=True`,
+ i.e.:
```python
-deeplabcut.create_labeled_video(config_path, ['fullpath/afolderofvideos'], videotype='.mp4', filtered=True)
-```
-You can also optionally add a skeleton to connect points and/or add a history of points for visualization. To set the "trailing points" you need to pass ``trailpoints``:
+deeplabcut.create_labeled_video(
+ config_path,
+ ["fullpath/afolderofvideos"],
+ videotype=".mp4",
+ filtered=True
+)
+```
+You can also optionally add a skeleton to connect points and/or add a history of points for visualization. To set the
+"trailing points" you need to pass `trailpoints`:
```python
-deeplabcut.create_labeled_video(config_path, ['fullpath/afolderofvideos'], videotype='.mp4', trailpoints=10)
-```
-To draw a skeleton, you need to first define the pairs of connected nodes (in the ``config.yaml`` file) and set the skeleton color (in the ``config.yaml`` file). There is also a GUI to help you do this, use by calling `deeplabcut.SkeletonBuilder(config+path)`!
-
-Here is how the ``config.yaml`` additions/edits should look (for example, on the Openfield demo data we provide):
+deeplabcut.create_labeled_video(
+ config_path,
+ ["fullpath/afolderofvideos"],
+ videotype=".mp4",
+ trailpoints=10
+)
+```
+To draw a skeleton, you need to first define the pairs of connected nodes (in the `config.yaml` file) and set the
+skeleton color (in the `config.yaml` file). There is also a GUI to help you do this, use by calling
+`deeplabcut.SkeletonBuilder(configpath)`!
+
+Here is how the `config.yaml` additions/edits should look (for example, on the Openfield demo data we provide):
```python
# Plotting configuration
-skeleton: [['snout', 'leftear'], ['snout', 'rightear'], ['leftear', 'tailbase'], ['leftear', 'rightear'], ['rightear','tailbase']]
+skeleton:
+ - ["snout", "leftear"]
+ - ["snout", "rightear"]
+ - ["leftear", "tailbase"]
+ - ["leftear", "rightear"]
+ - ["rightear", "tailbase"]
skeleton_color: white
pcutoff: 0.4
dotsize: 4
alphavalue: 0.5
colormap: jet
```
-Then pass ``draw_skeleton=True`` with the command:
+Then pass `draw_skeleton=True` with the command:
```python
-deeplabcut.create_labeled_video(config_path,['fullpath/afolderofvideos'], videotype='.mp4', draw_skeleton = True)
+deeplabcut.create_labeled_video(
+ config_path,
+ ["fullpath/afolderofvideos"],
+ videotype=".mp4",
+ draw_skeleton=True
+)
```
-**NEW** as of 2.2b8: You can create a video with only the "dots" plotted, i.e., in the [style of Johansson](https://link.springer.com/article/10.1007/BF00309043), by passing `keypoints_only=True`:
+**NEW** as of 2.2b8: You can create a video with only the "dots" plotted, i.e., in the
+[style of Johansson](https://link.springer.com/article/10.1007/BF00309043), by passing `keypoints_only=True`:
```python
-deeplabcut.create_labeled_video(config_path,['fullpath/afolderofvideos'], videotype='.mp4', keypoints_only=True)
+deeplabcut.create_labeled_video(
+ config_path,["fullpath/afolderofvideos"],
+ videotype=".mp4",
+ keypoints_only=True
+)
```
-**PRO TIP:** that the **best quality videos** are created when ``save_frames=True`` is passed. Therefore, when ``trailpoints`` and ``draw_skeleton`` are used, we **highly** recommend you also pass ``save_frames=True``!
+**PRO TIP:** that the **best quality videos** are created when `fastmode=False` is passed. Therefore, when
+`trailpoints` and `draw_skeleton` are used, we **highly** recommend you also pass `fastmode=False`!
