Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion deeplabcut/modelzoo/video_inference.py
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ def video_inference_superanimal(
)

Tips:
* max_individuals: make sure you correclty give the number of individuals. Our
* max_individuals: make sure you correctly give the number of individuals. Our
inference api will only give up to max_individuals number of predictions.
* pseudo_threshold: the higher you set, the more aggressive you filter low
confidence predictions during video adaptation.
Expand Down
2 changes: 1 addition & 1 deletion deeplabcut/pose_estimation_pytorch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -462,7 +462,7 @@ When `deeplabcut.pose_estimation_pytorch.apis.analyze_videos.video_inference` is
with a top-down model, it is assumed that a detector snapshot is given as well to obtain
bounding boxes with which to run pose estimation. It's possible that you've already
obtained bounding boxes for your video (with another object detector or through some
other means), and you want to re-use those bounding boxes instead of running an object
other means), and you want to reuse those bounding boxes instead of running an object
detector again.

You can easily do so by writing a bit of custom code, as shown in the example below:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ class CSPNeXt(HuggingFaceWeightsMixin, BaseBackbone):
out_indices: The branch indices to output. If a tuple of integers, the outputs
are returned as a list of tensors. If a single integer, a tensor is returned
containing the configured index.
channel_attention: Add chanel attention to all stages
channel_attention: Add channel attention to all stages
norm_layer: The type of normalization layer to use.
activation_fn: The type of activation function to use.
**kwargs: BaseBackbone kwargs.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ def _generate_standard(
target_y = np.zeros((N, K, H), dtype=np.float32)

for n, k in product(range(N), range(K)):
# skip unlabled keypoints
# skip unlabeled keypoints
if keypoints_visible[n, k] < 0.5:
continue

Expand Down Expand Up @@ -204,7 +204,7 @@ def _generate_gaussian(
y = np.arange(0, H, 1, dtype=np.float32)

for n, k in product(range(N), range(K)):
# skip unlabled keypoints
# skip unlabeled keypoints
if keypoints_visible[n, k] < 0.5:
continue

Expand Down
2 changes: 1 addition & 1 deletion deeplabcut/utils/pseudo_label.py
Original file line number Diff line number Diff line change
Expand Up @@ -402,7 +402,7 @@ def dlc3predictions_2_annotation_from_video(
# skipping every 4 frames should speed up and not impact the performance
predictions, image_paths = predictions[::10], image_paths[::10]

# because inference api does not return image path. I am assuming the predictions come in an oder from the video
# Since the inference API does not return the image path, I assume the predictions are provided in the same order as the frames in the video.
assert len(image_paths) == len(
predictions
), f"number of images must be equal to number of predictions. image_paths: {len(image_paths)} , predictions: {len(predictions)}"
Expand Down
10 changes: 5 additions & 5 deletions examples/COLAB/COLAB_YOURDATA_SuperAnimal.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -497,7 +497,7 @@
"\n",
"**SuperAnimal transfer-learning** The encoder was pre-trained first from ImageNet, then in pose datasets we colleceted. Then decoder is trained from scratch in downstream tasks.\n",
"\n",
"**SuperAnimal naive fine-tuning** Both the encoder and the decoder were pre-trained in pose datasets we collected. In downstream datsets, we only finetune convolutional channels that correspond to the annotated keypoints in the downstream datasets. This introduces catastrophic forgetting in keypoints that are not annotated in the downstream datasets.\n",
"**SuperAnimal naive fine-tuning** Both the encoder and the decoder were pre-trained in pose datasets we collected. In downstream datasets, we only finetune convolutional channels that correspond to the annotated keypoints in the downstream datasets. This introduces catastrophic forgetting in keypoints that are not annotated in the downstream datasets.\n",
"\n",
"**SuperAnimal memory-replay fine-tuning** If we apply fine-tuning with SuperAnimal without further cares, the models will forget about keypoints that are not annotated in the downstream datasets. To mitigate this, we mix the annotations and zero-shot predictions of SuperAnimal models to create a dataset that 'replays' the memory of the SuperAnimal keypoints.\n",
"\n",
Expand Down Expand Up @@ -785,7 +785,7 @@
"source": [
"#### Setup the weight init and dataset\n",
"\n",
"First we do keypoint matching. This steps make it possible to understand the correspondance between the existing annotations and SuperAnimal annotations. This step produces 3 outputs\n",
"First we do keypoint matching. This steps make it possible to understand the correspondence between the existing annotations and SuperAnimal annotations. This step produces 3 outputs\n",
"- The confusion matrix\n",
"- The conversion table\n",
"- Pseudo predictions over the whole dataset"
Expand All @@ -799,7 +799,7 @@
"source": [
"#### What is keypoint matching?\n",
"\n",
"Because SuperAnimal models have their pre-defined keypoints that are potentially different from your annotations, we porposed this algorithm to minimize the gap between the model and the dataset. We use our model to perform zero-shot inference on the whole dataset. This gives pairs of predictions and ground truth for every image. Then, we cast the matching between models’ predictions (2D coordinates)\n",
"Because SuperAnimal models have their pre-defined keypoints that are potentially different from your annotations, we proposed this algorithm to minimize the gap between the model and the dataset. We use our model to perform zero-shot inference on the whole dataset. This gives pairs of predictions and ground truth for every image. Then, we cast the matching between models’ predictions (2D coordinates)\n",
"and ground truth as bipartitematching using the Euclidean distance as the cost between paired of keypoints. We then solve the matching using the Hungarian algorithm. Thus for every image, we end up getting a matching matrix where 1 counts formatch and 0 counts for non-matching. Because the models’ predictions can be noisy from image to image, we average the aforementioned matching matrix across all the images and perform another bipartite matching, resulting in the final keypoint conversion table between the model and the dataset. Note that the quality of thematching will impact the performance\n",
"of the model, especially for zero-shot. In the case where, e.g., the annotation nose is mistakenly converted to keypoint tail and vice versa, the model will have to unlearn the channel that corresponds to nose and tail (see also case study in Mathis et al.)."
]
Expand Down Expand Up @@ -843,7 +843,7 @@
"source": [
"#### Display the confusion matrix\n",
"\n",
"The x axis lists the keypoints in the existing annotations. The y axis lists the keypoints in SuperAnimal keypoint space. Darker color encodes stronger correspondance between the human annotation and SuperAnimal annotations."
"The x axis lists the keypoints in the existing annotations. The y axis lists the keypoints in SuperAnimal keypoint space. Darker color encodes stronger correspondence between the human annotation and SuperAnimal annotations."
]
},
{
Expand Down Expand Up @@ -871,7 +871,7 @@
},
"source": [
"#### Display the conversion table\n",
"The gt columns represents the keypoint names in the existing dataset. The MasterName represents the correspoinding keypoints in SuperAnimal keypoint space."
"The gt columns represents the keypoint names in the existing dataset. The MasterName represents the corresponding keypoints in SuperAnimal keypoint space."
]
},
{
Expand Down