|
497 | 497 | "\n", |
498 | 498 | "**SuperAnimal transfer-learning** The encoder was pre-trained first from ImageNet, then in pose datasets we colleceted. Then decoder is trained from scratch in downstream tasks.\n", |
499 | 499 | "\n", |
500 | | - "**SuperAnimal naive fine-tuning** Both the encoder and the decoder were pre-trained in pose datasets we collected. In downstream datsets, we only finetune convolutional channels that correspond to the annotated keypoints in the downstream datasets. This introduces catastrophic forgetting in keypoints that are not annotated in the downstream datasets.\n", |
| 500 | + "**SuperAnimal naive fine-tuning** Both the encoder and the decoder were pre-trained in pose datasets we collected. In downstream datasets, we only finetune convolutional channels that correspond to the annotated keypoints in the downstream datasets. This introduces catastrophic forgetting in keypoints that are not annotated in the downstream datasets.\n", |
501 | 501 | "\n", |
502 | 502 | "**SuperAnimal memory-replay fine-tuning** If we apply fine-tuning with SuperAnimal without further cares, the models will forget about keypoints that are not annotated in the downstream datasets. To mitigate this, we mix the annotations and zero-shot predictions of SuperAnimal models to create a dataset that 'replays' the memory of the SuperAnimal keypoints.\n", |
503 | 503 | "\n", |
|
785 | 785 | "source": [ |
786 | 786 | "#### Setup the weight init and dataset\n", |
787 | 787 | "\n", |
788 | | - "First we do keypoint matching. This steps make it possible to understand the correspondance between the existing annotations and SuperAnimal annotations. This step produces 3 outputs\n", |
| 788 | + "First we do keypoint matching. This steps make it possible to understand the correspondence between the existing annotations and SuperAnimal annotations. This step produces 3 outputs\n", |
789 | 789 | "- The confusion matrix\n", |
790 | 790 | "- The conversion table\n", |
791 | 791 | "- Pseudo predictions over the whole dataset" |
|
799 | 799 | "source": [ |
800 | 800 | "#### What is keypoint matching?\n", |
801 | 801 | "\n", |
802 | | - "Because SuperAnimal models have their pre-defined keypoints that are potentially different from your annotations, we porposed this algorithm to minimize the gap between the model and the dataset. We use our model to perform zero-shot inference on the whole dataset. This gives pairs of predictions and ground truth for every image. Then, we cast the matching between models’ predictions (2D coordinates)\n", |
| 802 | + "Because SuperAnimal models have their pre-defined keypoints that are potentially different from your annotations, we proposed this algorithm to minimize the gap between the model and the dataset. We use our model to perform zero-shot inference on the whole dataset. This gives pairs of predictions and ground truth for every image. Then, we cast the matching between models’ predictions (2D coordinates)\n", |
803 | 803 | "and ground truth as bipartitematching using the Euclidean distance as the cost between paired of keypoints. We then solve the matching using the Hungarian algorithm. Thus for every image, we end up getting a matching matrix where 1 counts formatch and 0 counts for non-matching. Because the models’ predictions can be noisy from image to image, we average the aforementioned matching matrix across all the images and perform another bipartite matching, resulting in the final keypoint conversion table between the model and the dataset. Note that the quality of thematching will impact the performance\n", |
804 | 804 | "of the model, especially for zero-shot. In the case where, e.g., the annotation nose is mistakenly converted to keypoint tail and vice versa, the model will have to unlearn the channel that corresponds to nose and tail (see also case study in Mathis et al.)." |
805 | 805 | ] |
|
843 | 843 | "source": [ |
844 | 844 | "#### Display the confusion matrix\n", |
845 | 845 | "\n", |
846 | | - "The x axis lists the keypoints in the existing annotations. The y axis lists the keypoints in SuperAnimal keypoint space. Darker color encodes stronger correspondance between the human annotation and SuperAnimal annotations." |
| 846 | + "The x axis lists the keypoints in the existing annotations. The y axis lists the keypoints in SuperAnimal keypoint space. Darker color encodes stronger correspondence between the human annotation and SuperAnimal annotations." |
847 | 847 | ] |
848 | 848 | }, |
849 | 849 | { |
|
871 | 871 | }, |
872 | 872 | "source": [ |
873 | 873 | "#### Display the conversion table\n", |
874 | | - "The gt columns represents the keypoint names in the existing dataset. The MasterName represents the correspoinding keypoints in SuperAnimal keypoint space." |
| 874 | + "The gt columns represents the keypoint names in the existing dataset. The MasterName represents the corresponding keypoints in SuperAnimal keypoint space." |
875 | 875 | ] |
876 | 876 | }, |
877 | 877 | { |
|
0 commit comments