PyTorch training metrics incorrectly use multi-animal evaluation for single-animal projects #3518

Activity

1. Fabian97-Bumbee

   assigned

   MMathisLab

   on Sep 15, 2026
2. C-Achard

   assigned

   C-Achard

   and unassigned

   MMathisLab

   on Sep 15, 2026
3. C-Achard

   added

   bugSomething isn't workingSomething isn't working

   pytorch

   performanceRelated to (poor or unexpected) performance in any context (inference, training, labeling,...)Related to (poor or unexpected) performance in any context (inference, training, labeling,...)

   on Sep 15, 2026
4. C-Achard

   added the

   Bug

   issue type on Sep 15, 2026
5. C-Achard

   added

   metricsRelated to performance reporting and metrics calculationRelated to performance reporting and metrics calculation

   and removed

   performanceRelated to (poor or unexpected) performance in any context (inference, training, labeling,...)Related to (poor or unexpected) performance in any context (inference, training, labeling,...)

   on Sep 17, 2026
6. C-Achard commented on Sep 17, 2026

   

   C-Achard

   on Sep 17, 2026

   Member

   More actions

   Hello @Fabian97-Bumbee,

   Thanks for opening this, and apologies for the late reply as we were focusing on the next release.
   Your diagnosis is correct, this is the result of several edge cases not being handled entirely consistently in the code.

   Note that this does not affect the actual training process in that the "latest" weights generated are still what you'd have gotten without the issue, however the "best" snapshot saved may not have been derived correctly.

   Workaround (temporary, for 3.0.1-3.0.2)

   In your project config.yaml, change:
   snapshotindex: -2 (should be -1 by default)

   This will use the most trained (latest) snapshot instead of the early mislabeled "best" one.

   This will not fix the metrics (see below for the upcoming fixes), but will at least use a model that is properly trained for the rest of your analysis. Note that after the fix is released this will no longer be appropriate and should be switched to -1 again.

   You can also still diagnose training normally, in two ways:

   • Per epoch: use learning_stats.csv in the model folder, which records losses/train.total_loss and losses/eval.total_loss for every epoch
   • Per snapshot: set snapshotindex: all (same place as before) and run Evaluate Network. The results CSV in evaluation-results-pytorch has a row per snapshot with both train and test RMSE (in pixels).

   ---

   For reference

   Regarding the current code behavior:

   • The mAP/mAR metrics should not be defined in this case as there is only a single keypoint
   • The single animal parameter should indeed be propagated to the runner
   • Best snapshot selection should not use invalid metrics
     This and a few additional edge cases are now handled and fixed in Fix silent OKS metrics failure on invalid computation and problematic metrics propagation #3519.

   After the fix is released, best snapshots can no longer be selected using invalid metrics.
   If you would like best-snapshot selection then, you can select on RMSE instead, in pytorch_config.yaml:

   runner:
     key_metric: "test.rmse"
     key_metric_asc: false
   

   Note that sadly we cannot include this in 3.0.2 so it will have to be shipped at a later time, so this will only work in a future version, not on 3.0.1/3.0.2.

   Let me know if you have any questions and thanks again!

   Best,
   Cyril