Is there an existing issue for this?
Operating System
Windows 10
DeepLabCut version
3.0.0rc10
What engine are you using?
pytorch
DeepLabCut mode
multi animal
Device type
NVIDIA RTX A4000
Bug description 🐛
To DLC developers,
I have been using DLC to track two mice (12 keypoints each) very successfully in the past using a resnet50 core with tensorflow.
Since moving to pytorch, I trained new models using the same datasets so that we can have longer-term support. In the latest version (rc10), I keep coming across a (large) mismatch between the metrics reported during model training and model evaluation.
For example (see file below), my model (dlcrnet) is being trained normally and both train and test set errors are decreasing. The model of epoch 150 is selected as "best" and has a test rmse of ~4.4. See below:
learning_stats.csv
When I'm running the evaluation later, the same model only gets a test rmse of 10, which is much worse than what is reported during training. Visual inspection of the predictions is also off. See below.
DLC_DlcrnetStride16Ms5_TwoMicePredictionAug29shuffle1_snapshot_best-150-results.csv
Do you have an idea of what may be the issue? I have never had a similar issue with tensorflow models, or pytorch models trained with the rc9 version. As as side note, model iinference with rc9 was much slower.
Thank you very much for the amazing software.
Best,
Dimos
Steps To Reproduce
No response
Relevant log output
Anything else?
No response
Code of Conduct
Is there an existing issue for this?
Operating System
Windows 10
DeepLabCut version
3.0.0rc10
What engine are you using?
pytorch
DeepLabCut mode
multi animal
Device type
NVIDIA RTX A4000
Bug description 🐛
To DLC developers,
I have been using DLC to track two mice (12 keypoints each) very successfully in the past using a resnet50 core with tensorflow.
Since moving to pytorch, I trained new models using the same datasets so that we can have longer-term support. In the latest version (rc10), I keep coming across a (large) mismatch between the metrics reported during model training and model evaluation.
For example (see file below), my model (dlcrnet) is being trained normally and both train and test set errors are decreasing. The model of epoch 150 is selected as "best" and has a test rmse of ~4.4. See below:
learning_stats.csv
When I'm running the evaluation later, the same model only gets a test rmse of 10, which is much worse than what is reported during training. Visual inspection of the predictions is also off. See below.
DLC_DlcrnetStride16Ms5_TwoMicePredictionAug29shuffle1_snapshot_best-150-results.csv
Do you have an idea of what may be the issue? I have never had a similar issue with tensorflow models, or pytorch models trained with the rc9 version. As as side note, model iinference with rc9 was much slower.
Thank you very much for the amazing software.
Best,
Dimos
Steps To Reproduce
No response
Relevant log output
Anything else?
No response
Code of Conduct