Repository navigation
Object detection training/fine-tuning for Owl-vit/Owlv2 #33664
Description
Feature request
Currently the Owl-vit models support inference and CLIP-style contrastive pre-training, but don't provide a way to train (or fine-tune) the detection part of the model. According to the paper, detection training is similar to Detr.
Motivation
It would be really awesome to be able to train or fine-tune one of these already-existing open-vocabulary object detection models.
Your contribution
I may be able to help some with this, not sure at present
Activity
LysandreJik commentedon Sep 24, 2024
cc @molbap @qubvel
qubvel commentedon Sep 24, 2024
Hi @catalys1, thank you so much for your feature request! We agree that the ability to fine-tune Owl-vit/Owlv2 would be a great addition. If you have the time and are interested in contributing, we would love to collaborate with you on this! Your help would be greatly appreciated 🤗
This PR might be also helpful
Object detection training for Owl-vit/Owlv2
daniel-bogdoll commentedon Oct 22, 2024
@qubvel Is #34057 maybe the solution here? I would also be happy to see the ability to finetune (some) ZeroShotObjectDetection models.
qubvel commentedon Oct 29, 2024
Hi @daniel-bogdoll, not sure linked PR solved the issue, there is no object detection loss added there.
cc @yonigozlan
aryanchauhan31 commentedon May 29, 2025
Hi @qubvel and team 👋
Thanks so much for the discussion and for flagging this as a "Good Second Issue" — I'd love to contribute to this feature!
vietvo89 commentedon May 31, 2025
Hi all, I found a related discussion #28778, but I could not find a complete script or instructions.
SuccessMoses commentedon Jun 11, 2025
Hi @aryanchauhan31 are you actively working on this. I have been working on a PR part-time for the past few weeks because I needed to a way to fine-tune detection models. I should have mentioned this in the discussion weeks ago, my mistake. I just want to know if you are actively working on it
aryanchauhan31 commentedon Jun 11, 2025
@SuccessMoses No, I was kinda waiting for someone to confirm if i can work on this. Please go ahead and if u need any help do let me know, I would love to collaborate
sun421-cloud commentedon Aug 1, 2025
Hello, I am currently doing the same thing and want to fine-tune the effect of owlv2. Have you made any progress?
0x-avi commentedon Aug 27, 2025
@qubvel @LysandreJik if no ones working on this can i try my hand at it?
qubvel commentedon Sep 1, 2025
Hey @0x-avi, you can work on it!
0x-avi commentedon Sep 20, 2025
hey sorry something had come up , actively working on this now
0x-avi commentedon Sep 30, 2025
hey @qubvel sorry for the ping i needed a bit of help understanding certain things, please correct me anywhere if im wrong, so i read up the entire owl-vit paper firstly and broke down the tasks , firstly i need to implement their adaption of bipartite matching loss that they copeid from DETR into the modeling file for owlv2 and the original owl-vit then return those losses from the objectdetection model. after this it gets kind of confusing but from what i infer, as each loss returned is for a single time step so i need to handle how the losses are processed over an entire epoch and just call accelerate.Accelerator.backward() to get grads which then i use to update the weights till desired number of epoch, and all of this is already handled by the trainer class and i just need to pass in the object detection model?
Now i had two questions:
1 - the original paper seems to have a custom jax training loop, do i use that or try to adhere to how every other model uses hf trainer?
2 - i tried searching through other relevant PRs for implementing a training loop for a model but i didn't manage to find one, could i possibly get a link to one such closed PR?
0x-avi commentedon Oct 11, 2025
@molbap @qubvel still waiting on a reply , sorry for the ping
anidoesdev commentedon Jun 27, 2026
Is this assigned to someone... If no can I work on this ?? @molbap @qubvel
molbap commentedon Jul 1, 2026
@anidoesdev you can work on it!