Repository navigation
Git repo state seems to be corrupt? #492
Description
[backend] | 2025-09-05T20:44:17.357Z error: [repo-manager] Failed to sync repository gerrit.hostname.com/MyRepo (id: 8) after 3 attempts. Error: Error: Failed to fetch repository: /data/.sourcebot/repos/8. Reason: fatal: couldn't find remote ref HEAD
[backend] |
I updated Sourcebot from v4.6.3 -> v4.6.6, running in k8s. terminationGracePeriod for my deployment was set to 15s (meaning k8s would send SIGTERM to Sourcebot process, wait 15 seconds, hope Sourcebot gracefully shut down, then send SIGKILL after 15 seconds regardless). Perhaps this was too low.
In either case, after Sourcebot came up with v4.6.6 image, certain repos are now failing to sync/index, and retrying doesn't help. All fail with the same error signature (above).
Not sure how to debug this / bandaid fix these failing repos. Redeploying SB from scratch will certainly fix things but that's sort of undesirable when there are 5000+ repos indexed already.
Activity
brendan-kellam commentedon Sep 6, 2025
Hmm ok a regression was introduced in 4.6.5 but I (thought) was fixed in 4.6.6 but clearly not.
For context, the change in 4.6.6 removed the remote.origin.url from the config (see #488). Presumably these repos are going into a bad state since the origin is not set.
Will look into this in a bit when I'm back on my laptop. Thx for reporting.
brendan-kellam commentedon Sep 6, 2025
How many repos are corrupted?
brendan-kellam commentedon Sep 6, 2025
Also clarify: did you upgrade to v4.6.5 at any point?
musabshak commentedon Sep 6, 2025
I went straight from v4.6.3 to v4.6.6, after which I ran into this issue.
I then went down to v4.6.4 (because I wanted the Gerrit branch indexing feature), and the issue resolved itself (ie I guess the repos weren't "corrupt", there are just some bad code changes in v4.6.5 / v4.6.6)
That is, I never deployed v4.6.5.
brendan-kellam commentedon Sep 8, 2025
Ok gotcha, thanks for clarifying.
If you don't mind, could you try to spin up a sourcebot deployment locally on v4.6.6 with this single repository listed in the config? I'm wondering if this issue will consistently reproduce with the same repos.
musabshak commentedon Sep 8, 2025
A fresh deploy from scratch at v4.6.6 works fine. I guess the problematic aspect then is that updating to v4.6.6 breaks things (at least it did for me).
I brought up a new instance at v4.6.4 from scratch and everything works fine now, I'm just hesitant to update to v4.6.6 now in
prodbecause it may break again.musabshak commentedon Sep 8, 2025
UPDATE: the fresh deploy from scratch at v4.6.6 fails at the first sync for the same repo. That is, the initial clone + index seemed to work fine, but after an hour, when it resync+reindexed, that failed. This suggests v4.6.6 is actively broken.
musabshak commentedon Sep 8, 2025
Same error signature as in top-level message. And suspiciously, the failing repo(s) have the characteristic that they're empty repos.
brendan-kellam commentedon Sep 8, 2025
Hmm this is useful information thanks. Let me try on my side.
brendan-kellam commentedon Sep 8, 2025
@musabshak when you get a second, could you try out
ghcr.io/sourcebot-dev/sourcebot:mainand see if you still run into this issue?