[bug] Unable to upgrade to v3

Describe the bug

Hello folks,

I just tried to upgrade to v3 (from 2.8.4) following https://docs.sourcebot.dev/self-hosting/upgrade/v2-to-v3-guide but upon startup of the v3, I've got errors related to Postgres.

I haven't had time to investigate yet, this may be a permission issue on our end.

In doubt, I thought I'd share it :)

Logs during startup:

[Info] Waiting for the database to be ready...
postgres: could not access the server configuration file "/data/.sourcebot/db/postgresql.conf": No such file or directory
localhost:5432 - no response
[Info] Waiting for the database to be ready...
localhost:5432 - no response
[Info] Waiting for the database to be ready...
localhost:5432 - no response
[Info] Waiting for the database to be ready...
localhost:5432 - no response
[Info] Waiting for the database to be ready...
localhost:5432 - no response

To reproduce

N/A

Sourcebot deployment information

Sourcebot version (e.g. v3.0.1): v3.0.2

Additional information

No response

The one time I've seen this happen is when postgres is unable to initialize on the initial startup. What is the setup you have with mounting the disk to the container?

What is the setup you have with mounting the disk to the container?

It's a regular volume mount through Kubernetes. Volume is NFS though. I've had issues in the past with NFS volumes and Postgres, so that could be it. But there's not much logs for me to investigate.

I'll give it locally on my laptop to troubleshoot

This definitely look's like a permission issue.

On my laptop, I'm able to make it work but I see that the postgres folder is owned by another user:

ll .sourcebot   
total 16K
drwx------. 19   100069   100069 4,0K  7 avril 13:06 db
drwxr-xr-x.  2 gael gael 4,0K  7 avril 13:06 index
drwxr-xr-x.  2 gael gael 4,0K  7 avril 13:07 redis
drwxr-xr-x.  3 gael gael 4,0K  7 avril 13:06 repos

I know NFS is a bit crappy for such things. I've had a look at another setup where we have a NFS volume mount for a Postgres container and we've had to add a couple of configurations to make it work (runAsUser and runAsGroup on the postgres container with same uid than the owner of NFS files).

Thus I guess it's something related to our infra rather than an issue with Sourcebot.

But I would expect Sourcebot to raise a more explicit error that it's an issue with permissions.

That being said, for now, I don't have a workaround on my end to make it work. I'll probably switch to another type of volume anyway.

It's the following line that raises the error https://github.com/sourcebot-dev/sourcebot/blob/bf8b28252d1aacdf9d54e0f5feb8f8994ecb9006/entrypoint.sh#L133C5-L133C52

Even after giving all possible rights to the folder. From inside the container:

/app # su postgres -c "postgres -D $DATABASE_DATA_DIR"
postgres: could not access the server configuration file "/data/.sourcebot/db/postgresql.conf": No such file or directory
/app # ls -al /data/.sourcebot/db/
total 8
drwxrwxrwx    2 postgres postgres      4096 Apr  7 11:15 .
drwxr-xr-x    4 nobody   nobody        4096 Apr  7 11:15 ..

Thanks for investigating! Yeah seems like the internal postgres deployment doesn't like certain volumes. I don't have too much experience with postgres, so not exactly sure why without further investiagation

We should definitely add a better error message when this happens. In the meantime, could you try a different mount? Alternatively, you could spin up an external postgres server and connect sourcebot to it using DATABASE_URL

I've had this issue as well and have found a workaround.
The issue is that Sourcebot can't chown the DB directory:

[Info] Initializing database at /data/.sourcebot/db...
chown: /data/.sourcebot/db: Operation not permitted

This might be related to NFS root squashing? No idea.
This message will only be logged in the first container Kubernetes creates, which immediately crashes and gets replaced. If you don't have a log aggregation system, it is very easy to miss

Workaround

Connect to your NFS server and navigate to the .sourcebot directory of the container and chown the db directory:

chown 70:70 $path_to_db_directory # 70 = postgres

Open a shell inside the Sourcebot container and execute the following command:

su postgres -c "initdb -D $DATABASE_DATA_DIR"

Restart the deployment and it should work.

The line su postgres -c "initdb -D $DATABASE_DATA_DIR" is taken directly from the entrypoint.sh script, it just gets skipped if the db directory already exists.

@belaw

Thanks for finding a workaround @belaw ! We'll try to get this logic backed into the script

The workaround was not enough for me ; I ended up spinning a dedicated external Postgres DB.

We had to give our container the CHOWN capability:

          securityContext:
            runAsNonRoot: false
            readOnlyRootFilesystem: false
            capabilities:
              add: [ "SETGID", "SETUID", "CHOWN" ]

I ended up creating a volume and then mounting both that and the config, but this was on a fresh install. I'm only experimenting with Sourcebot on my laptop so not sure if this is actually a sensible fix or an outright hack.

For reference I'm running it on Mac OS 15.6 in Rancher, using QEMU and 9p (I haven't changed back to Apple VZ and virtiofs since switching to using a Docker volume, so they may also work).

docker volume create sourcebot-vlm

docker run \
    -p 3000:3000 \
    --pull=always \
    --rm \
    -v sourcebot-vlm:/data \
    -v $(pwd)/config.json:/data/config.json \
    -e CONFIG_PATH=/data/config.json \
    --name sourcebot \
    ghcr.io/sourcebot-dev/sourcebot:latest

@brendan-kellam

going to reopen this @brendan-kellam but we should probably create separate issues to track underlying issues here

Any updates on this running into this same issue using deployment guide and running the command

docker run \
    -p 3000:3000 \
    --pull=always \
    --rm \
    -v $(pwd):/data \
    -e CONFIG_PATH=/data/config.json \
    --name sourcebot \
    ghcr.io/sourcebot-dev/sourcebot:latest

gives an error of

[Info] Initializing database at /data/.sourcebot/db...
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with this locale configuration:
  provider:    libc
  LC_COLLATE:  C
  LC_CTYPE:    C.UTF-8
  LC_MESSAGES: C
  LC_MONETARY: C
  LC_NUMERIC:  C
  LC_TIME:     C
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /data/.sourcebot/db ... initdb: error: could not change permissions of directory "/data/.sourcebot/db": Operation not permitted

The folder /data/.sourcebot/db is created but empty on a second run of same command I get the following error:

[Warning] Failed to send install event.
localhost:5432 - no response
[Info] Waiting for the database to be ready...
postgres: could not access the server configuration file "/data/.sourcebot/db/postgresql.conf": No such file or directory
localhost:5432 - no response
[Info] Waiting for the database to be ready...

at which point it just loops endlessly waiting for database to be ready. Deleting the .sourcebot folder and running again just gives first error. I am running locally on my computer in Ubuntu.

Closing since #599 should have resolved this issue. Feel free to re-open if not resolved.

Navigation Menu

Repository navigation

[bug] Unable to upgrade to v3 #263

Describe the bug

To reproduce

Sourcebot deployment information

Additional information

Workaround

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[bug] Unable to upgrade to v3 #263

Description

Describe the bug

To reproduce

Sourcebot deployment information

Additional information

Activity

msukkari commented on Apr 4, 2025

gaeljw commented on Apr 7, 2025

gaeljw commented on Apr 7, 2025

gaeljw commented on Apr 7, 2025

msukkari commented on Apr 10, 2025

belaw commented on Apr 14, 2025

Workaround

msukkari commented on Apr 25, 2025

gaeljw commented on Jun 6, 2025

komapa commented on Aug 5, 2025

chedabob commented on Aug 12, 2025

msukkari commented on Aug 13, 2025

igloo15 commented on Oct 3, 2025

brendan-kellam commented on Jan 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions