ROX-29509: Reload TLS certificates in central-db and scanner-v4-db by vladbologa · Pull Request #20335 · stackrox/stackrox

vladbologa · 2026-05-04T09:30:51Z

Description

This PR introduces TLS certificate hot reloading for central-db and scanner-v4-db, by means of a sidecar container that notifies PostgreSQL via pg_ctl to reload its configuration, when it detects a change in configuration.

The cert-watcher.sh script is duplicated between image/postgres/scripts/ (central-db) and scanner/image/db/scripts/ (scanner-v4-db) because these two images have separate build contexts and independent Dockerfiles with no existing mechanism for sharing scripts between them. This is consistent with how docker-entrypoint.sh and init-entrypoint.sh are already maintained independently in both locations.

Alternatives considered:

adding the script in docker-entrypoint.sh is a simple solution, but it's not really observable (compared to the sidecar where we can inspect the logs of the cert-watcher container)
a Go implementation would have the problem that we don't have Go building in the Postgres images, and we don't have pg_ctl in the main image

User-facing documentation

CHANGELOG.md is updated OR update is not needed
documentation PR is created and is linked above OR is not needed

Testing and quality

the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
CI results are inspected

Automated testing

How I validated my change

Generated new certificates (with the CA from central-tls) and replaced central-db-tls.
Checked:

cert-watcher sidecar logs:

2026-05-15 17:20:08.658 UTC cert-watcher: watching /run/secrets/stackrox.io/certs for changes (interval: 5s, pgdata: /var/lib/postgresql/data/pgdata)
2026-05-15 17:25:36.356 UTC cert-watcher: TLS certificates changed, reloading PostgreSQL
server signaled

central-db postgres logs:

2026-05-15 17:25:36.358 UTC [22] LOG: received SIGHUP, reloading configuration files

verified with openssl s_client that central-db now presents the new certificates

Also tried bogus certificates, in that case postgres refuses to load the new certs:

2026-05-08 13:12:07.556 UTC [22] LOG: received SIGHUP, reloading configuration files
2026-05-08 13:12:07.558 UTC [22] LOG: could not load server certificate file "/run/secrets/stackrox.io/certs/server.crt": no start line
2026-05-08 13:12:07.558 UTC [22] LOG: SSL configuration was not reloaded

The same tests were performed for scanner-v4-db as well.

openshift-ci · 2026-05-04T09:30:55Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

github-actions · 2026-05-04T09:39:41Z

🚀 Build Images Ready

Images are ready for commit 7cc8a15. To use with deploy scripts:

export MAIN_IMAGE_TAG=4.11.x-1036-g7cc8a1553a

codecov · 2026-05-04T10:10:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 49.64%. Comparing base (0114d50) to head (09f7b82).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #20335   +/-   ##
=======================================
  Coverage   49.64%   49.64%           
=======================================
  Files        2799     2799           
  Lines      212917   212917           
=======================================
  Hits       105692   105692           
  Misses      99486    99486           
  Partials     7739     7739

Flag	Coverage Δ
go-unit-tests	`49.64% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

vladbologa · 2026-05-11T13:39:13Z

@coderabbitai review

coderabbitai · 2026-05-11T13:39:20Z

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai · 2026-05-11T13:45:17Z

📝 Walkthrough

Walkthrough

This PR introduces automatic TLS certificate reloading for PostgreSQL databases by adding a cert-watcher sidecar that polls certificate files, detects changes via MD5 hashing, and triggers PostgreSQL reload without requiring manual SIGHUP signals.

Changes

PostgreSQL Certificate Reload

Layer / File(s)	Summary
Certificate Watcher Script `image/postgres/scripts/cert-watcher.sh`, `scanner/image/db/scripts/cert-watcher.sh`	Bash script that polls `ROX_CERT_DIR` at `ROX_CERT_POLL_INTERVAL`, computes MD5 hash of `server.crt` and `server.key`, and invokes `pg_ctl reload -D $PGDATA` when hash changes. Logs detection and reload failures.
Helm Sidecar Template `image/templates/helm/shared/templates/_cert-watcher.tpl`	Defines `srox.certWatcherSidecar` template that renders a cert-watcher container with postgres image, cert-watcher.sh entrypoint, PGDATA and env vars, read-only TLS volume mount, disk volume mount, resource requests/limits, and restricted security context (user/group 70, no privilege escalation).
Docker Image Build `scanner/image/db/Dockerfile`	Copies `cert-watcher.sh` into `/usr/local/bin/` during image build.
Kubernetes Deployments `image/templates/helm/shared/templates/02-scanner-v4-07-db-deployment.yaml`, `image/templates/helm/stackrox-central/templates/01-central-12-central-db.yaml`	Enables `shareProcessNamespace: true` in pod spec for scanner-v4-db and central-db; injects `srox.certWatcherSidecar` container configured with respective TLS volumes (`certs` and `central-db-tls-volume`).
Documentation `pkg/mtls/README.md`	Adds reference to cert-watcher.sh mechanism for Postgres certificate reload in Central DB, Scanner DB, and Scanner V4 DB. Removes outdated claim that Postgres SSL certificates require SIGHUP reload.

Sequence Diagram(s)

sequenceDiagram
    participant CertWatcher as cert-watcher<br/>sidecar
    participant FSys as TLS Certificate<br/>Files
    participant PostgreSQL as PostgreSQL<br/>Database
    
    loop Every ROX_CERT_POLL_INTERVAL
        CertWatcher->>FSys: Read and hash<br/>server.crt + server.key
        FSys-->>CertWatcher: Combined MD5 hash
        CertWatcher->>CertWatcher: Compare with<br/>previous hash
    end
    
    alt Hash changed
        CertWatcher->>PostgreSQL: pg_ctl reload -D $PGDATA
        PostgreSQL-->>CertWatcher: Reload completed
        CertWatcher->>CertWatcher: Update stored hash
    else Hash unchanged
        CertWatcher->>CertWatcher: Wait for next poll
    end

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: introducing TLS certificate reloading for central-db and scanner-v4-db PostgreSQL deployments.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description provides a detailed explanation of changes, addresses documentation requirements, testing/quality checkpoints, and includes comprehensive validation details.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch vb/postgres-reload-certs

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

Generate code and open pull requests
Plan features and break down work
Investigate incidents and troubleshoot customer tickets together
Automate recurring tasks and respond to alerts with triggers
Summarize progress and report instantly

Built for teams:

Shared memory across your entire org—no repeating context
Per-thread sandboxes to safely plan and execute work
Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

porridge

How about adding a CI check that makes sure these two scripts are in sync? This will allow engineers to realize they need to be maintained in lockstep.

vladbologa · 2026-05-15T14:42:05Z

/test ocp-4-12-scanner-v4-install-tests

coderabbitai · 2026-05-15T17:27:30Z

CodeRabbit chat interactions are restricted to organization members for this repository. Ask an organization member to interact with CodeRabbit, or set chat.allow_non_org_members: true in your configuration.

mclasmeier

lgtm

openshift-ci Bot added the do-not-merge/work-in-progress label May 4, 2026

github-actions Bot added area/helm area/postgres labels May 4, 2026

vladbologa force-pushed the vb/postgres-reload-certs branch 7 times, most recently from 6b18d32 to ccdd1ca Compare May 11, 2026 11:50

github-actions Bot added the area/scanner label May 11, 2026

vladbologa changed the title ~~ROX-29509: Reload TLS certificates in central-db~~ ROX-29509: Reload TLS certificates in central-db and scanner-v4-db May 11, 2026

vladbologa marked this pull request as ready for review May 11, 2026 13:46

vladbologa requested review from a team as code owners May 11, 2026 13:46

vladbologa requested review from porridge and removed request for a team May 11, 2026 13:46

openshift-ci Bot removed the do-not-merge/work-in-progress label May 11, 2026

vladbologa requested review from dashrews78, erthalion and mclasmeier May 11, 2026 13:46

mclasmeier requested changes May 12, 2026

View reviewed changes

mclasmeier reviewed May 13, 2026

View reviewed changes

Comment thread image/postgres/scripts/cert-watcher.sh Outdated

Comment thread scanner/image/db/scripts/cert-watcher.sh

Comment thread scanner/image/db/scripts/cert-watcher.sh Outdated

porridge approved these changes May 14, 2026

View reviewed changes

Comment thread image/postgres/scripts/cert-watcher.sh Outdated

vladbologa force-pushed the vb/postgres-reload-certs branch 2 times, most recently from cea552b to cdb4420 Compare May 15, 2026 12:51

vladbologa force-pushed the vb/postgres-reload-certs branch from cdb4420 to 789126b Compare May 15, 2026 15:05

vladbologa requested a review from mclasmeier May 18, 2026 08:46

mclasmeier requested changes May 18, 2026

View reviewed changes

Comment thread tests/e2e/run-scanner-v4-install.bats Outdated

vladbologa added 16 commits May 18, 2026 15:05

ROX-29509: Reload TLS certificates in central-db

d2a9300

Move cert watcher script to sidecar

da5e07f

Extend to scanner-v4-db

62caef9

Fix helm unit tests

4c80b27

Fix scanner

1aded65

Print warning on startup if pg_ctl is missing

2cad239

Add e2e tests

b41aa82

Add seconds to default duration

00bbbe5

Move sleep

59ec3e0

Show PGDATA

75a67df

cat instead of first md5sum

4053315

retry reload on transient failures

e1d1a0a

Add check that files are identical

b1b13e0

Log time

c785154

Fix grep bug in tests

7226a4f

Improve script startup

1f0a93d

vladbologa force-pushed the vb/postgres-reload-certs branch from 789126b to 1f0a93d Compare May 18, 2026 13:05

vladbologa requested a review from mclasmeier May 18, 2026 13:37

mclasmeier approved these changes May 19, 2026

View reviewed changes

vladbologa merged commit 7cc8a15 into master May 19, 2026
103 checks passed

vladbologa deleted the vb/postgres-reload-certs branch May 19, 2026 07:32

vladbologa mentioned this pull request Oct 20, 2026

chore(operator-e2e): add cert-watcher verification to kuttl tests #23501

Merged

5 of 9 tasks

Conversation

vladbologa commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

User-facing documentation

Testing and quality

Automated testing

How I validated my change

Uh oh!

openshift-ci Bot commented May 4, 2026

Uh oh!

github-actions Bot commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Build Images Ready

Uh oh!

codecov Bot commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vladbologa commented May 11, 2026

Uh oh!

coderabbitai Bot commented May 11, 2026

Uh oh!

coderabbitai Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

porridge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vladbologa commented May 15, 2026

Uh oh!

coderabbitai Bot commented May 15, 2026

Uh oh!

Uh oh!

mclasmeier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vladbologa commented May 4, 2026 •

edited

Loading

github-actions Bot commented May 4, 2026 •

edited

Loading

codecov Bot commented May 4, 2026 •

edited

Loading

coderabbitai Bot commented May 11, 2026 •

edited

Loading