ROX-37814: add compliance data freshness sweeper - #23851

Description

The time-driven freshness sweeper for ROX-37814 — the last of the three enforcement points that keep a compliance check result's data_freshness correct (feature ROX-35712). The two write-time hooks (ingest → CURRENT, scan timeout → STALE) land in #23765; this PR adds the periodic backstop that catches staleness no event reports: a scheduled scan that silently stops cycling, or a secured cluster whose sensor is disconnected so no timeout ever fires.

Stacked on #23765 (base branch is gualvare/rox-35712-freshness-hooks). Retarget to master once that merges — the diff here is only the two sweeper commits.

Each ROX_COMPLIANCE_FRESHNESS_SWEEP_INTERVAL (default 5m), gated on the ComplianceSurfaceStaleData feature flag, the sweeper:

• for every scan configuration that has a schedule, computes cutoff = previousScheduledFire − grace, where grace = ComplianceScanScheduleWatcherTimeout + ComplianceScanWatcherTimeout (85m);
• marks STALE every check result of that configuration whose LastStartedTime predates the cutoff.

Design choices:

• STALE-only. The sweeper only ever adds STALE; CURRENT is owned by the ingest path and nil-timestamp rows stay UNKNOWN. Because it never writes CURRENT, it structurally cannot revert a concurrent ingest or fight the timeout hook — agreement is a property of the design, not a tuning coincidence. It reuses the hooks' existing markStale helper (read-modify-upsert under the same KeyFence).
• Config-name scope (vs. the timeout hook's scan-ref scope): the sweeper's evidence is the schedule, a config-level property that applies to every scan ref under it, so one cutoff is valid for all. A timeout, by contrast, is evidence about one scan on one cluster.
• Standalone package central/complianceoperator/v2/freshness, modeled on the CVE unsuppress loop (central/cve/suppress): ticker + concurrency.Stopper + singleton, wired in main.go (both Start() and the stoppable gated on the same flag).

New surface: datastore method MarkScanChecksStaleByConfig (mirrors MarkScanChecksStale, keyed by config name) and env ROX_COMPLIANCE_FRESHNESS_SWEEP_INTERVAL.

Scope: sweeper + datastore method + wiring only. API surfacing (so the UI can read/filter/sort by freshness), an on-demand ("scan now") refresh term, and orphan-row cleanup are deferred to later slices.

Alternatives considered:

• 3-way authoritative sweeper (set CURRENT/STALE/UNKNOWN each tick, as an earlier PoC did) — rejected: it duplicates ingest's CURRENT logic and needs matching age rules to avoid reverting the hooks. STALE-only is smaller and reverts nothing by construction.
• Fold into the report manager — rejected: broadens its single responsibility and couples a correctness loop to the report subsystem. Fold into the pruning GC — rejected: that subsystem deletes rows on a coarse hourly cadence; wrong domain and cadence.

User-facing documentation

• CHANGELOG.md is updated OR update is not needed
• documentation PR is created and is linked above OR is not needed

Not needed: behind the ComplianceSurfaceStaleData feature flag (default off); no user-visible API surface in this PR.

Testing and quality

• the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
• CI results are inspected

Gated by the ComplianceSurfaceStaleData feature flag (default off).

Automated testing

• added unit tests
• added e2e tests
• added regression tests
• added compatibility tests
• modified existing tests

How I validated my change

• Unit (freshness/sweeper_test.go): expectedRefresh (nil / valid daily / unconvertible schedule); sweep (scheduled config → MarkScanChecksStaleByConfig called with cutoff = prevTick − grace, asserted; nil-schedule skipped; list error tolerated; a per-config error does not stop the next); Start()→Stop() terminates.

• sql_integration (checkresults/datastore/datastore_freshness_test.go): TestMarkScanChecksStaleByConfig (old → STALE; recent and nil-timestamp rows untouched; other config untouched; write context required; idempotent re-run) and TestMarkScanChecksStaleByConfigAgreesWithWriteHooks (ingest-CURRENT + timeout-STALE rows, run the sweeper's method at the tightest cutoff → 0 reverts).

• Live OpenShift cluster (grace shortened to 2m, sweep 1m; managed config rox35712-sweeper, ocp4-moderate, 125 check results). datafreshness column, count:

  S0 initial scan ............... 1|125   (CURRENT)
  S1 schedule edited to past .... 2|125   (STALE, within one 1m tick)
  S2 on-demand rescan ........... 1|125   (CURRENT via ingest, no revert across ≥4 ticks)
  

  • Marks overdue STALE: editing the config's schedule to a past daily time did not trigger a rescan (LastStartedTime frozen), so cutoff fell after the stored timestamp and all 125 rows went CURRENT→STALE within one tick, with the log line compliance freshness sweeper: marked 125 check result(s) stale for scan configuration "rox35712-sweeper".
  • No false positives: a future-scheduled config stayed CURRENT across ≥4 ticks, no mark line.
  • Fresh clears, no revert: an on-demand rescan set the rows CURRENT via ingest; later ticks never reverted them (newer than the cutoff).
  • Exactly one mark line in the whole window — the same running loop marked only the genuinely overdue config. No timezone skew (Central runs UTC; schedule cron and time.Now() agree).

🤖 This change was developed with AI assistance (ai-assisted).