Skip to content

[Backport release-1.6] fix(api): preserve foreign HelmRelease metadata and repair empty required fields - #4919

Merged
myasnikovdaniil merged 5 commits into
release-1.6from
backport-2914-3956-to-release-1.6
Oct 14, 2026
Merged

myasnikovdaniil merged 5 commits into
release-1.6from
backport-2914-3956-to-release-1.6

Conversation

@myasnikovdaniil

@myasnikovdaniil myasnikovdaniil commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

Backport of #2914 and #3956 to release-1.6. It sits on top of #4683, which is merged here now.

#2914: Update on an Application rebuilds the HelmRelease and sends a full-replace PUT, so metadata owned by other controllers was dropped, helm-controller's finalizers.fluxcd.io among it. A delete landing before helm-controller added the finalizer back removed the HelmRelease without helm uninstall and orphaned every resource of the release. Now Update carries over finalizers, ownerReferences and labels/annotations outside the apps.cozystack.io- prefix from the live object, mergeMaps no longer returns the shared config map that requests wrote into, and a conflict retry rebuilds the carried metadata from the object it just read instead of writing the stale copy back.

#3956: a required spec field written with nothing under it (replicas:) reached Helm as null, Helm coalescing deletes it together with the chart default before schema validation, and the whole release failed with "missing property" for a property the spec does contain. Such a null is removed on Create and Update now so the chart default applies, and the client gets a warning with the field name. Optional fields, free-form subtrees, user-keyed map entries and array elements keep their nulls. Admission validates the normalized copy, the caller's object is not mutated anymore.

On this branch #4683 inlined the garbage collection finalizer merge and the deletion metadata copy, because applyLiveMetadata didn't exist yet. They move into applyLiveMetadata here, so Update and its helpers match main. Suspend preservation (#4411) is not on this branch and not part of this backport.

The example field in #3956, talos.registryMirrors, doesn't exist in the kubernetes chart on this branch, so the tests use required fields that do (talos.installerRepository, talos.version, controlPlane.replicas). The node group null is its own case, because nodeGroups is still a typed map here and its entries require instanceType. Three test helpers main has from #3315 are copied into a test-only file. Production code needed one import beyond the cherry-picks.

Validation

  • go build ./... and go vet ./pkg/registry/... are clean and go test ./pkg/registry/... passes on the head, rebased onto release-1.6 after [Backport release-1.6] fix(api): let the garbage collector finalize Applications #4683 merged.
  • The Helm coalescing test renders this branch's own chart values and schemas, Helm rejects the empty required fields before the repair and accepts them after it.
  • Reverting applyLiveMetadata to the plain live finalizer list, skipping the retry rebuild, or validating the un-normalized object each turns the related tests red.

Downstream repositories

The change touches the Application REST handler and its tests only: no chart, values, schema, CRD, version enum or hack/ file, so none of the trigger map entries match.

Release note

fix(api): updating an Application no longer strips finalizers, ownerReferences and foreign labels/annotations from its HelmRelease, which could make a following delete skip helm uninstall and orphan the release's resources. A required field written with no value in an application spec is treated as unset so the chart default applies, instead of failing the whole release.

mattia-eleuteri and others added 5 commits October 14, 2026 12:36
Update rebuilds the HelmRelease from the Application and issues a
full-replace PUT, which stripped metadata owned by other controllers:
finalizers, ownerReferences, and any labels or annotations outside the
apps.cozystack.io- prefix. Losing helm-controller's
finalizers.fluxcd.io finalizer is the harmful case: a delete landing
before helm-controller re-adds it removes the HelmRelease without
running helm uninstall, orphaning every resource of the release (VMs,
databases, PVCs). A label patch immediately followed by a delete - a
common client pattern - hits that window almost every time.

Fetch the current HelmRelease before the PUT and carry over its
finalizers, ownerReferences, and non-prefixed labels and annotations.
Keys under the apps.cozystack.io- prefix remain owned by the API and
are recomputed from the Application, so label/annotation deletions
still propagate.

Signed-off-by: mattia-eleuteri <mattia@hidora.io>
(cherry picked from commit a18cbc5)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
mergeMaps returned one of its inputs whenever the other was nil. Create
and Update seed the HelmRelease labels with
mergeMaps(r.releaseConfig.Labels, ...) and then write the application
metadata labels into the result in place, so for an Application with no
labels the writes landed on the long-lived config map shared by every
request for the kind: requests cross-contaminate (application.name
varies per request) and concurrent map writes crash the apiserver.

Signed-off-by: mattia-eleuteri <mattia@hidora.io>
(cherry picked from commit 6d541c6)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A 409 on the HelmRelease PUT means the live object changed after the
read the carry-over was built from, and the writer behind it is normally
helm-controller - the controller whose finalizer the carry-over exists
to protect, and which re-adds it on the reconcile following a
metadata-only update. Retrying with the metadata read before the
conflict wrote that stale copy back and dropped the finalizer the
reconcile had just restored, reopening the orphaning window the
carry-over closes.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
(cherry picked from commit 31948c8)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
An Application spec reaches flux verbatim as HelmRelease .spec.values,
and Helm's value coalescing deletes a null-valued key together with the
chart default underneath it, before the result is validated against
values.schema.json. A required field written with nothing under it is
therefore rejected for being absent:

    talos:
      registryMirrors:

fails the whole release with "at '/talos': missing property
'registryMirrors'", naming a property the spec does contain, and nothing
reaching the user points at the empty key they typed.

Remove such a key on the write path, so the chart's own default applies
to it. Filling in the schema default instead would pin today's value
into the stored HelmRelease and freeze the application on it, so a later
bump of that default in the chart would never reach an application that
had once been created with an empty key.

Only required properties are repaired. A required property's null cannot
be load-bearing for anyone, because today it fails the render outright,
so resolving it is pure repair. Where the schema allows the field to be
absent, a null keeps the meaning it has now - the chart default is
suppressed and the key does not reach the template - and that meaning is
worth keeping: 167 optional fields across the catalogue carry a
non-trivial default, placeholder credentials and example S3 buckets
among them, so resolving a user's null to one of those would be worse
than leaving the key out.

Free-form subtrees, null map entries under a user-chosen key (a node
group, a registry host) and array elements are left as written, because
there a null may be the payload rather than an omission. Every removal
is reported back to the client as a warning, so the repair is visible
rather than silent.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 2e88b39)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A schema default cannot fill arbitrary map entries or replacement array
items. Keep those nulls, nullable fields and required fields without a
non-null default unchanged so Helm retains their validation semantics.

Validate the normalized copy on both write paths so admission sees the
values that reach storage without mutating the caller object.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 0474613)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
@coderabbitai

coderabbitai Bot commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: cozystack/cozystack/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 402e2b94-4c15-4a19-bdb2-d0aad0d53b42

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size/XXL This PR changes 1000+ lines, ignoring generated files area/api Issues or PRs related to the cozystack-api aggregated API server area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug labels Oct 14, 2026
@myasnikovdaniil
myasnikovdaniil merged commit 63dc235 into release-1.6 Oct 14, 2026
43 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-2914-3956-to-release-1.6 branch October 14, 2026 10:11
myasnikovdaniil added a commit that referenced this pull request Oct 14, 2026
…ed across an app update (#4928)

## What this PR does

Backport of #4411 to `release-1.6`.

`REST.Update` rebuilds the HelmRelease from the Application and writes
it back whole, and Application has no suspension, so every edit through
the API resumes a release that an operator or a controller suspended.
The CNPG restore driver depends on it: it suspends the target release,
patches the Postgres app through this API, purges the Cluster and
resumes only after the purge, so the next render lands
`bootstrap.recovery` on an empty namespace. When the patch resumes the
release, helm-controller upgrades right away, the Cluster gets rendered
while the purge still runs, the driver deletes it, and nothing renders
it again because the values don't change anymore. RestoreJob stays
`Running` until its deadline.

Why it is needed on this line now. The race was hidden by timing: Update
dropped `finalizers.fluxcd.io`, helm-controller spent its first pass
putting it back and requeued, and that delay usually let the purge
finish first. #4683 and #4919 keep the finalizer across the write, so
the delay is gone. After they landed, `postgres-2-backup-roundtrip`
failed on v1.6.5-rc.1
(https://github.com/cozystack/cozystack/actions/runs/38583543706) and in
two earlier PR runs, each time stuck at `TargetPurged: Cluster ...
deleted; awaiting chart-rendered re-bootstrap`. In the snapshot of the
rc run the HelmRelease of the restore target upgraded to v2 at
16:16:13Z, the Cluster was deleted at 16:16:22Z, and no Cluster of the
target exists afterwards.

The cherry-pick conflicted only in comments of `REST.Update`, where this
branch already has the metadata carry-over. The two `Spec.Suspend`
assignments merged unchanged and the comments read as they do on main.

### Validation

- `go build ./...` is clean, `go vet ./pkg/registry/...` is clean, `go
test ./pkg/registry/... ./internal/backupcontroller/...` passes.
- Removing either `Spec.Suspend` assignment turns its test red:
`TestUpdate_PreservesSuspend` for the first write,
`TestUpdate_RefreshesSuspendOnConflictRetry` for the conflict retry.
- Not run here: the restore round trip on a live cluster, the E2E of
this PR covers it.

### Downstream repositories

It changes how the aggregated API writes the HelmRelease back, no
downstream repository restates that.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(api): app edit through cozystack API no longer resumes a suspended HelmRelease, so postgres restore waits for its purge before the cluster is rendered again.
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api Issues or PRs related to the cozystack-api aggregated API server area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants