Skip to content

Replace y-kustomize with buckety-controller#80

Open
solsson wants to merge 8 commits into
mainfrom
buckety-instead-of-y-kustomize
Open

Replace y-kustomize with buckety-controller#80
solsson wants to merge 8 commits into
mainfrom
buckety-instead-of-y-kustomize

Conversation

@solsson

@solsson solsson commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Replaces y-kustomize from #70 with https://github.com/Yolean/buckety-controller/.

Inspired by https://github.com/GoogleCloudPlatform/k8s-config-connector but vendor neutral and with the assumption that we can make in-tree scope creeps if/when our need grows.

k8s-qa and others added 8 commits June 16, 2026 08:18
…ller

First downstream consumer of Yolean/buckety-controller (initial-impl
on main). Swaps the y-kustomize HTTP-fetched setup-topic Job pattern
for the Buckety CRD on the kafka path; blobs path stays on the
existing y-kustomize Job until the s3 buckety driver lands.

What's in:

- buckety-controller/upstream/ vendors deploy/kustomize/{crd,controller}/
  from the buckety-controller repo. The webhook.yaml is intentionally
  not vendored (and the corresponding `- webhook.yaml` line is
  stripped from controller/kustomization.yaml): ystack does not run
  cert-manager, and the upstream webhook requires it for CA-bundle
  injection. Documented in upstream/README.md.

- buckety-controller/ is the local overlay: namespace `buckety`, a
  secretGenerator-driven config Secret with one kadm backend pointing
  at y-bootstrap.kafka.svc.cluster.local:9092, a clusterrolebinding
  subject patch onto the namespaced ServiceAccount, and a
  deployment-patch.yaml that passes --enable-webhook=false so the
  controller starts without a TLS cert (matches the cert-less
  deployment recipe documented in buckety-controller's
  docs/SCAFFOLDING.md after issue #4).

- k3s/15-buckety/ is the new yconverge step. It depends only on
  40-kafka (controller talks to the broker on first reconcile, so
  staging it after kafka avoids early-reconcile noise).

- registry/builds-topic/ replaces its HTTP fetch with a Buckety:
  backend=cluster-kafka, spec.name=ystack.builds-registry.stream.json
  (the friendly operator-chosen name COSI could not give us),
  defaultAccess mints Secret topic-builds-registry with flat keys
  bootstrap+topic that the registry consumes via plain
  secretKeyRef. No init-container shim.

- kafka/validate-topic/ does the same swap with
  spec.name=y-cluster-validate-ystack so the existing
  y-cluster-validate-ystack produce/consume script can keep
  using its hard-coded topic name.

- registry/builds-prep/ drops the setup-topic-prep URL fetch
  (no per-namespace ServiceAccount needed when the controller
  does the topic create). The setup-bucket-prep URL stays until
  the s3 driver lands.

- k3s/60-builds-registry/yconverge.cue replaces the
  41-kafka-y-kustomize dep with 15-buckety. The blobs path
  keeps 31-blobs-y-kustomize.

- bin/y-cluster-validate-ystack swaps the kafka topic create
  check from `kubectl wait Job complete` to
  `kubectl wait Buckety Ready`, and drops the now-orphan
  y-kustomize kafka base check.

- e2e/agents-clusterautomation-acceptance-linux-amd64.sh adds
  one sideload step after provision:
    y-cluster images load $BUCKETY_CONTROLLER_OCI --context=local
  Default path: ~/Yolean/buckety-controller/oci, overridable via
  env. Fails fast with a build hint if the OCI layout is missing.

Verified locally against PR #3 head (commit cb45545, image digest
sha256:11ae9b338310e953b48b26fe31cbde3247fdd09928a1d0c3efd89964d387ccfd):

- 8 yconverge phases green from a clean k3s+docker provision
- y-cluster-validate-ystack: 35 passed / 0 failed (including
  kafka topic create + produce + consume against the
  buckety-provisioned topic)
- Buckety/builds-registry reaches Ready=True in ~5s
- `rpk topic list` shows the operator-chosen names
  (ystack.builds-registry.stream.json,
   y-cluster-validate-ystack) end-to-end
- Issued Secrets carry the flat keys (bootstrap, topic) per
  the kadm driver schema

Not in this commit (deliberate scope):

- s3 driver swap. The buckety-controller s3 driver is still a
  stub returning ErrNotImplemented; blobs-versitygw remains on
  the existing y-kustomize Job pattern. Lands in a follow-up
  once the upstream s3 driver fills in.

- Image pin. The acceptance script sideloads a locally-built
  OCI from ~/Yolean/buckety-controller/oci; once
  buckety-controller cuts a tagged release with the new
  per-commit timestamp tag + pinned digest workflow
  (Yolean/buckety-controller@172e1ca), this overlay should
  switch to vendoring deploy/kustomize/release/ and pinning
  the image by digest. Follow-up.

- Driver-version e2e (SPEC scenario #9 in
  buckety-controller). Not exercised; needs three
  differently-stamped binaries that the upstream
  driver-version harness doesn't yet ship.
Builds on 7a8e4cc (kafka topic path) with the s3 bucket path now
live against the buckety-controller s3 driver
(Yolean/buckety-controller@8ea9e13, image digest
sha256:0c56d6f9205b243766e777fa2a4b0f028fa62c2487c4c24ccc54e587943ef495).

What's new:

- buckety-controller/buckety-controller.yaml adds a second
  backend `cluster-objects` for the s3 driver: implementation
  versitygw, endpoint y-s3-api.blobs.svc.cluster.local:9000,
  region us-east-1, forcePathStyle. accessKeyID/secretAccessKey
  are envsubst:"true" fields populated from a local-copy
  versitygw-creds Secret via the controller Deployment's env.

- buckety-controller/kustomization.yaml adds a second
  secretGenerator (versitygw-creds) carrying the same dev
  literal credentials as blobs-versitygw/defaultsecret/. The
  upstream Secret in the blobs namespace cannot cross into
  buckety; envsubst is startup-time so the controller Pod needs
  these as env vars at first boot. Rotating credentials means
  rotating both copies.

- buckety-controller/deployment-patch.yaml adds VERSITYGW_ROOT_ACCESSKEY
  and VERSITYGW_ROOT_SECRETKEY env vars sourced from
  versitygw-creds via secretKeyRef. Kustomize rewrites the
  Secret name-suffix hash automatically.

- registry/builds-bucket/ swaps from the y-kustomize URL fetch
  for the setup-bucket Job to a Buckety:
    metadata.name builds-registry-bucket (distinct from the
      kafka-side builds-registry-topic),
    backend cluster-objects,
    spec.name ystack-builds-registry (friendly bucket name on
      versitygw),
    defaultAccess ReadWrite with credentialsSecretName
      builds-registry-bucket.

- registry/builds-topic/ rename: metadata.name builds-registry
  -> builds-registry-topic so both Bucketies coexist in the
  ystack namespace.

- k3s/60-builds-registry/deployment-s3.yaml secret key names
  updated to match the s3 driver's output keys:
    accesskey -> accessKeyID
    secretkey -> secretAccessKey
  REGISTRY_STORAGE_S3_BUCKET and REGISTRY_STORAGE_S3_REGION
  switch from hardcoded literals to secretKeyRef so the
  bucket name and region come from the issued Secret.

- registry/builds-prep/ removed. Its only contents were two
  y-kustomize URL fetches for per-namespace setup-Job RBAC;
  with both Buckety swaps in place the per-namespace SA is
  redundant (the controller does the backend work).

- k3s/60-builds-registry/kustomization.yaml drops the
  builds-prep resource entry. yconverge.cue drops the
  31-blobs-y-kustomize and 29-y-kustomize imports; the dep
  chain is now 60-builds-registry -> 15-buckety ->
  {30-blobs, 40-kafka}. y-kustomize is entirely retired from
  the converge graph for this stack.

- k3s/15-buckety/yconverge.cue adds 30-blobs to its deps:
  the s3 driver needs versitygw reachable on first reconcile.

Verified locally:

- 8 yconverge phases green, no y-kustomize-related fetches in
  the converge path.
- y-cluster-validate-ystack: 33 passed / 0 failed. Drop from
  the 35 in 7a8e4cc reflects the now-retired y-kustomize
  blobs-base serving checks.
- y-build builds an image and pushes it to the
  buckety-provisioned S3-backed registry; node-pull via the
  registries.yaml mirror succeeds.
- Both Bucketies (builds-registry-topic,
  builds-registry-bucket) reach Ready=True; issued Secrets
  carry the documented flat keys consumed by the registry
  Deployment via plain secretKeyRef.
- Backend resource names on the broker / object store match
  the operator-chosen identities
  (ystack.builds-registry.stream.json,
  ystack-builds-registry,
  y-cluster-validate-ystack).

Not in this commit (deliberate scope):

- Image pin via buckety-controller tagged-release artefacts.
  Once the per-commit timestamp tag + pinned-digest workflow
  (Yolean/buckety-controller@172e1ca) cuts a release, this
  overlay should vendor deploy/kustomize/release/ instead of
  base/ and pin the image by digest.

- Driver-version e2e. Needs three differently-stamped binaries
  the upstream harness doesn't yet ship.

- Per-consumer credential scoping (SASL/SCRAM, IAM users).
  v1alpha2 work upstream; for now all BucketyAccess instances
  for the same Buckety receive identical root credentials.
Builds on 946ce24 (s3 path swap). With both Bucketies in place
nothing in the converge graph references y-kustomize anymore.
This commit deletes the now-unreachable source tree, the
provision-side port forward, and the comment scaffolding around
the old serving model.

Deleted:

- y-kustomize/ (Deployment + Service + HTTPRoute + serve config
  for the in-cluster y-cluster-serve instance)
- k3s/29-y-kustomize/, k3s/31-blobs-y-kustomize/,
  k3s/41-kafka-y-kustomize/ (the yconverge steps that
  installed the Deployment and loaded per-domain source
  Secrets)
- blobs-versitygw/setup-bucket-{,prep-}y-kustomize/ and
  kafka/setup-topic-{,prep-}y-kustomize/ (the source bases
  that were rendered into y-kustomize.{group}.{name} Secrets
  for the in-cluster service to serve)
- bin/acceptance-y-kustomize-local (the standalone test for
  the host-local serve mode; no consumer in ystack any more)

Updated:

- cluster-configs/local-{docker,qemu}/y-cluster-provision.yaml
  drop the host:8944 -> guest:8944 PortForward and the
  surrounding comment block. ystack now only forwards
  6443 / 80 / 443.
- bin/y-cluster-validate-ystack drops the y-kustomize-bases
  section; rewords one example comment that referenced
  y-kustomize:8944 to a generic placeholder.
- e2e/agents-clusterautomation-acceptance-linux-amd64.sh
  drops the cleanup-trap probe for stale host-local
  y-cluster serve on :8944, the Phase 2 comment block about
  the in-cluster Deployment, and the y-kustomize-flavoured
  phase labels.
- k3s/README.md rewritten to describe the buckety-controller
  flow instead of the y-kustomize-serving flow that no longer
  exists.

Retained (intentional):

- bin/y-kustomize and bin/y-kustomize-traverse wrappers for
  the upstream sigs.k8s.io/kustomize CLI and y-cluster's
  kustomize-traverse subcommand respectively. Unrelated to
  the in-cluster serve.
- One comment in k3s/60-builds-registry/yconverge.cue noting
  the historical migration ("y-kustomize is retired from this
  dep chain ..."). Kept for archaeology.
- runner.Dockerfile lines copying the wrapper binaries.

Verified:

- y-cluster-validate-ystack: 33 passed / 0 failed against a
  clean provision with no y-kustomize Deployment, no host
  port 8944 forward, no host-local serve.
- Acceptance script runs to teardown without
  --keep-on-failure tripping.
- git grep "y-kustomize" returns only the intentional
  references above.

Companion spec filed: Yolean/specs@9281c13 documents which
y-cluster surfaces ystack has stopped depending on, so the
y-cluster maintainer can plan accordingly. No phase-out
strategy proposed there or here.
Parallel converge bases for a minio-backed objects path:
buckety-controller-minio/, k3s/15-buckety-minio-disabled/,
k3s/60-builds-registry-minio-disabled/, k3s/62-buildkit-minio-disabled/.
Versitygw stays the default; the -disabled suffix keeps the minio
bases out of any global converge sweep. Consumers select minio by
pointing yconverge at the -minio-disabled bases.

The e2e acceptance and y-cluster-validate-ystack gain an internal
toggle for exercising the minio path; the rollout/restart checks
target deploy/minio when set. 30-blobs-minio-disabled/yconverge.cue
is no longer a stub.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The existing call in k3s/20-gateway/yconverge.cue ran with no
HTTPRoutes/GRPCRoutes yet present, so /etc/hosts never picked up
route hostnames like builds-registry.ystack.svc.cluster.local.
Acceptance silently relied on whatever stale entry happened to be
in /etc/hosts from prior testing; the y-build registry probe
("Skaffold need local access ...") surfaced as a sporadic late
failure whenever that stale entry got cleared.

Adds the same idempotent exec check to every base that creates a
route -- 50-monitoring, 60-builds-registry{,-minio-disabled},
62-buildkit{,-minio-disabled} -- so y-k8s-ingress-hosts runs after
the new routes exist and the write picks them up.

Also drops the `|| echo 'WARNING ...'` fallback from all six call
sites. Swallowing the failure let the converge step succeed with
/etc/hosts inconsistent, then deferred the user-visible failure to
a much later step with no traceable cause. Hard fail at the source.

This deepens the y-cluster <-> y-k8s-ingress-hosts coupling that
Yolean/y-cluster#11 flags as a
distribution concern: the host-side /etc/hosts contract now hard
requires the helper to run inside the converge, not as an optional
post-step. The dns-hint-ip mechanism the helper consumes was
adopted in commit 2058e5c ("gateway: retire OVERRIDE_IP env var,
adopt yolean.se/dns-hint-ip"); this commit extends that contract
to every base that adds routes after the Gateway is up.

Verified with BLOBSTORE=minio acceptance: 33/0 (including y-build,
which previously failed at host-side curl for both blobstores once
/etc/hosts had been cleared between runs).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the `y-cluster lifetime` subcommand for cost-control
auto-expiry (Yolean/y-cluster#29).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Includes workflow-scripts refactor (#27) and Fable-5 code review
(#30). No consumer-visible surface change from ystack's side.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adopts the CI-tagged buckety-controller image from commit ccfb662
on the YoleanAgents/buckety-controller secret-safety-and-events
branch. Vendored CRD + controller manifests are unchanged (they've
been identical since edc8eb0 "initial impl"); only the image ref
in upstream/controller/deployment.yaml pins to the per-commit
timestamp tag published by the fork's e2e workflow.

Consumer setup: crane pull the image into $HOME/Yolean/buckety-controller/oci
(or override BUCKETY_CONTROLLER_OCI) so the acceptance script's
sideload picks up the tagged image; kubelet then finds it locally
at IfNotPresent.

Verified: 33/0 on default (versitygw) and BLOBSTORE=minio acceptance
runs. Buckety reconcile of both the kafka topic and the s3 bucket
succeeds; registry pod picks up the resulting credential Secret and
serves /v2/ as before.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@solsson

solsson commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator Author

Review from the buckety-controller side. The direction is right; the way the controller is consumed can shrink a lot, and the webhook can stay on. Four points:

1. Replace buckety-controller/upstream/ with the remote release base

The vendored copy is ccfb662 (agents-fork branch, 2026-07-10, pre-v0.1.0). Everything since is missing: Secret cache scoping + the buckety.yolean.se/owned Secret label contract, recursive Delete, driver families (backend parameters defaults), and adoption safety - the vendored CRD has no spec.adoption/status.provenance. Vendored CRDs also drift from the image on every bump. checkit's cluster-g2/buckety/ consumes everything in one line instead:

resources:
- github.com/Yolean/buckety-controller/deploy/kustomize/release?ref=<full-sha>

That brings CRDs, RBAC, Deployment, webhook and a digest-pinned image that CI asserted matches the source, atomically per ref. The whole upstream/ tree, its README caveats and the refresh procedure go away; the ystack overlay keeps only buckety-controller.yaml (secretGenerator) and the creds env patch. If converge must work offline, vendor via a refresh script from the release base rather than hand-copying - still one ref, still digest-pinned.

2. Keep the webhook enabled: release-tls-selfsigned needs no cert-manager

--enable-webhook=false moves every apply-time rejection (invalid/unknown parameters, immutability violations, invalid resolved names, refused adoptions) into Ready=False conditions someone has to go read - a silent degradation under converge. The cert-manager objection is now solved upstream: deploy/kustomize/release-tls-selfsigned/ composes the release base with two digest-pinned kube-webhook-certgen Jobs (the ingress-nginx pattern) that mint the serving Secret and patch the VWC caBundle. Completed Jobs self-delete so converge re-runs them idempotently; startup ordering self-heals.

So this overlay becomes, in full:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/Yolean/buckety-controller/deploy/kustomize/release-tls-selfsigned?ref=<sha>
secretGenerator:
- name: buckety-controller-config
  files: [buckety-controller.yaml]
- name: versitygw-creds
  ...
patches:
- path: deployment-patch.yaml   # env only; the args override disappears

Smoke-verified on a throwaway k3d (no cert-manager): certgen Jobs complete, controller serves, and the webhook denies both an unregistered backend and an unknown driver parameter at apply time. Suggested acceptance lock-in: have y-cluster-validate-ystack apply a deliberately invalid Buckety and require the denial.

Ref to use: the first main sha containing deploy/kustomize/release-tls-selfsigned (branch webhook-certgen on buckety-controller, pending merge). Until then the newest release base is ee02df3 with image 20260717T064548Z published, which includes adoption safety - relevant to ystack, since converge-recreated Bucketys re-adopting their Retained buckets is exactly this repo's lifecycle.

3. Namespace via the k3s step convention

Drop buckety-controller/namespace.yaml and add a k3s/04-namespace-buckety step alongside 00-namespace-ystack/01-namespace-blobs/02-namespace-kafka. checkit's base deliberately ships no Namespace resource ("a delete of this base must never cascade into the namespace") - same rationale applies here, and the step convention already expresses namespace lifecycle separately.

4. buckety-controller-minio/ dedup

With the remote base, the minio variant reduces to its own buckety-controller.yaml + env patch over the same one-line resource ref - the duplicated namespace/clusterrolebinding/args plumbing goes away along with the main overlay's.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant