Replace y-kustomize with buckety-controller#80
Conversation
…ller
First downstream consumer of Yolean/buckety-controller (initial-impl
on main). Swaps the y-kustomize HTTP-fetched setup-topic Job pattern
for the Buckety CRD on the kafka path; blobs path stays on the
existing y-kustomize Job until the s3 buckety driver lands.
What's in:
- buckety-controller/upstream/ vendors deploy/kustomize/{crd,controller}/
from the buckety-controller repo. The webhook.yaml is intentionally
not vendored (and the corresponding `- webhook.yaml` line is
stripped from controller/kustomization.yaml): ystack does not run
cert-manager, and the upstream webhook requires it for CA-bundle
injection. Documented in upstream/README.md.
- buckety-controller/ is the local overlay: namespace `buckety`, a
secretGenerator-driven config Secret with one kadm backend pointing
at y-bootstrap.kafka.svc.cluster.local:9092, a clusterrolebinding
subject patch onto the namespaced ServiceAccount, and a
deployment-patch.yaml that passes --enable-webhook=false so the
controller starts without a TLS cert (matches the cert-less
deployment recipe documented in buckety-controller's
docs/SCAFFOLDING.md after issue #4).
- k3s/15-buckety/ is the new yconverge step. It depends only on
40-kafka (controller talks to the broker on first reconcile, so
staging it after kafka avoids early-reconcile noise).
- registry/builds-topic/ replaces its HTTP fetch with a Buckety:
backend=cluster-kafka, spec.name=ystack.builds-registry.stream.json
(the friendly operator-chosen name COSI could not give us),
defaultAccess mints Secret topic-builds-registry with flat keys
bootstrap+topic that the registry consumes via plain
secretKeyRef. No init-container shim.
- kafka/validate-topic/ does the same swap with
spec.name=y-cluster-validate-ystack so the existing
y-cluster-validate-ystack produce/consume script can keep
using its hard-coded topic name.
- registry/builds-prep/ drops the setup-topic-prep URL fetch
(no per-namespace ServiceAccount needed when the controller
does the topic create). The setup-bucket-prep URL stays until
the s3 driver lands.
- k3s/60-builds-registry/yconverge.cue replaces the
41-kafka-y-kustomize dep with 15-buckety. The blobs path
keeps 31-blobs-y-kustomize.
- bin/y-cluster-validate-ystack swaps the kafka topic create
check from `kubectl wait Job complete` to
`kubectl wait Buckety Ready`, and drops the now-orphan
y-kustomize kafka base check.
- e2e/agents-clusterautomation-acceptance-linux-amd64.sh adds
one sideload step after provision:
y-cluster images load $BUCKETY_CONTROLLER_OCI --context=local
Default path: ~/Yolean/buckety-controller/oci, overridable via
env. Fails fast with a build hint if the OCI layout is missing.
Verified locally against PR #3 head (commit cb45545, image digest
sha256:11ae9b338310e953b48b26fe31cbde3247fdd09928a1d0c3efd89964d387ccfd):
- 8 yconverge phases green from a clean k3s+docker provision
- y-cluster-validate-ystack: 35 passed / 0 failed (including
kafka topic create + produce + consume against the
buckety-provisioned topic)
- Buckety/builds-registry reaches Ready=True in ~5s
- `rpk topic list` shows the operator-chosen names
(ystack.builds-registry.stream.json,
y-cluster-validate-ystack) end-to-end
- Issued Secrets carry the flat keys (bootstrap, topic) per
the kadm driver schema
Not in this commit (deliberate scope):
- s3 driver swap. The buckety-controller s3 driver is still a
stub returning ErrNotImplemented; blobs-versitygw remains on
the existing y-kustomize Job pattern. Lands in a follow-up
once the upstream s3 driver fills in.
- Image pin. The acceptance script sideloads a locally-built
OCI from ~/Yolean/buckety-controller/oci; once
buckety-controller cuts a tagged release with the new
per-commit timestamp tag + pinned digest workflow
(Yolean/buckety-controller@172e1ca), this overlay should
switch to vendoring deploy/kustomize/release/ and pinning
the image by digest. Follow-up.
- Driver-version e2e (SPEC scenario #9 in
buckety-controller). Not exercised; needs three
differently-stamped binaries that the upstream
driver-version harness doesn't yet ship.
Builds on 7a8e4cc (kafka topic path) with the s3 bucket path now live against the buckety-controller s3 driver (Yolean/buckety-controller@8ea9e13, image digest sha256:0c56d6f9205b243766e777fa2a4b0f028fa62c2487c4c24ccc54e587943ef495). What's new: - buckety-controller/buckety-controller.yaml adds a second backend `cluster-objects` for the s3 driver: implementation versitygw, endpoint y-s3-api.blobs.svc.cluster.local:9000, region us-east-1, forcePathStyle. accessKeyID/secretAccessKey are envsubst:"true" fields populated from a local-copy versitygw-creds Secret via the controller Deployment's env. - buckety-controller/kustomization.yaml adds a second secretGenerator (versitygw-creds) carrying the same dev literal credentials as blobs-versitygw/defaultsecret/. The upstream Secret in the blobs namespace cannot cross into buckety; envsubst is startup-time so the controller Pod needs these as env vars at first boot. Rotating credentials means rotating both copies. - buckety-controller/deployment-patch.yaml adds VERSITYGW_ROOT_ACCESSKEY and VERSITYGW_ROOT_SECRETKEY env vars sourced from versitygw-creds via secretKeyRef. Kustomize rewrites the Secret name-suffix hash automatically. - registry/builds-bucket/ swaps from the y-kustomize URL fetch for the setup-bucket Job to a Buckety: metadata.name builds-registry-bucket (distinct from the kafka-side builds-registry-topic), backend cluster-objects, spec.name ystack-builds-registry (friendly bucket name on versitygw), defaultAccess ReadWrite with credentialsSecretName builds-registry-bucket. - registry/builds-topic/ rename: metadata.name builds-registry -> builds-registry-topic so both Bucketies coexist in the ystack namespace. - k3s/60-builds-registry/deployment-s3.yaml secret key names updated to match the s3 driver's output keys: accesskey -> accessKeyID secretkey -> secretAccessKey REGISTRY_STORAGE_S3_BUCKET and REGISTRY_STORAGE_S3_REGION switch from hardcoded literals to secretKeyRef so the bucket name and region come from the issued Secret. - registry/builds-prep/ removed. Its only contents were two y-kustomize URL fetches for per-namespace setup-Job RBAC; with both Buckety swaps in place the per-namespace SA is redundant (the controller does the backend work). - k3s/60-builds-registry/kustomization.yaml drops the builds-prep resource entry. yconverge.cue drops the 31-blobs-y-kustomize and 29-y-kustomize imports; the dep chain is now 60-builds-registry -> 15-buckety -> {30-blobs, 40-kafka}. y-kustomize is entirely retired from the converge graph for this stack. - k3s/15-buckety/yconverge.cue adds 30-blobs to its deps: the s3 driver needs versitygw reachable on first reconcile. Verified locally: - 8 yconverge phases green, no y-kustomize-related fetches in the converge path. - y-cluster-validate-ystack: 33 passed / 0 failed. Drop from the 35 in 7a8e4cc reflects the now-retired y-kustomize blobs-base serving checks. - y-build builds an image and pushes it to the buckety-provisioned S3-backed registry; node-pull via the registries.yaml mirror succeeds. - Both Bucketies (builds-registry-topic, builds-registry-bucket) reach Ready=True; issued Secrets carry the documented flat keys consumed by the registry Deployment via plain secretKeyRef. - Backend resource names on the broker / object store match the operator-chosen identities (ystack.builds-registry.stream.json, ystack-builds-registry, y-cluster-validate-ystack). Not in this commit (deliberate scope): - Image pin via buckety-controller tagged-release artefacts. Once the per-commit timestamp tag + pinned-digest workflow (Yolean/buckety-controller@172e1ca) cuts a release, this overlay should vendor deploy/kustomize/release/ instead of base/ and pin the image by digest. - Driver-version e2e. Needs three differently-stamped binaries the upstream harness doesn't yet ship. - Per-consumer credential scoping (SASL/SCRAM, IAM users). v1alpha2 work upstream; for now all BucketyAccess instances for the same Buckety receive identical root credentials.
Builds on 946ce24 (s3 path swap). With both Bucketies in place nothing in the converge graph references y-kustomize anymore. This commit deletes the now-unreachable source tree, the provision-side port forward, and the comment scaffolding around the old serving model. Deleted: - y-kustomize/ (Deployment + Service + HTTPRoute + serve config for the in-cluster y-cluster-serve instance) - k3s/29-y-kustomize/, k3s/31-blobs-y-kustomize/, k3s/41-kafka-y-kustomize/ (the yconverge steps that installed the Deployment and loaded per-domain source Secrets) - blobs-versitygw/setup-bucket-{,prep-}y-kustomize/ and kafka/setup-topic-{,prep-}y-kustomize/ (the source bases that were rendered into y-kustomize.{group}.{name} Secrets for the in-cluster service to serve) - bin/acceptance-y-kustomize-local (the standalone test for the host-local serve mode; no consumer in ystack any more) Updated: - cluster-configs/local-{docker,qemu}/y-cluster-provision.yaml drop the host:8944 -> guest:8944 PortForward and the surrounding comment block. ystack now only forwards 6443 / 80 / 443. - bin/y-cluster-validate-ystack drops the y-kustomize-bases section; rewords one example comment that referenced y-kustomize:8944 to a generic placeholder. - e2e/agents-clusterautomation-acceptance-linux-amd64.sh drops the cleanup-trap probe for stale host-local y-cluster serve on :8944, the Phase 2 comment block about the in-cluster Deployment, and the y-kustomize-flavoured phase labels. - k3s/README.md rewritten to describe the buckety-controller flow instead of the y-kustomize-serving flow that no longer exists. Retained (intentional): - bin/y-kustomize and bin/y-kustomize-traverse wrappers for the upstream sigs.k8s.io/kustomize CLI and y-cluster's kustomize-traverse subcommand respectively. Unrelated to the in-cluster serve. - One comment in k3s/60-builds-registry/yconverge.cue noting the historical migration ("y-kustomize is retired from this dep chain ..."). Kept for archaeology. - runner.Dockerfile lines copying the wrapper binaries. Verified: - y-cluster-validate-ystack: 33 passed / 0 failed against a clean provision with no y-kustomize Deployment, no host port 8944 forward, no host-local serve. - Acceptance script runs to teardown without --keep-on-failure tripping. - git grep "y-kustomize" returns only the intentional references above. Companion spec filed: Yolean/specs@9281c13 documents which y-cluster surfaces ystack has stopped depending on, so the y-cluster maintainer can plan accordingly. No phase-out strategy proposed there or here.
Parallel converge bases for a minio-backed objects path: buckety-controller-minio/, k3s/15-buckety-minio-disabled/, k3s/60-builds-registry-minio-disabled/, k3s/62-buildkit-minio-disabled/. Versitygw stays the default; the -disabled suffix keeps the minio bases out of any global converge sweep. Consumers select minio by pointing yconverge at the -minio-disabled bases. The e2e acceptance and y-cluster-validate-ystack gain an internal toggle for exercising the minio path; the rollout/restart checks target deploy/minio when set. 30-blobs-minio-disabled/yconverge.cue is no longer a stub. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The existing call in k3s/20-gateway/yconverge.cue ran with no
HTTPRoutes/GRPCRoutes yet present, so /etc/hosts never picked up
route hostnames like builds-registry.ystack.svc.cluster.local.
Acceptance silently relied on whatever stale entry happened to be
in /etc/hosts from prior testing; the y-build registry probe
("Skaffold need local access ...") surfaced as a sporadic late
failure whenever that stale entry got cleared.
Adds the same idempotent exec check to every base that creates a
route -- 50-monitoring, 60-builds-registry{,-minio-disabled},
62-buildkit{,-minio-disabled} -- so y-k8s-ingress-hosts runs after
the new routes exist and the write picks them up.
Also drops the `|| echo 'WARNING ...'` fallback from all six call
sites. Swallowing the failure let the converge step succeed with
/etc/hosts inconsistent, then deferred the user-visible failure to
a much later step with no traceable cause. Hard fail at the source.
This deepens the y-cluster <-> y-k8s-ingress-hosts coupling that
Yolean/y-cluster#11 flags as a
distribution concern: the host-side /etc/hosts contract now hard
requires the helper to run inside the converge, not as an optional
post-step. The dns-hint-ip mechanism the helper consumes was
adopted in commit 2058e5c ("gateway: retire OVERRIDE_IP env var,
adopt yolean.se/dns-hint-ip"); this commit extends that contract
to every base that adds routes after the Gateway is up.
Verified with BLOBSTORE=minio acceptance: 33/0 (including y-build,
which previously failed at host-side curl for both blobstores once
/etc/hosts had been cleared between runs).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the `y-cluster lifetime` subcommand for cost-control auto-expiry (Yolean/y-cluster#29). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adopts the CI-tagged buckety-controller image from commit ccfb662 on the YoleanAgents/buckety-controller secret-safety-and-events branch. Vendored CRD + controller manifests are unchanged (they've been identical since edc8eb0 "initial impl"); only the image ref in upstream/controller/deployment.yaml pins to the per-commit timestamp tag published by the fork's e2e workflow. Consumer setup: crane pull the image into $HOME/Yolean/buckety-controller/oci (or override BUCKETY_CONTROLLER_OCI) so the acceptance script's sideload picks up the tagged image; kubelet then finds it locally at IfNotPresent. Verified: 33/0 on default (versitygw) and BLOBSTORE=minio acceptance runs. Buckety reconcile of both the kafka topic and the s3 bucket succeeds; registry pod picks up the resulting credential Secret and serves /v2/ as before. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Review from the buckety-controller side. The direction is right; the way the controller is consumed can shrink a lot, and the webhook can stay on. Four points: 1. Replace
|
Replaces y-kustomize from #70 with https://github.com/Yolean/buckety-controller/.
Inspired by https://github.com/GoogleCloudPlatform/k8s-config-connector but vendor neutral and with the assumption that we can make in-tree scope creeps if/when our need grows.