Snapshot writer unconditionally sends x-amz-tagging header, breaks Cloudflare R2 (and other non-AWS backends) #3882

What happened?

vcluster snapshot create fails when writing to Cloudflare R2 with:

StatusCode: 501, api error NotImplemented: Header 'x-amz-tagging' with value '' not implemented

The S3 writer at pkg/snapshot/s3/store.go:249 unconditionally sets Tagging: aws.String(o.tagging) on every PutObject input. When o.tagging is empty, the AWS SDK v2 serializer still emits the x-amz-tagging header. R2 rejects the request because it does not implement object tagging at all — regardless of tag value.

input := &s3.PutObjectInput{
    Bucket:  aws.String(o.bucket),
    Key:     aws.String(o.key),
    Body:    &wrapper{body},
    Tagging: aws.String(o.tagging), // always set, breaks R2
}

What did you expect to happen?

Snapshot writes successfully to R2 (or any S3-compatible backend that doesn't support object tagging), as it does against AWS S3. Tagging should only be sent when the user explicitly provides a value.

How can we reproduce it (as minimally and precisely as possible)?

1. Configure an R2 bucket with S3-compat credentials.

2. Run:

   vcluster snapshot create -n my-vcluster my-vcluster \
     's3://my-r2-bucket/path/snap.tar.gz?access-key-id=<base64>&secret-access-key=<base64>&url=<base64 of https://<account>.r2.cloudflarestorage.com>&region=auto&force-path-style=true'

3. Inspect the snapshot-request ConfigMap:

   kubectl -n <host-ns> get cm -l vcluster.loft.sh/snapshot-request -o yaml

   Status reports phase: Failed with the 501 error above.

Setting a non-empty tagging URL query param doesn't help — R2 rejects the header regardless of value.

Anything else we need to know?

This file is forked from Velero's object store (the license header attributes Copyright the Velero contributors). Velero hit the same issue in vmware-tanzu/velero#7940 and resolved it in v1.17; the vcluster fork never caught up.

Suggested one-line fix:

input := &s3.PutObjectInput{
    Bucket: aws.String(o.bucket),
    Key:    aws.String(o.key),
    Body:   &wrapper{body},
}
if o.tagging != "" {
    input.Tagging = aws.String(o.tagging)
}

Host cluster Kubernetes version

$ kubectl version
Client Version: v1.35.4
Kustomize Version: v5.7.1
Server Version: v1.34.4+rke2r1

vcluster version

$ vcluster --version
0.33.1

VCluster Config

controlPlane:
  distro:
    k8s:
      enabled: true
  backingStore:
    etcd:
      deploy:
        enabled: true
        statefulSet:
          highAvailability:
            replicas: 3
  statefulSet:
    highAvailability:
      replicas: 3
  service:
    spec:
      type: LoadBalancer