containerd/containerd#3412

k3s has upgraded to 1.3.0 in master since 834f7df7ecb8eb8be27ebb300f2bcb23f24397b2
and ofc we don't know when GKE's _CONTAINERD nodes will upgrade.

and is quite impractical with Kustomize due to:

 - Large repo that takes a long time to fetch
 - Namespace "default" is hard coded in bundle.yaml

... so let's install it by default.

http://prometheus-now-0.prometheus-operated.ystack.svc.cluster.local:9090/

prometheus-now-0.prometheus-operated.ystack.svc.cluster.local:9090/graph

…etention

… annotations

... despite such annotations being inflexible.
The rule with ServiceMonitor having to be in the same namespace as the
prometheus instance is quite impractical, in particular with Kustomize.

breaks kubefwd, but reachable with:
kubectl -n ystack port-forward svc/prometheus-operated 9090
kubectl -n ystack port-forward svc/alertmanager-operated 9093

... and the alert doesn't work

provision failures at k3s crictl info with the message
failed to connect: failed to connect, make sure you are running as root and the runtime has been started: context deadline exceeded

This reverts commit 09712775dc3b2a3f6bb677ec689aaac1b62bc230.

because kubefwd found so much to forward, and failed to actually forward alertmanager.
We do need local access to prometheus but kubefwd was pretty much my only reason
to keep monotiring resources in the ystack namespaces. It's cleaner this way.

For some reason I failed to get service discovery with a plain alertmanager-main service.

To troubleshoot I did:
kubectl -n ystack-monitoring get secret prometheus-now -o=jsonpath='{.data.prometheus\.yaml\.gz}' | base64 -D - | gunzip -
kubectl -n ystack-monitoring get endpoints alertmanager-operated

https://github.com/envoyproxy/envoy/blob/master/configs/google_com_proxy.v2.yaml

http://prometheus-now.ystack.svc.cluster.local/graph?g0.range_input=5m&g0.expr=envoy_cluster_upstream_cx_tx_bytes_total&g0.tab=0