Deploy Prometheus, Grafana, and Alertmanager

Deploy Prometheus And Grafana

This guide installs the Prometheus stack with Grafana and Alertmanager, exposes them via the Tailscale Gateway, and wires Headlamp metrics into Prometheus.

Step 1: Store Grafana admin credentials in Vault

Create the Vault KV entry that External Secrets will sync into the grafana-admin Secret.

kubectl -n vault exec -it vault-0 -- vault kv put kv/monitoring/grafana-admin \
  admin-user="REPLACE_ME" \
  admin-password="REPLACE_ME"

Step 2: Sync the Prometheus stack

ArgoCD will create the monitoring namespace, install the Prometheus Operator CRDs, and deploy the stack from infrastructure/prometheus/.

kubectl -n argocd get applications | rg prometheus

Step 3: Access the dashboards

Open the following URLs on the tailnet:

• Grafana: https://grafana.sudhanva.me
• Prometheus: https://prometheus.sudhanva.me
• Alertmanager: https://alertmanager.sudhanva.me

Step 4: Confirm Headlamp metrics

Headlamp exposes /metrics once HEADLAMP_CONFIG_METRICS_ENABLED is set. Prometheus discovers it via the ServiceMonitor in apps/headlamp/servicemonitor.yaml.

kubectl -n monitoring get servicemonitors