feat(infra): self-monitor the observability stack + Overpass alert

Adds the visibility that would have caught the corrupt-WAL incident,
plus a direct Overpass upstream alert.

- prometheus.yml: self-scrape Prometheus (localhost:9090) and Loki
  (loki:3100). Prometheus scraped everything except itself, so TSDB
  health (compaction failures, WAL corruption, head series, retention
  deletions) was invisible.
- monitoring-health.json: new "Monitoring Health" dashboard — TSDB
  compaction/WAL failures, retention deletions/hour, head series,
  samples/s, block bytes vs size-retention limit, retention depth
  (oldest-sample age), Loki ingestion rate + memory chunks.
- alerts.yml: prometheus-compaction-failing (any compaction failure or
  WAL corruption in 1h) and overpass-upstream-unhealthy (>20% upstream
  failure over 10m — sustained public-Overpass degradation, distinct
  from the symptom-level Caddy-502 alert).

Validated: promtool check config, YAML parse, JSON parse.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-06-09 12:21:04 +02:00
parent 42b90a7b78
commit 3a43784f07
3 changed files with 254 additions and 0 deletions

View file

@ -43,6 +43,20 @@ scrape_configs:
static_configs:
- targets: ["caddy:2019"]
# Self-monitoring. Prometheus did not scrape its own /metrics, so TSDB
# health (compaction failures, WAL corruption, head series, retention
# deletions) was invisible — which is how a corrupt WAL silently
# collapsed retention to a ~2h window for days. Scraping itself + Loki
# surfaces that on the "Monitoring Health" dashboard and the
# prometheus-compaction-failing alert.
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "loki"
static_configs:
- targets: ["loki:3100"]
# BRouter runs on a separate Hetzner Robot host (ullrich.is), reached
# over the vSwitch at 10.0.1.10. cAdvisor there is scoped to trails-
# labeled containers only — we don't collect metrics for any of the