feat(infra): self-monitor the observability stack + Overpass alert
Adds the visibility that would have caught the corrupt-WAL incident, plus a direct Overpass upstream alert. - prometheus.yml: self-scrape Prometheus (localhost:9090) and Loki (loki:3100). Prometheus scraped everything except itself, so TSDB health (compaction failures, WAL corruption, head series, retention deletions) was invisible. - monitoring-health.json: new "Monitoring Health" dashboard — TSDB compaction/WAL failures, retention deletions/hour, head series, samples/s, block bytes vs size-retention limit, retention depth (oldest-sample age), Loki ingestion rate + memory chunks. - alerts.yml: prometheus-compaction-failing (any compaction failure or WAL corruption in 1h) and overpass-upstream-unhealthy (>20% upstream failure over 10m — sustained public-Overpass degradation, distinct from the symptom-level Caddy-502 alert). Validated: promtool check config, YAML parse, JSON parse. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
42b90a7b78
commit
3a43784f07
3 changed files with 254 additions and 0 deletions
|
|
@ -43,6 +43,20 @@ scrape_configs:
|
|||
static_configs:
|
||||
- targets: ["caddy:2019"]
|
||||
|
||||
# Self-monitoring. Prometheus did not scrape its own /metrics, so TSDB
|
||||
# health (compaction failures, WAL corruption, head series, retention
|
||||
# deletions) was invisible — which is how a corrupt WAL silently
|
||||
# collapsed retention to a ~2h window for days. Scraping itself + Loki
|
||||
# surfaces that on the "Monitoring Health" dashboard and the
|
||||
# prometheus-compaction-failing alert.
|
||||
- job_name: "prometheus"
|
||||
static_configs:
|
||||
- targets: ["localhost:9090"]
|
||||
|
||||
- job_name: "loki"
|
||||
static_configs:
|
||||
- targets: ["loki:3100"]
|
||||
|
||||
# BRouter runs on a separate Hetzner Robot host (ullrich.is), reached
|
||||
# over the vSwitch at 10.0.1.10. cAdvisor there is scoped to trails-
|
||||
# labeled containers only — we don't collect metrics for any of the
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue