Adds the visibility that would have caught the corrupt-WAL incident,
plus a direct Overpass upstream alert.
- prometheus.yml: self-scrape Prometheus (localhost:9090) and Loki
(loki:3100). Prometheus scraped everything except itself, so TSDB
health (compaction failures, WAL corruption, head series, retention
deletions) was invisible.
- monitoring-health.json: new "Monitoring Health" dashboard — TSDB
compaction/WAL failures, retention deletions/hour, head series,
samples/s, block bytes vs size-retention limit, retention depth
(oldest-sample age), Loki ingestion rate + memory chunks.
- alerts.yml: prometheus-compaction-failing (any compaction failure or
WAL corruption in 1h) and overpass-upstream-unhealthy (>20% upstream
failure over 10m — sustained public-Overpass degradation, distinct
from the symptom-level Caddy-502 alert).
Validated: promtool check config, YAML parse, JSON parse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
3a43784f07