d8fd4ff655
The flagship Caddy ran its admin API on 0.0.0.0:2019, reachable by every container on the Docker network. The admin API can rewrite routes and proxy targets, so a journal/planner RCE could repoint traffic with no further auth. (The brouter-host Caddyfile already binds admin to localhost; the flagship didn't.) The admin endpoint also served Prometheus metrics (global `metrics` option → admin endpoint), and Prometheus scrapes caddy:2019 cross-container — so admin couldn't just move to loopback without breaking metrics. Split them: - admin localhost:2019 (loopback only). All reloads are in-container (`docker compose exec caddy caddy reload`) so they use this endpoint unaffected. - a dedicated `:2020` server exposes the read-only metrics handler; prometheus.yml now scrapes caddy:2020. Validated with `caddy validate` + `caddy adapt` against caddy:2: admin.listen=localhost:2019, srv on :2020 carries the metrics handler, the :443 site is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
75 lines
2.8 KiB
YAML
75 lines
2.8 KiB
YAML
global:
|
|
# 20s is plenty of resolution for a single-box self-hosted deployment.
|
|
# The previous 5s tripled (vs the 15s default) the sample volume and
|
|
# WAL growth for no practical benefit; all alert/recording rules use
|
|
# [5m] windows and `for:` >= 1m, so 20s evaluation is fine.
|
|
scrape_interval: 20s
|
|
evaluation_interval: 20s
|
|
|
|
scrape_configs:
|
|
- job_name: "journal"
|
|
metrics_path: "/api/metrics"
|
|
static_configs:
|
|
- targets: ["journal:3000"]
|
|
|
|
- job_name: "planner"
|
|
metrics_path: "/metrics"
|
|
static_configs:
|
|
- targets: ["planner:3001"]
|
|
|
|
- job_name: "postgres"
|
|
static_configs:
|
|
- targets: ["postgres-exporter:9187"]
|
|
|
|
- job_name: "node"
|
|
static_configs:
|
|
- targets: ["node-exporter:9100"]
|
|
|
|
- job_name: "cadvisor"
|
|
static_configs:
|
|
- targets: ["cadvisor:8080"]
|
|
# cAdvisor emits ~100 per-container metric families; our dashboards
|
|
# and alerts only use CPU, memory, and container_start_time. Drop the
|
|
# high-cardinality families we never query (per-filesystem, block-IO,
|
|
# per-task state, memory-failure counters, socket tables). These
|
|
# dominated the ~12k active series — dropping them is the main
|
|
# cardinality cut. Add a name back here if you start charting it.
|
|
metric_relabel_configs:
|
|
- source_labels: [__name__]
|
|
regex: 'container_(fs_.*|blkio_.*|tasks_state|memory_failures_total|memory_numa_.*|network_tcp.*|network_udp.*)'
|
|
action: drop
|
|
|
|
- job_name: "caddy"
|
|
static_configs:
|
|
# Caddy's admin API is loopback-only; metrics are served on a
|
|
# dedicated port (see infrastructure/caddy/Caddyfile).
|
|
- targets: ["caddy:2020"]
|
|
|
|
# Self-monitoring. Prometheus did not scrape its own /metrics, so TSDB
|
|
# health (compaction failures, WAL corruption, head series, retention
|
|
# deletions) was invisible — which is how a corrupt WAL silently
|
|
# collapsed retention to a ~2h window for days. Scraping itself + Loki
|
|
# surfaces that on the "Monitoring Health" dashboard and the
|
|
# prometheus-compaction-failing alert.
|
|
- job_name: "prometheus"
|
|
static_configs:
|
|
- targets: ["localhost:9090"]
|
|
|
|
- job_name: "loki"
|
|
static_configs:
|
|
- targets: ["loki:3100"]
|
|
|
|
# BRouter runs on a separate Hetzner Robot host (ullrich.is), reached
|
|
# over the vSwitch at 10.0.1.10. cAdvisor there is scoped to trails-
|
|
# labeled containers only — we don't collect metrics for any of the
|
|
# operator's other workloads on that shared host.
|
|
- job_name: "brouter-cadvisor"
|
|
static_configs:
|
|
- targets: ["10.0.1.10:8080"]
|
|
labels:
|
|
host: "brouter"
|
|
# Same high-cardinality drop as the local cadvisor job above.
|
|
metric_relabel_configs:
|
|
- source_labels: [__name__]
|
|
regex: 'container_(fs_.*|blkio_.*|tasks_state|memory_failures_total|memory_numa_.*|network_tcp.*|network_udp.*)'
|
|
action: drop
|