BRouter host observability + docs
Lands sections 6 and 8 of relocate-brouter-to-dedicated-host on top of the host compose in #291. ## Observability (section 6) - **Prometheus**: new `brouter-cadvisor` job scraping `10.0.1.10:8080` over the vSwitch, labeled `host="brouter"`. - **cAdvisor sidecar** on the dedicated host's compose (`--docker_only --whitelisted_container_labels=trails.cool.service`) so metrics only cover trails containers, never the operator's unrelated workloads on the shared box. - **Promtail sidecar** on the dedicated host, Docker SD with relabel- drop on missing `trails.cool.service` label, pushing to flagship Loki at `http://10.0.0.2:3100/loki/api/v1/push`. - **Flagship compose**: Loki now publishes port 3100 on the vSwitch IP only (10.0.0.2:3100) — Hetzner Cloud firewall blocks it from the public internet. - **Grafana dashboard**: `brouter.json` — scrape up/down, request rate (from Planner-side `brouter_request_duration_seconds`), p50/p95/p99, container memory/CPU, Loki logs panel. - **Alert**: `brouter-scrape-down` fires on `up{job="brouter-cadvisor"} < 1 for 2m`; `noDataState: Alerting` so a total scrape failure still pages. Operator needs one UFW rule on the dedicated host for the cAdvisor port — documented in `infrastructure/brouter-host/README.md`. ## Documentation (section 8) - `CLAUDE.md` — hosts table + updated deployment table with SSH targets per workflow; BRouter host SSH is `-p 2232 trails@...`, different key. - `docs/architecture.md` — Hosting section rewritten to cover both hosts, vSwitch boundary, and the observability-scoping rationale for the shared dedicated host. - `docs/deployment.md` (new) — full operator runbook: host layout, first-time BRouter provisioning, SOPS rotation (including the macOS `SOPS_AGE_KEY_FILE` gotcha), cutover procedure with rollback, manual workflow triggers. Task 8.4 (infrastructure/README.md) skipped: that file doesn't exist and the ground is covered by brouter-host/README.md + docs/deployment.md. ## Validation - `docker compose config` on both the flagship and brouter-host compose files — both validate. - `pnpm typecheck`, `pnpm lint`, `pnpm test` — all clean (full turbo cache hits; no code changes in this commit). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
34b2e44ed1
commit
9e598fb6a1
12 changed files with 390 additions and 23 deletions
|
|
@ -68,9 +68,61 @@ services:
|
|||
labels:
|
||||
trails.cool.service: "brouter-caddy"
|
||||
|
||||
# cAdvisor — container metrics scraped by flagship Prometheus over the
|
||||
# vSwitch. We set container label filters so the exposed metrics only
|
||||
# cover trails-labeled containers (brouter + caddy), not the operator's
|
||||
# other workloads on this shared host.
|
||||
cadvisor:
|
||||
image: gcr.io/cadvisor/cadvisor:latest
|
||||
container_name: trails-brouter-cadvisor
|
||||
restart: unless-stopped
|
||||
privileged: true
|
||||
ports:
|
||||
- "10.0.1.10:8080:8080"
|
||||
volumes:
|
||||
- /:/rootfs:ro
|
||||
- /var/run:/var/run:ro
|
||||
- /sys:/sys:ro
|
||||
- /var/lib/docker:/var/lib/docker:ro
|
||||
- /dev/disk/:/dev/disk:ro
|
||||
command:
|
||||
# Only consider containers (not the host) and only those that carry
|
||||
# our label; filters out everything else on the shared host.
|
||||
- --docker_only=true
|
||||
- --store_container_labels=false
|
||||
- --whitelisted_container_labels=trails.cool.service
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
labels:
|
||||
trails.cool.service: "brouter-cadvisor"
|
||||
|
||||
# Promtail — tails Docker logs for trails-labeled containers only and
|
||||
# pushes to flagship Loki over the vSwitch. Does NOT scrape other
|
||||
# containers' logs on this host.
|
||||
promtail:
|
||||
image: grafana/promtail:latest
|
||||
container_name: trails-brouter-promtail
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- ./promtail-config.yml:/etc/promtail/config.yml:ro
|
||||
- promtail-positions:/tmp
|
||||
command: ["-config.file=/etc/promtail/config.yml"]
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
labels:
|
||||
trails.cool.service: "brouter-promtail"
|
||||
|
||||
volumes:
|
||||
caddy-data:
|
||||
caddy-config:
|
||||
promtail-positions:
|
||||
|
||||
networks:
|
||||
trails-brouter-internal:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue