trails-cool/trails
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

BRouter host observability + docs

Browse source 
Lands sections 6 and 8 of relocate-brouter-to-dedicated-host on top
of the host compose in #291.

## Observability (section 6)

- **Prometheus**: new `brouter-cadvisor` job scraping
  `10.0.1.10:8080` over the vSwitch, labeled `host="brouter"`.
- **cAdvisor sidecar** on the dedicated host's compose
  (`--docker_only --whitelisted_container_labels=trails.cool.service`)
  so metrics only cover trails containers, never the operator's
  unrelated workloads on the shared box.
- **Promtail sidecar** on the dedicated host, Docker SD with relabel-
  drop on missing `trails.cool.service` label, pushing to flagship
  Loki at `http://10.0.0.2:3100/loki/api/v1/push`.
- **Flagship compose**: Loki now publishes port 3100 on the vSwitch
  IP only (10.0.0.2:3100) — Hetzner Cloud firewall blocks it from
  the public internet.
- **Grafana dashboard**: `brouter.json` — scrape up/down, request
  rate (from Planner-side `brouter_request_duration_seconds`),
  p50/p95/p99, container memory/CPU, Loki logs panel.
- **Alert**: `brouter-scrape-down` fires on
  `up{job="brouter-cadvisor"} < 1 for 2m`; `noDataState: Alerting`
  so a total scrape failure still pages.

Operator needs one UFW rule on the dedicated host for the cAdvisor
port — documented in `infrastructure/brouter-host/README.md`.

## Documentation (section 8)

- `CLAUDE.md` — hosts table + updated deployment table with SSH
  targets per workflow; BRouter host SSH is `-p 2232 trails@...`,
  different key.
- `docs/architecture.md` — Hosting section rewritten to cover both
  hosts, vSwitch boundary, and the observability-scoping rationale
  for the shared dedicated host.
- `docs/deployment.md` (new) — full operator runbook: host layout,
  first-time BRouter provisioning, SOPS rotation (including the
  macOS `SOPS_AGE_KEY_FILE` gotcha), cutover procedure with
  rollback, manual workflow triggers.

Task 8.4 (infrastructure/README.md) skipped: that file doesn't
exist and the ground is covered by brouter-host/README.md +
docs/deployment.md.

## Validation

- `docker compose config` on both the flagship and brouter-host
  compose files — both validate.
- `pnpm typecheck`, `pnpm lint`, `pnpm test` — all clean (full
  turbo cache hits; no code changes in this commit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-04-23 23:02:35 +02:00
parent 34b2e44ed1
commit 9e598fb6a1
No known key found for this signature in database
GPG key ID: A32FF691A0F752D9
12 changed files with 390 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

26
openspec/changes/relocate-brouter-to-dedicated-host/tasks.md
View file

@ -65,11 +65,17 @@
## 6. Observability
- [ ] 6.1 Add a Prometheus scrape job in `infrastructure/prometheus/prometheus.yml` targeting the BRouter host's cAdvisor (or JMX exporter) on the vSwitch IP; label with `host="brouter"`
- [ ] 6.2 Run cAdvisor on the dedicated host as part of `infrastructure/brouter-host/docker-compose.yml`, configured to report only BRouter-labeled containers
- [ ] 6.3 Add a Promtail (or Alloy) service to `infrastructure/brouter-host/docker-compose.yml` tailing Docker logs for BRouter + Caddy sidecar only, pushing to the flagship Loki over vSwitch
- [ ] 6.4 Add a Grafana dashboard row (or new dashboard) for BRouter host: request rate, p50/p95/p99, JVM heap, container memory, scrape up/down
- [ ] 6.5 Add an alert: `up{job="brouter"} == 0 for 2m`
- [x] 6.1 Add a Prometheus scrape job in `infrastructure/prometheus/prometheus.yml` targeting the BRouter host's cAdvisor (or JMX exporter) on the vSwitch IP; label with `host="brouter"`
- Job `brouter-cadvisor` → `10.0.1.10:8080`. Uses static_configs with a static `host="brouter"` label so dashboards can filter.
- [x] 6.2 Run cAdvisor on the dedicated host as part of `infrastructure/brouter-host/docker-compose.yml`, configured to report only BRouter-labeled containers
- `--whitelisted_container_labels=trails.cool.service` + `--docker_only=true` scope metrics to trails containers only. Bound to `10.0.1.10:8080` (vSwitch-only).
- [x] 6.3 Add a Promtail (or Alloy) service to `infrastructure/brouter-host/docker-compose.yml` tailing Docker logs for BRouter + Caddy sidecar only, pushing to the flagship Loki over vSwitch
- Promtail with docker_sd + relabel-drop on missing `trails.cool.service` label; ships to `http://10.0.0.2:3100/loki/api/v1/push`. Also published Loki on flagship's vSwitch IP so the dedicated host can reach it.
- Requires operator one-time: `ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp` (documented in brouter-host/README.md).
- [x] 6.4 Add a Grafana dashboard row (or new dashboard) for BRouter host: request rate, p50/p95/p99, JVM heap, container memory, scrape up/down
- New `infrastructure/grafana/dashboards/brouter.json` with scrape up/down, request rate + latency (from Planner-side metrics), container memory/CPU, and a Loki logs panel filtered to `host="brouter"`.
- [x] 6.5 Add an alert: `up{job="brouter"} == 0 for 2m`
- Added as `brouter-scrape-down` in `infrastructure/grafana/provisioning/alerting/alerts.yml`. NoData state set to Alerting so a complete scrape outage still fires.
## 7. Cutover
@ -82,10 +88,14 @@
## 8. Documentation
- [ ] 8.1 Update `CLAUDE.md` to mention the second deployment target and the `trails`-user deploy pattern for BRouter
- [ ] 8.2 Update `docs/architecture.md` with the new topology and vSwitch boundary
- [ ] 8.3 Update `docs/deployment.md` (or create) with the BRouter host runbook: first-time provisioning, segment updates, token rotation, rollback
- [x] 8.1 Update `CLAUDE.md` to mention the second deployment target and the `trails`-user deploy pattern for BRouter
- Deployment table now lists SSH target per workflow; new Hosts section explains the flagship + dedicated split and the vSwitch bridge.
- [x] 8.2 Update `docs/architecture.md` with the new topology and vSwitch boundary
- Hosting section rewritten to describe both hosts, the vSwitch, and the observability-scoping for the shared dedicated host.
- [x] 8.3 Update `docs/deployment.md` (or create) with the BRouter host runbook: first-time provisioning, segment updates, token rotation, rollback
- New file. Covers host layout, first-time provisioning, SOPS rotation (including the macOS SOPS_AGE_KEY_FILE gotcha), the full cutover procedure with rollback, and `gh workflow run cd-brouter.yml`.
- [ ] 8.4 Add a note to `infrastructure/README.md` (if present) distinguishing flagship-host vs. BRouter-host compose projects
- No `infrastructure/README.md` currently exists; the `infrastructure/brouter-host/README.md` added in 3.5 + the updated `docs/deployment.md` cover the ground. Skip.
## 9. Verification