Ullrich Schäfer 9e598fb6a1

Lands sections 6 and 8 of relocate-brouter-to-dedicated-host on top
of the host compose in #291.

## Observability (section 6)

- **Prometheus**: new `brouter-cadvisor` job scraping
  `10.0.1.10:8080` over the vSwitch, labeled `host="brouter"`.
- **cAdvisor sidecar** on the dedicated host's compose
  (`--docker_only --whitelisted_container_labels=trails.cool.service`)
  so metrics only cover trails containers, never the operator's
  unrelated workloads on the shared box.
- **Promtail sidecar** on the dedicated host, Docker SD with relabel-
  drop on missing `trails.cool.service` label, pushing to flagship
  Loki at `http://10.0.0.2:3100/loki/api/v1/push`.
- **Flagship compose**: Loki now publishes port 3100 on the vSwitch
  IP only (10.0.0.2:3100) — Hetzner Cloud firewall blocks it from
  the public internet.
- **Grafana dashboard**: `brouter.json` — scrape up/down, request
  rate (from Planner-side `brouter_request_duration_seconds`),
  p50/p95/p99, container memory/CPU, Loki logs panel.
- **Alert**: `brouter-scrape-down` fires on
  `up{job="brouter-cadvisor"} < 1 for 2m`; `noDataState: Alerting`
  so a total scrape failure still pages.

Operator needs one UFW rule on the dedicated host for the cAdvisor
port — documented in `infrastructure/brouter-host/README.md`.

## Documentation (section 8)

- `CLAUDE.md` — hosts table + updated deployment table with SSH
  targets per workflow; BRouter host SSH is `-p 2232 trails@...`,
  different key.
- `docs/architecture.md` — Hosting section rewritten to cover both
  hosts, vSwitch boundary, and the observability-scoping rationale
  for the shared dedicated host.
- `docs/deployment.md` (new) — full operator runbook: host layout,
  first-time BRouter provisioning, SOPS rotation (including the
  macOS `SOPS_AGE_KEY_FILE` gotcha), cutover procedure with
  rollback, manual workflow triggers.

Task 8.4 (infrastructure/README.md) skipped: that file doesn't
exist and the ground is covered by brouter-host/README.md +
docs/deployment.md.

## Validation

- `docker compose config` on both the flagship and brouter-host
  compose files — both validate.
- `pnpm typecheck`, `pnpm lint`, `pnpm test` — all clean (full
  turbo cache hits; no code changes in this commit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-23 23:10:45 +02:00

3.7 KiB

Raw Blame History

BRouter host compose project

Runs on a dedicated Hetzner Robot server (currently ullrich.is, private IP 10.0.1.10 over vSwitch #80672), owned by the non-root trails user. Services:

brouter — the BRouter Java server, planet-scale segments, 8 GB JVM heap, no public port.
caddy — thin sidecar enforcing the X-BRouter-Auth shared-secret header. Bound to 10.0.1.10:17777 (vSwitch IP only).

Public ingress is blocked at the host's UFW (port 17777 is only allowed on the VLAN interface from 10.0.0.2, the flagship's vSwitch IP).

One-time provisioning

Operator (as root) — one-time firewall rules

The dedicated host's UFW policy rejects anything not explicitly allowed. Open the vSwitch ports the flagship needs:

# BRouter Caddy sidecar (already added during section 1.2):
# ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 17777 proto tcp \
#   comment 'trails brouter via flagship vSwitch'

# cAdvisor metrics endpoint (section 6) — add if not already:
ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp \
  comment 'trails brouter cadvisor via flagship vSwitch'

Application bring-up (as the `trails` user)

# 1. Land the compose project
cd ~
git clone https://github.com/trails-cool/trails.git repo
mkdir -p brouter
cp -r repo/infrastructure/brouter-host/* brouter/
cd brouter

# 2. Provide the shared secret (matches BROUTER_AUTH_TOKEN in SOPS)
#    The CD workflow normally writes this file; for manual bring-up,
#    do it yourself.
cat > .env <<'EOF'
BROUTER_AUTH_TOKEN=<paste value from sops -d infrastructure/secrets.app.env | grep BROUTER_AUTH_TOKEN>
EOF
chmod 0600 .env

# 3. Seed segments (multi-hour, ~60–80 GB)
./download-segments.sh

# 4. Start services
docker compose pull
docker compose up -d

# 5. Smoke test from the flagship (over vSwitch)
#    Should return 200 with the token, 403 without.
# ssh root@trails.cool 'curl -sSf -H "X-BRouter-Auth: <TOKEN>" http://10.0.1.10:17777/brouter?lonlats=... '

Subsequent deploys

The cd-brouter GitHub Actions workflow handles routine updates: it pulls the latest image, rewrites the compose file + Caddyfile from the repo, and restarts.

Segment updates

Segments are refreshed by brouter.de weekly. To pull updates:

./download-segments.sh
docker compose restart brouter

Schedule via cron if you want automatic updates (not wired in this repo yet).

Token rotation

Regenerate: openssl rand -base64 32.
Update SOPS: sops infrastructure/secrets.app.env (writer uses the sops -d | append | sops -e pattern via the CD workflow; editing directly works too).
Merge the SOPS change to main.
cd-apps redeploys the Planner (sends the new token outbound).
cd-brouter redeploys Caddy (matches on the new token).
Brief overlap window where Planner sends new token but Caddy still accepts old: both deploys should fire within a minute of each other, so a few 403s are the worst case.

Rollback

If BRouter is misbehaving and the flagship BRouter is still warm (during the 48 h soak window post-cutover), flip BROUTER_URL in infrastructure/secrets.app.env back to http://brouter:17777 and redeploy the Planner. After the soak window, see the change's design.md for the longer rollback path.

Logging

The dedicated host's Docker daemon default logging driver is loki (the operator's personal Loki). Our compose file explicitly overrides each service to json-file so logs stay local; a promtail sidecar (section 6.3 of the relocate change) tails them and ships to trails.cool's Loki over the vSwitch. If you disable that sidecar, the BRouter logs will NOT flow to trails.cool's Grafana — they'll just accumulate locally and eventually rotate.

3.7 KiB Raw Blame History Unescape Escape