9e598fb6a1
Lands sections 6 and 8 of relocate-brouter-to-dedicated-host on top of the host compose in #291. ## Observability (section 6) - **Prometheus**: new `brouter-cadvisor` job scraping `10.0.1.10:8080` over the vSwitch, labeled `host="brouter"`. - **cAdvisor sidecar** on the dedicated host's compose (`--docker_only --whitelisted_container_labels=trails.cool.service`) so metrics only cover trails containers, never the operator's unrelated workloads on the shared box. - **Promtail sidecar** on the dedicated host, Docker SD with relabel- drop on missing `trails.cool.service` label, pushing to flagship Loki at `http://10.0.0.2:3100/loki/api/v1/push`. - **Flagship compose**: Loki now publishes port 3100 on the vSwitch IP only (10.0.0.2:3100) — Hetzner Cloud firewall blocks it from the public internet. - **Grafana dashboard**: `brouter.json` — scrape up/down, request rate (from Planner-side `brouter_request_duration_seconds`), p50/p95/p99, container memory/CPU, Loki logs panel. - **Alert**: `brouter-scrape-down` fires on `up{job="brouter-cadvisor"} < 1 for 2m`; `noDataState: Alerting` so a total scrape failure still pages. Operator needs one UFW rule on the dedicated host for the cAdvisor port — documented in `infrastructure/brouter-host/README.md`. ## Documentation (section 8) - `CLAUDE.md` — hosts table + updated deployment table with SSH targets per workflow; BRouter host SSH is `-p 2232 trails@...`, different key. - `docs/architecture.md` — Hosting section rewritten to cover both hosts, vSwitch boundary, and the observability-scoping rationale for the shared dedicated host. - `docs/deployment.md` (new) — full operator runbook: host layout, first-time BRouter provisioning, SOPS rotation (including the macOS `SOPS_AGE_KEY_FILE` gotcha), cutover procedure with rollback, manual workflow triggers. Task 8.4 (infrastructure/README.md) skipped: that file doesn't exist and the ground is covered by brouter-host/README.md + docs/deployment.md. ## Validation - `docker compose config` on both the flagship and brouter-host compose files — both validate. - `pnpm typecheck`, `pnpm lint`, `pnpm test` — all clean (full turbo cache hits; no code changes in this commit). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
109 lines
3.7 KiB
Markdown
109 lines
3.7 KiB
Markdown
# BRouter host compose project
|
||
|
||
Runs on a dedicated Hetzner Robot server (currently `ullrich.is`,
|
||
private IP `10.0.1.10` over vSwitch #80672), owned by the non-root
|
||
`trails` user. Services:
|
||
|
||
- **brouter** — the BRouter Java server, planet-scale segments, 8 GB
|
||
JVM heap, no public port.
|
||
- **caddy** — thin sidecar enforcing the `X-BRouter-Auth` shared-secret
|
||
header. Bound to `10.0.1.10:17777` (vSwitch IP only).
|
||
|
||
Public ingress is blocked at the host's UFW (port 17777 is only allowed
|
||
on the VLAN interface from `10.0.0.2`, the flagship's vSwitch IP).
|
||
|
||
## One-time provisioning
|
||
|
||
### Operator (as root) — one-time firewall rules
|
||
|
||
The dedicated host's UFW policy rejects anything not explicitly
|
||
allowed. Open the vSwitch ports the flagship needs:
|
||
|
||
```bash
|
||
# BRouter Caddy sidecar (already added during section 1.2):
|
||
# ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 17777 proto tcp \
|
||
# comment 'trails brouter via flagship vSwitch'
|
||
|
||
# cAdvisor metrics endpoint (section 6) — add if not already:
|
||
ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp \
|
||
comment 'trails brouter cadvisor via flagship vSwitch'
|
||
```
|
||
|
||
### Application bring-up (as the `trails` user)
|
||
|
||
```bash
|
||
# 1. Land the compose project
|
||
cd ~
|
||
git clone https://github.com/trails-cool/trails.git repo
|
||
mkdir -p brouter
|
||
cp -r repo/infrastructure/brouter-host/* brouter/
|
||
cd brouter
|
||
|
||
# 2. Provide the shared secret (matches BROUTER_AUTH_TOKEN in SOPS)
|
||
# The CD workflow normally writes this file; for manual bring-up,
|
||
# do it yourself.
|
||
cat > .env <<'EOF'
|
||
BROUTER_AUTH_TOKEN=<paste value from sops -d infrastructure/secrets.app.env | grep BROUTER_AUTH_TOKEN>
|
||
EOF
|
||
chmod 0600 .env
|
||
|
||
# 3. Seed segments (multi-hour, ~60–80 GB)
|
||
./download-segments.sh
|
||
|
||
# 4. Start services
|
||
docker compose pull
|
||
docker compose up -d
|
||
|
||
# 5. Smoke test from the flagship (over vSwitch)
|
||
# Should return 200 with the token, 403 without.
|
||
# ssh root@trails.cool 'curl -sSf -H "X-BRouter-Auth: <TOKEN>" http://10.0.1.10:17777/brouter?lonlats=... '
|
||
```
|
||
|
||
## Subsequent deploys
|
||
|
||
The `cd-brouter` GitHub Actions workflow handles routine updates:
|
||
it pulls the latest image, rewrites the compose file + Caddyfile from
|
||
the repo, and restarts.
|
||
|
||
## Segment updates
|
||
|
||
Segments are refreshed by brouter.de weekly. To pull updates:
|
||
|
||
```bash
|
||
./download-segments.sh
|
||
docker compose restart brouter
|
||
```
|
||
|
||
Schedule via cron if you want automatic updates (not wired in this repo
|
||
yet).
|
||
|
||
## Token rotation
|
||
|
||
1. Regenerate: `openssl rand -base64 32`.
|
||
2. Update SOPS: `sops infrastructure/secrets.app.env` (writer uses the
|
||
`sops -d | append | sops -e` pattern via the CD workflow; editing
|
||
directly works too).
|
||
3. Merge the SOPS change to `main`.
|
||
4. `cd-apps` redeploys the Planner (sends the new token outbound).
|
||
5. `cd-brouter` redeploys Caddy (matches on the new token).
|
||
6. Brief overlap window where Planner sends new token but Caddy still
|
||
accepts old: both deploys should fire within a minute of each other,
|
||
so a few 403s are the worst case.
|
||
|
||
## Rollback
|
||
|
||
If BRouter is misbehaving and the flagship BRouter is still warm
|
||
(during the 48 h soak window post-cutover), flip `BROUTER_URL` in
|
||
`infrastructure/secrets.app.env` back to `http://brouter:17777` and
|
||
redeploy the Planner. After the soak window, see the change's
|
||
design.md for the longer rollback path.
|
||
|
||
## Logging
|
||
|
||
The dedicated host's Docker daemon default logging driver is `loki`
|
||
(the operator's personal Loki). Our compose file explicitly overrides
|
||
each service to `json-file` so logs stay local; a `promtail` sidecar
|
||
(section 6.3 of the relocate change) tails them and ships to
|
||
trails.cool's Loki over the vSwitch. If you disable that sidecar, the
|
||
BRouter logs will NOT flow to trails.cool's Grafana — they'll just
|
||
accumulate locally and eventually rotate.
|