trails/openspec/changes/relocate-brouter-to-dedicated-host/tasks.md

1. Pre-flight (operator)

• 1.1 Confirm Hetzner vSwitch is provisioned between the flagship cx23 and the dedicated host; record both vSwitch IPs
  • vSwitch #80672 (VLAN 4000, 1 TB), flagship 10.0.0.2 on enp7s0, dedicated 10.0.1.10 on enp4s0.4000 (MTU 1400), Cloud Network 10.0.0.0/16 with cloud subnet 10.0.0.0/24 + vSwitch subnet 10.0.1.0/24 (Terraform)
  • Dedicated host VLAN persistence: /etc/netplan/60-trails-vswitch.yaml on Ubuntu 22.04; verified surviving a reboot (VLAN reachable 64 s after reboot, 0% packet loss steady-state)
  • Flagship side auto-configured by Hetzner cloud-init on attach (no manual OS config)
• 1.2 Confirm the dedicated host's firewall allows inbound TCP from the flagship vSwitch IP to the planned BRouter service port, and blocks that port from the public interface
  • UFW active with INPUT default-DROP. Allow rule: 17777/tcp on enp4s0.4000 from 10.0.0.2. Public interface is blocked by default-deny (17777 not in the public allowlist: 2232/SSH, 80, 443).
• 1.3 Confirm the trails user exists on the dedicated host, is in the docker group, and can run docker ps without sudo
  • uid 1002, groups include 114(docker); sudo -iu trails docker ps returns container list (host runs many other self-hosted services, all user-owned)
• 1.4 Generate an SSH keypair for the deploy workflow (BROUTER_DEPLOY_SSH_KEY) and install the public key in ~trails/.ssh/authorized_keys
  • ed25519 keypair at ~/.ssh/trails-brouter-deploy{,.pub} on operator laptop; public key installed at /home/trails/.ssh/authorized_keys on ullrich.is (perms 0600/0700, owner trails:trails)
  • Verified: ssh -i ~/.ssh/trails-brouter-deploy -p 2232 trails@ullrich.is 'docker ps' succeeds with no password prompt
  • Note: SSH on ullrich.is listens on port 2232, not 22. The CD workflow must use -p 2232. Add BROUTER_DEPLOY_SSH_PORT=2232 as a GitHub Actions secret (or hard-code it in the workflow step).
  • Before merging: add the contents of ~/.ssh/trails-brouter-deploy as GitHub Actions secret BROUTER_DEPLOY_SSH_KEY; then consider whether to keep or delete the local copy.
• 1.5 Verify the dedicated host's Docker daemon version supports the compose file features used in infrastructure/docker-compose.yml
  • Docker Engine 29.1.5 (API 1.52), Compose v5.0.1, storage driver overlay2 — all modern
  • Gotcha: default LoggingDriver=loki on this host (routes to user's personal Loki). Our BRouter compose must override logging: per service so our logs go to trails.cool's Loki (flagship) instead. See 6.3 for the two viable approaches: (a) direct Loki driver with loki-url: http://10.0.0.2:3100/loki/api/v1/push, or (b) json-file + our own Promtail sidecar.

2. Secrets and config

• 2.1 Generate a 32-byte random BROUTER_AUTH_TOKEN
  • openssl rand -base64 32; active token lives only in SOPS and the operator's clipboard history. Rotate by regenerating and re-running 2.3.
• 2.2 Add BROUTER_AUTH_TOKEN to infrastructure/secrets.infra.env (SOPS) obsoleted — after relocation, cd-infra no longer deploys BRouter and therefore doesn't need the token. cd-brouter reads it from secrets.app.env instead (see 5.1). Single source of truth.
• 2.3 Add BROUTER_AUTH_TOKEN to infrastructure/secrets.app.env (SOPS) for the Planner
  • Token added via sops -d | append | sops -e; round-trip decrypt confirms. Committed in this branch.
• 2.4 Add GitHub Actions secrets: BROUTER_DEPLOY_HOST, BROUTER_DEPLOY_SSH_KEY, BROUTER_DEPLOY_SSH_PORT
  • Set via gh secret set from operator laptop: BROUTER_DEPLOY_HOST=ullrich.is, BROUTER_DEPLOY_SSH_PORT=2232, BROUTER_DEPLOY_SSH_KEY from ~/.ssh/trails-brouter-deploy.
• 2.5 Document the rotation runbook in docs/deployment.md (or equivalent)

3. BRouter host compose project

• 3.1 Create infrastructure/brouter-host/docker-compose.yml with services brouter (bound only to the internal Docker network) and caddy (published on the vSwitch IP, auth-enforcing)
  • Compose has explicit logging: driver: json-file on each service to bypass the dedicated host's default loki logging driver. Caddy binds to 10.0.1.10:17777; brouter has no published port.
• 3.2 Create infrastructure/brouter-host/Caddyfile that requires X-BRouter-Auth equal to the configured token and forwards matching requests to brouter:17777; redact the header from access logs
  • Header matcher + 403 fallback; auto_https off since vSwitch-only. Caddy default access log format does not include request headers, so token is not logged.
• 3.3 Set JAVA_OPTS=-Xmx8g (or equivalent BRouter env) on the brouter service
  • Also patched docker/brouter/Dockerfile to honor JAVA_OPTS (was hardcoded -Xmx1024M in CMD). Default env keeps flagship behavior unchanged.
• 3.4 Create infrastructure/brouter-host/download-segments.sh that fetches the planet RD5 tile list idempotently into ./segments/
  • Crawls brouter.de directory listing, uses wget -N for Last-Modified-based incremental updates, prints heartbeat every 25 tiles.
• 3.5 Add a README in infrastructure/brouter-host/ with one-shot provisioning notes (git clone, first segment download, first compose up)
  • Covers bring-up, segment refresh, token rotation, rollback. Paired with the CD workflow which handles routine updates.

4. Planner changes

• 4.1 Add BROUTER_AUTH_TOKEN env var to apps/planner/app/lib/brouter.ts; send X-BRouter-Auth on every fetch
  • authHeaders() helper reads env at call time (testable); attached to both computeRoute and computeSegmentGpx fetch sites.
• 4.2 Fail the Planner startup with a clear error when NODE_ENV=production and BROUTER_AUTH_TOKEN is unset
  • Module-level throw at import. Prod container fails fast; dev/test unaffected.
• 4.3 Update infrastructure/docker-compose.yml Planner service env to pass BROUTER_AUTH_TOKEN through from the SOPS env file
  • Also made BROUTER_URL overridable so cutover is a single SOPS edit away.
• 4.4 Add a unit test covering the header-attachment path in apps/planner/app/lib/brouter.ts
  • 3 new tests: token set → header attached, token unset → header omitted, covers both computeRoute and computeSegmentGpx.

5. CD workflow

• 5.1 Rewrite .github/workflows/cd-brouter.yml deploy job: SSH as trails@${{ secrets.BROUTER_DEPLOY_HOST }}, cd ~trails/brouter, docker compose pull && docker compose up -d
  • SSH on port BROUTER_DEPLOY_SSH_PORT (2232), dedicated key BROUTER_DEPLOY_SSH_KEY.
• 5.2 Update workflow paths: trigger to include infrastructure/brouter-host/**
• 5.3 Move the segment-download logic out of the workflow into the on-host download-segments.sh; workflow calls it but tolerates a long-running invocation (or skips on subsequent deploys if segments already present)
  • Workflow does NOT call download-segments.sh — first-time seed is a manual operator step (per README and task 7.1); routine re-runs are cron-able on the dedicated host.
• 5.4 Keep the Grafana annotation step, pointing at the flagship Grafana over its existing path
  • Still uses DEPLOY_HOST + DEPLOY_SSH_KEY to reach the flagship for the annotation.
• 5.5 Remove the brouter: service from infrastructure/docker-compose.yml on the flagship (deferred to cutover step 7.5)

6. Observability

• 6.1 Add a Prometheus scrape job in infrastructure/prometheus/prometheus.yml targeting the BRouter host's cAdvisor (or JMX exporter) on the vSwitch IP; label with host="brouter"
  • Job brouter-cadvisor → 10.0.1.10:8080. Uses static_configs with a static host="brouter" label so dashboards can filter.
• 6.2 Run cAdvisor on the dedicated host as part of infrastructure/brouter-host/docker-compose.yml, configured to report only BRouter-labeled containers
  • --whitelisted_container_labels=trails.cool.service + --docker_only=true scope metrics to trails containers only. Bound to 10.0.1.10:8080 (vSwitch-only).
• 6.3 Add a Promtail (or Alloy) service to infrastructure/brouter-host/docker-compose.yml tailing Docker logs for BRouter + Caddy sidecar only, pushing to the flagship Loki over vSwitch
  • Promtail with docker_sd + relabel-drop on missing trails.cool.service label; ships to http://10.0.0.2:3100/loki/api/v1/push. Also published Loki on flagship's vSwitch IP so the dedicated host can reach it.
  • Requires operator one-time: ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp (documented in brouter-host/README.md).
• 6.4 Add a Grafana dashboard row (or new dashboard) for BRouter host: request rate, p50/p95/p99, JVM heap, container memory, scrape up/down
  • New infrastructure/grafana/dashboards/brouter.json with scrape up/down, request rate + latency (from Planner-side metrics), container memory/CPU, and a Loki logs panel filtered to host="brouter".
• 6.5 Add an alert: up{job="brouter"} == 0 for 2m
  • Added as brouter-scrape-down in infrastructure/grafana/provisioning/alerting/alerts.yml. NoData state set to Alerting so a complete scrape outage still fires.

7. Cutover

• 7.1 Deploy infrastructure/brouter-host/ to the dedicated host manually the first time; run download-segments.sh (expect multi-hour runtime)
• 7.2 Verify the new BRouter responds to a curl from the flagship host over vSwitch with the auth header, and returns 403 without it
• 7.3 Deploy the Planner with BROUTER_AUTH_TOKEN set but BROUTER_URL still pointing at the flagship BRouter (no-op change; validates wiring)
• 7.4 Flip BROUTER_URL in SOPS to the new vSwitch URL; deploy Planner; monitor brouter_request_duration_seconds error rate for 30 minutes
• 7.5 After 48 hours of clean metrics: remove the brouter service + ./segments volume from infrastructure/docker-compose.yml; run cd-infra.yml to restart without BRouter; docker image prune on the flagship
• 7.6 Document rollback path (revert BROUTER_URL flip, redeploy Planner, old container warm for 48h) in the PR description

8. Documentation

• 8.1 Update CLAUDE.md to mention the second deployment target and the trails-user deploy pattern for BRouter
  • Deployment table now lists SSH target per workflow; new Hosts section explains the flagship + dedicated split and the vSwitch bridge.
• 8.2 Update docs/architecture.md with the new topology and vSwitch boundary
  • Hosting section rewritten to describe both hosts, the vSwitch, and the observability-scoping for the shared dedicated host.
• 8.3 Update docs/deployment.md (or create) with the BRouter host runbook: first-time provisioning, segment updates, token rotation, rollback
  • New file. Covers host layout, first-time provisioning, SOPS rotation (including the macOS SOPS_AGE_KEY_FILE gotcha), the full cutover procedure with rollback, and gh workflow run cd-brouter.yml.
• 8.4 Add a note to infrastructure/README.md (if present) distinguishing flagship-host vs. BRouter-host compose projects
  • No infrastructure/README.md currently exists; the infrastructure/brouter-host/README.md added in 3.5 + the updated docs/deployment.md cover the ground. Skip.

9. Verification

• 9.1 pnpm typecheck && pnpm lint && pnpm test pass with the new env handling and unit test
• 9.2 pnpm test:e2e passes with the Planner hitting the relocated BRouter (or a mocked upstream that enforces the auth header)
• 9.3 A manual smoke test from Grafana confirms BRouter metrics and logs appear under the brouter host label after cutover
• 9.4 openspec archive relocate-brouter-to-dedicated-host runs cleanly after cutover + documentation are merged