BRouter host compose + Planner auth + cd-brouter rewrite

Lands sections 3-5 of the relocate-brouter-to-dedicated-host change:
everything needed to run BRouter on the dedicated Hetzner Robot host
and have the Planner talk to it with the shared-secret header. Does
NOT flip the cutover — the flagship BRouter stays warm during soak.

## BRouter host compose (section 3)

New `infrastructure/brouter-host/` — a standalone compose project that
runs as the `trails` user on `ullrich.is`:

- `docker-compose.yml` — brouter + caddy sidecar. BRouter has no host
  port; caddy binds only to `10.0.1.10:17777` (vSwitch IP). Every
  service explicitly overrides the host's default Loki logging driver
  to `json-file` so logs don't leak to the operator's personal Loki.
- `Caddyfile` — single-purpose reverse proxy that requires
  `X-BRouter-Auth: ${BROUTER_AUTH_TOKEN}` on every request. `auto_https
  off` (vSwitch-only); default access log format omits request
  headers, so the token is never written to disk.
- `download-segments.sh` — crawls brouter.de, pulls planet-wide RD5
  tiles via `wget -N` (incremental). Idempotent, safe to cron.
- `README.md` — one-shot provisioning + token rotation + rollback
  notes.

`docker/brouter/Dockerfile` is patched to honor `JAVA_OPTS` (was
hardcoded `-Xmx1024M` in CMD). Default keeps the flagship's current
heap; compose on the dedicated host overrides to `-Xmx8g` for planet
scale on a 32 GB box.

## Planner shared-secret header (section 4)

`apps/planner/app/lib/brouter.ts`:

- Module-level guard: throws at startup in production if
  `BROUTER_AUTH_TOKEN` is unset.
- `authHeaders()` helper (reads env at call time, so tests can
  `vi.stubEnv` without module reset).
- Header attached on both `computeRoute` (per-segment) and
  `computeSegmentGpx`.

3 new unit tests cover header attachment + the no-token path.

`infrastructure/docker-compose.yml` passes `BROUTER_AUTH_TOKEN` to
the Planner service, and makes `BROUTER_URL` overridable via SOPS so
the cutover is a one-variable flip.

## cd-brouter workflow (section 5)

Rewritten to deploy to the dedicated host:

- SSH as `trails@${BROUTER_DEPLOY_HOST}` on port
  `${BROUTER_DEPLOY_SSH_PORT}` (2232) using
  `${BROUTER_DEPLOY_SSH_KEY}`.
- Decrypts SOPS, extracts ONLY `BROUTER_AUTH_TOKEN` into a `.env`
  file, scp'd alongside the compose project.
- `paths:` trigger now includes `infrastructure/brouter-host/**`.
- Segment download is NOT run here — first-time seed is a manual
  operator step (multi-hour). Routine re-runs are cron-able on the
  dedicated host.
- Grafana annotation step preserved (reaches flagship Grafana as
  before).

## What's NOT here

- `brouter:` service on the flagship is intentionally left in place
  (removed in section 7.5 after the 48 h soak window post-cutover).
- Observability (section 6) — Prometheus scrape + Loki shipping from
  the dedicated host — comes in a follow-up PR.
- Cutover itself (section 7) — flip `BROUTER_URL`, verify, remove the
  flagship brouter — is an operator action gated on first-time
  provisioning + smoke testing.

## Verification

`pnpm typecheck && pnpm lint && pnpm test` all clean; planner build
passes (the CI regression from #286 was fixed in #290).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-04-23 22:52:48 +02:00
parent 12010b48c2
commit c49047fd33
No known key found for this signature in database
GPG key ID: A32FF691A0F752D9
10 changed files with 500 additions and 55 deletions

View file

@ -0,0 +1,47 @@
# BRouter Caddy sidecar.
#
# Role: enforce the X-BRouter-Auth shared-secret header so the exposed
# vSwitch port can only be used by the Planner, not by any other process
# that happens to land on the private network.
#
# Note on auth hygiene: by default, Caddy's access log does NOT include
# request headers, so the token value is not written to disk. If you
# ever enable `log { format json }` with `fields_exclude`, make sure
# `X-BRouter-Auth` is NOT added to the log's request headers set.
{
# Bind admin socket to localhost only (default behavior, explicit here
# as defense-in-depth).
admin localhost:2019
# Don't attempt TLS — we only listen on the private vSwitch and
# terminate plain HTTP on :17777. No certs, no Let's Encrypt, no
# leakage of this hostname to the ACME CAs.
auto_https off
}
:17777 {
# Match requests carrying the correct shared secret.
@authed header X-BRouter-Auth {$BROUTER_AUTH_TOKEN}
handle @authed {
reverse_proxy brouter:17777 {
# Don't forward the auth header upstream — BRouter doesn't
# use it, and it's cleaner to contain the credential at
# the proxy boundary.
header_up -X-BRouter-Auth
}
}
# Everything else: blunt 403.
handle {
respond "Forbidden" 403
}
# Access log to stdout (structured JSON). Container logs are captured
# by Docker's json-file driver and shipped to Loki by the promtail
# sidecar. Header values are NOT logged unless explicitly configured.
log {
output stdout
format json
}
}

View file

@ -0,0 +1,94 @@
# BRouter host compose project
Runs on a dedicated Hetzner Robot server (currently `ullrich.is`,
private IP `10.0.1.10` over vSwitch #80672), owned by the non-root
`trails` user. Services:
- **brouter** — the BRouter Java server, planet-scale segments, 8 GB
JVM heap, no public port.
- **caddy** — thin sidecar enforcing the `X-BRouter-Auth` shared-secret
header. Bound to `10.0.1.10:17777` (vSwitch IP only).
Public ingress is blocked at the host's UFW (port 17777 is only allowed
on the VLAN interface from `10.0.0.2`, the flagship's vSwitch IP).
## One-time provisioning
Runs as the `trails` user on the dedicated host.
```bash
# 1. Land the compose project
cd ~
git clone https://github.com/trails-cool/trails.git repo
mkdir -p brouter
cp -r repo/infrastructure/brouter-host/* brouter/
cd brouter
# 2. Provide the shared secret (matches BROUTER_AUTH_TOKEN in SOPS)
# The CD workflow normally writes this file; for manual bring-up,
# do it yourself.
cat > .env <<'EOF'
BROUTER_AUTH_TOKEN=<paste value from sops -d infrastructure/secrets.app.env | grep BROUTER_AUTH_TOKEN>
EOF
chmod 0600 .env
# 3. Seed segments (multi-hour, ~6080 GB)
./download-segments.sh
# 4. Start services
docker compose pull
docker compose up -d
# 5. Smoke test from the flagship (over vSwitch)
# Should return 200 with the token, 403 without.
# ssh root@trails.cool 'curl -sSf -H "X-BRouter-Auth: <TOKEN>" http://10.0.1.10:17777/brouter?lonlats=... '
```
## Subsequent deploys
The `cd-brouter` GitHub Actions workflow handles routine updates:
it pulls the latest image, rewrites the compose file + Caddyfile from
the repo, and restarts.
## Segment updates
Segments are refreshed by brouter.de weekly. To pull updates:
```bash
./download-segments.sh
docker compose restart brouter
```
Schedule via cron if you want automatic updates (not wired in this repo
yet).
## Token rotation
1. Regenerate: `openssl rand -base64 32`.
2. Update SOPS: `sops infrastructure/secrets.app.env` (writer uses the
`sops -d | append | sops -e` pattern via the CD workflow; editing
directly works too).
3. Merge the SOPS change to `main`.
4. `cd-apps` redeploys the Planner (sends the new token outbound).
5. `cd-brouter` redeploys Caddy (matches on the new token).
6. Brief overlap window where Planner sends new token but Caddy still
accepts old: both deploys should fire within a minute of each other,
so a few 403s are the worst case.
## Rollback
If BRouter is misbehaving and the flagship BRouter is still warm
(during the 48 h soak window post-cutover), flip `BROUTER_URL` in
`infrastructure/secrets.app.env` back to `http://brouter:17777` and
redeploy the Planner. After the soak window, see the change's
design.md for the longer rollback path.
## Logging
The dedicated host's Docker daemon default logging driver is `loki`
(the operator's personal Loki). Our compose file explicitly overrides
each service to `json-file` so logs stay local; a `promtail` sidecar
(section 6.3 of the relocate change) tails them and ships to
trails.cool's Loki over the vSwitch. If you disable that sidecar, the
BRouter logs will NOT flow to trails.cool's Grafana — they'll just
accumulate locally and eventually rotate.

View file

@ -0,0 +1,78 @@
# BRouter host compose project — runs on the dedicated Hetzner Robot
# server `ullrich.is` under the `trails` user. See README.md for first-time
# provisioning notes.
#
# Exposed surface: Caddy listens on 10.0.1.10:17777 (vSwitch IP only).
# BRouter itself is not published to the host — only reachable via the
# internal Docker network from the Caddy sidecar.
#
# Logging: the host's default logging driver is `loki` (user's personal
# Loki). Every service here explicitly overrides to `json-file` so logs
# stay local and are picked up by the promtail sidecar (section 6.3) for
# shipping to trails.cool's Loki.
services:
brouter:
image: ghcr.io/trails-cool/brouter:latest
container_name: trails-brouter
restart: unless-stopped
# Planet-scale coverage: segments live on the host and are mounted in.
# 8 GB heap for segment cache; -Xms generous because routing is
# memory-heavy and we don't benefit from a slow JVM warmup.
environment:
JAVA_OPTS: "-Xmx8g -Xms512M"
volumes:
- ./segments:/data/segments:ro
networks:
- trails-brouter-internal
# Scope logs to json-file so we don't leak to the host's default Loki
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
labels:
trails.cool.service: "brouter"
healthcheck:
test: ["CMD-SHELL", "wget -q -O- http://localhost:17777/ >/dev/null 2>&1 || exit 1"]
interval: 30s
timeout: 5s
retries: 3
start_period: 60s
caddy:
image: caddy:2-alpine
container_name: trails-brouter-caddy
restart: unless-stopped
depends_on:
brouter:
condition: service_healthy
# Bind ONLY to the vSwitch IP on the host — the dedicated host's
# public IP remains unaffected. UFW further restricts this to traffic
# sourced from the flagship's private IP (10.0.0.2).
ports:
- "10.0.1.10:17777:17777"
environment:
BROUTER_AUTH_TOKEN: ${BROUTER_AUTH_TOKEN:?BROUTER_AUTH_TOKEN must be set}
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy-data:/data
- caddy-config:/config
networks:
- trails-brouter-internal
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
labels:
trails.cool.service: "brouter-caddy"
volumes:
caddy-data:
caddy-config:
networks:
trails-brouter-internal:
driver: bridge
# Container-to-container only; no host-level exposure via this net

View file

@ -0,0 +1,80 @@
#!/bin/bash
# Download / refresh BRouter RD5 segments from brouter.de for planet-wide
# coverage. Idempotent: re-running only fetches files that are new or
# updated upstream (wget -N uses Last-Modified). First run takes hours
# and pulls ~6080 GB; subsequent runs are cheap.
#
# Usage:
# ./download-segments.sh [dest_dir]
# dest_dir defaults to ./segments relative to this script
#
# Runs safely as non-root; no privileged operations. Can be cron'd.
#
# After a successful run, restart the brouter container so it reloads
# any updated segments:
# docker compose restart brouter
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEST_DIR="${1:-$SCRIPT_DIR/segments}"
BASE_URL="https://brouter.de/brouter/segments4"
mkdir -p "$DEST_DIR"
echo "Listing tiles at $BASE_URL/ ..."
# Extract RD5 filenames from the Apache-style directory listing.
# Pattern matches the standard brouter tile naming: W120_N40.rd5 etc.
tiles=$(curl --fail --silent --show-error --location "$BASE_URL/" \
| grep -oE '[WE][0-9]+_[NS][0-9]+\.rd5' \
| sort -u)
if [ -z "$tiles" ]; then
echo "ERROR: no tiles found at $BASE_URL/ (directory listing empty or blocked)" >&2
exit 1
fi
total=$(printf '%s\n' "$tiles" | wc -l | tr -d ' ')
echo "Found $total tiles. Destination: $DEST_DIR"
echo
cd "$DEST_DIR"
i=0
skipped=0
failed=0
while read -r tile; do
[ -z "$tile" ] && continue
i=$((i + 1))
# -N: only fetch if remote is newer than local (Last-Modified)
# -q: quiet; we print our own progress
if wget --no-verbose --timestamping --tries=3 --timeout=60 "$BASE_URL/$tile" 2>&1 | grep -q 'not retrieving'; then
skipped=$((skipped + 1))
fi
if [ ! -s "$tile" ]; then
echo " [$i/$total] FAILED: $tile"
failed=$((failed + 1))
fi
# Print heartbeat every 25 tiles so hour-long runs don't look hung
if [ $((i % 25)) -eq 0 ]; then
echo " [$i/$total] ... $skipped already-current, $failed failed so far"
fi
done <<< "$tiles"
echo
echo "Done. Totals:"
echo " attempted: $i"
echo " already current: $skipped"
echo " failed: $failed"
echo
echo "Destination size:"
du -sh "$DEST_DIR"
echo
echo "Tile count on disk:"
ls "$DEST_DIR"/*.rd5 2>/dev/null | wc -l
if [ "$failed" -gt 0 ]; then
echo
echo "WARNING: $failed downloads failed. Re-run the script to retry." >&2
exit 2
fi

View file

@ -54,7 +54,16 @@ services:
image: ghcr.io/trails-cool/planner:latest
restart: unless-stopped
environment:
BROUTER_URL: http://brouter:17777
# BROUTER_URL overridable via SOPS: during the cutover to the
# dedicated BRouter host, flip to `http://10.0.1.10:17777` without
# touching this compose file. Default keeps the in-tree BRouter
# for the soak window and local dev.
BROUTER_URL: ${BROUTER_URL:-http://brouter:17777}
# Shared secret the Planner attaches as X-BRouter-Auth on every
# BRouter request. The Caddy sidecar on the dedicated BRouter
# host enforces this header; for the in-tree flagship BRouter
# it's an unused extra header. Set in SOPS secrets.app.env.
BROUTER_AUTH_TOKEN: ${BROUTER_AUTH_TOKEN}
# Ordered failover list for the Overpass proxy. The code defaults
# to the same pair if unset; declaring it here makes the prod
# upstream explicit and easy to reshuffle via env when a