BRouter host compose + Planner auth + cd-brouter rewrite
Lands sections 3-5 of the relocate-brouter-to-dedicated-host change:
everything needed to run BRouter on the dedicated Hetzner Robot host
and have the Planner talk to it with the shared-secret header. Does
NOT flip the cutover — the flagship BRouter stays warm during soak.
## BRouter host compose (section 3)
New `infrastructure/brouter-host/` — a standalone compose project that
runs as the `trails` user on `ullrich.is`:
- `docker-compose.yml` — brouter + caddy sidecar. BRouter has no host
port; caddy binds only to `10.0.1.10:17777` (vSwitch IP). Every
service explicitly overrides the host's default Loki logging driver
to `json-file` so logs don't leak to the operator's personal Loki.
- `Caddyfile` — single-purpose reverse proxy that requires
`X-BRouter-Auth: ${BROUTER_AUTH_TOKEN}` on every request. `auto_https
off` (vSwitch-only); default access log format omits request
headers, so the token is never written to disk.
- `download-segments.sh` — crawls brouter.de, pulls planet-wide RD5
tiles via `wget -N` (incremental). Idempotent, safe to cron.
- `README.md` — one-shot provisioning + token rotation + rollback
notes.
`docker/brouter/Dockerfile` is patched to honor `JAVA_OPTS` (was
hardcoded `-Xmx1024M` in CMD). Default keeps the flagship's current
heap; compose on the dedicated host overrides to `-Xmx8g` for planet
scale on a 32 GB box.
## Planner shared-secret header (section 4)
`apps/planner/app/lib/brouter.ts`:
- Module-level guard: throws at startup in production if
`BROUTER_AUTH_TOKEN` is unset.
- `authHeaders()` helper (reads env at call time, so tests can
`vi.stubEnv` without module reset).
- Header attached on both `computeRoute` (per-segment) and
`computeSegmentGpx`.
3 new unit tests cover header attachment + the no-token path.
`infrastructure/docker-compose.yml` passes `BROUTER_AUTH_TOKEN` to
the Planner service, and makes `BROUTER_URL` overridable via SOPS so
the cutover is a one-variable flip.
## cd-brouter workflow (section 5)
Rewritten to deploy to the dedicated host:
- SSH as `trails@${BROUTER_DEPLOY_HOST}` on port
`${BROUTER_DEPLOY_SSH_PORT}` (2232) using
`${BROUTER_DEPLOY_SSH_KEY}`.
- Decrypts SOPS, extracts ONLY `BROUTER_AUTH_TOKEN` into a `.env`
file, scp'd alongside the compose project.
- `paths:` trigger now includes `infrastructure/brouter-host/**`.
- Segment download is NOT run here — first-time seed is a manual
operator step (multi-hour). Routine re-runs are cron-able on the
dedicated host.
- Grafana annotation step preserved (reaches flagship Grafana as
before).
## What's NOT here
- `brouter:` service on the flagship is intentionally left in place
(removed in section 7.5 after the 48 h soak window post-cutover).
- Observability (section 6) — Prometheus scrape + Loki shipping from
the dedicated host — comes in a follow-up PR.
- Cutover itself (section 7) — flip `BROUTER_URL`, verify, remove the
flagship brouter — is an operator action gated on first-time
provisioning + smoke testing.
## Verification
`pnpm typecheck && pnpm lint && pnpm test` all clean; planner build
passes (the CI regression from #286 was fixed in #290).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
12010b48c2
commit
c49047fd33
10 changed files with 500 additions and 55 deletions
47
infrastructure/brouter-host/Caddyfile
Normal file
47
infrastructure/brouter-host/Caddyfile
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
# BRouter Caddy sidecar.
|
||||
#
|
||||
# Role: enforce the X-BRouter-Auth shared-secret header so the exposed
|
||||
# vSwitch port can only be used by the Planner, not by any other process
|
||||
# that happens to land on the private network.
|
||||
#
|
||||
# Note on auth hygiene: by default, Caddy's access log does NOT include
|
||||
# request headers, so the token value is not written to disk. If you
|
||||
# ever enable `log { format json }` with `fields_exclude`, make sure
|
||||
# `X-BRouter-Auth` is NOT added to the log's request headers set.
|
||||
|
||||
{
|
||||
# Bind admin socket to localhost only (default behavior, explicit here
|
||||
# as defense-in-depth).
|
||||
admin localhost:2019
|
||||
# Don't attempt TLS — we only listen on the private vSwitch and
|
||||
# terminate plain HTTP on :17777. No certs, no Let's Encrypt, no
|
||||
# leakage of this hostname to the ACME CAs.
|
||||
auto_https off
|
||||
}
|
||||
|
||||
:17777 {
|
||||
# Match requests carrying the correct shared secret.
|
||||
@authed header X-BRouter-Auth {$BROUTER_AUTH_TOKEN}
|
||||
|
||||
handle @authed {
|
||||
reverse_proxy brouter:17777 {
|
||||
# Don't forward the auth header upstream — BRouter doesn't
|
||||
# use it, and it's cleaner to contain the credential at
|
||||
# the proxy boundary.
|
||||
header_up -X-BRouter-Auth
|
||||
}
|
||||
}
|
||||
|
||||
# Everything else: blunt 403.
|
||||
handle {
|
||||
respond "Forbidden" 403
|
||||
}
|
||||
|
||||
# Access log to stdout (structured JSON). Container logs are captured
|
||||
# by Docker's json-file driver and shipped to Loki by the promtail
|
||||
# sidecar. Header values are NOT logged unless explicitly configured.
|
||||
log {
|
||||
output stdout
|
||||
format json
|
||||
}
|
||||
}
|
||||
94
infrastructure/brouter-host/README.md
Normal file
94
infrastructure/brouter-host/README.md
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
# BRouter host compose project
|
||||
|
||||
Runs on a dedicated Hetzner Robot server (currently `ullrich.is`,
|
||||
private IP `10.0.1.10` over vSwitch #80672), owned by the non-root
|
||||
`trails` user. Services:
|
||||
|
||||
- **brouter** — the BRouter Java server, planet-scale segments, 8 GB
|
||||
JVM heap, no public port.
|
||||
- **caddy** — thin sidecar enforcing the `X-BRouter-Auth` shared-secret
|
||||
header. Bound to `10.0.1.10:17777` (vSwitch IP only).
|
||||
|
||||
Public ingress is blocked at the host's UFW (port 17777 is only allowed
|
||||
on the VLAN interface from `10.0.0.2`, the flagship's vSwitch IP).
|
||||
|
||||
## One-time provisioning
|
||||
|
||||
Runs as the `trails` user on the dedicated host.
|
||||
|
||||
```bash
|
||||
# 1. Land the compose project
|
||||
cd ~
|
||||
git clone https://github.com/trails-cool/trails.git repo
|
||||
mkdir -p brouter
|
||||
cp -r repo/infrastructure/brouter-host/* brouter/
|
||||
cd brouter
|
||||
|
||||
# 2. Provide the shared secret (matches BROUTER_AUTH_TOKEN in SOPS)
|
||||
# The CD workflow normally writes this file; for manual bring-up,
|
||||
# do it yourself.
|
||||
cat > .env <<'EOF'
|
||||
BROUTER_AUTH_TOKEN=<paste value from sops -d infrastructure/secrets.app.env | grep BROUTER_AUTH_TOKEN>
|
||||
EOF
|
||||
chmod 0600 .env
|
||||
|
||||
# 3. Seed segments (multi-hour, ~60–80 GB)
|
||||
./download-segments.sh
|
||||
|
||||
# 4. Start services
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
|
||||
# 5. Smoke test from the flagship (over vSwitch)
|
||||
# Should return 200 with the token, 403 without.
|
||||
# ssh root@trails.cool 'curl -sSf -H "X-BRouter-Auth: <TOKEN>" http://10.0.1.10:17777/brouter?lonlats=... '
|
||||
```
|
||||
|
||||
## Subsequent deploys
|
||||
|
||||
The `cd-brouter` GitHub Actions workflow handles routine updates:
|
||||
it pulls the latest image, rewrites the compose file + Caddyfile from
|
||||
the repo, and restarts.
|
||||
|
||||
## Segment updates
|
||||
|
||||
Segments are refreshed by brouter.de weekly. To pull updates:
|
||||
|
||||
```bash
|
||||
./download-segments.sh
|
||||
docker compose restart brouter
|
||||
```
|
||||
|
||||
Schedule via cron if you want automatic updates (not wired in this repo
|
||||
yet).
|
||||
|
||||
## Token rotation
|
||||
|
||||
1. Regenerate: `openssl rand -base64 32`.
|
||||
2. Update SOPS: `sops infrastructure/secrets.app.env` (writer uses the
|
||||
`sops -d | append | sops -e` pattern via the CD workflow; editing
|
||||
directly works too).
|
||||
3. Merge the SOPS change to `main`.
|
||||
4. `cd-apps` redeploys the Planner (sends the new token outbound).
|
||||
5. `cd-brouter` redeploys Caddy (matches on the new token).
|
||||
6. Brief overlap window where Planner sends new token but Caddy still
|
||||
accepts old: both deploys should fire within a minute of each other,
|
||||
so a few 403s are the worst case.
|
||||
|
||||
## Rollback
|
||||
|
||||
If BRouter is misbehaving and the flagship BRouter is still warm
|
||||
(during the 48 h soak window post-cutover), flip `BROUTER_URL` in
|
||||
`infrastructure/secrets.app.env` back to `http://brouter:17777` and
|
||||
redeploy the Planner. After the soak window, see the change's
|
||||
design.md for the longer rollback path.
|
||||
|
||||
## Logging
|
||||
|
||||
The dedicated host's Docker daemon default logging driver is `loki`
|
||||
(the operator's personal Loki). Our compose file explicitly overrides
|
||||
each service to `json-file` so logs stay local; a `promtail` sidecar
|
||||
(section 6.3 of the relocate change) tails them and ships to
|
||||
trails.cool's Loki over the vSwitch. If you disable that sidecar, the
|
||||
BRouter logs will NOT flow to trails.cool's Grafana — they'll just
|
||||
accumulate locally and eventually rotate.
|
||||
78
infrastructure/brouter-host/docker-compose.yml
Normal file
78
infrastructure/brouter-host/docker-compose.yml
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
# BRouter host compose project — runs on the dedicated Hetzner Robot
|
||||
# server `ullrich.is` under the `trails` user. See README.md for first-time
|
||||
# provisioning notes.
|
||||
#
|
||||
# Exposed surface: Caddy listens on 10.0.1.10:17777 (vSwitch IP only).
|
||||
# BRouter itself is not published to the host — only reachable via the
|
||||
# internal Docker network from the Caddy sidecar.
|
||||
#
|
||||
# Logging: the host's default logging driver is `loki` (user's personal
|
||||
# Loki). Every service here explicitly overrides to `json-file` so logs
|
||||
# stay local and are picked up by the promtail sidecar (section 6.3) for
|
||||
# shipping to trails.cool's Loki.
|
||||
|
||||
services:
|
||||
brouter:
|
||||
image: ghcr.io/trails-cool/brouter:latest
|
||||
container_name: trails-brouter
|
||||
restart: unless-stopped
|
||||
# Planet-scale coverage: segments live on the host and are mounted in.
|
||||
# 8 GB heap for segment cache; -Xms generous because routing is
|
||||
# memory-heavy and we don't benefit from a slow JVM warmup.
|
||||
environment:
|
||||
JAVA_OPTS: "-Xmx8g -Xms512M"
|
||||
volumes:
|
||||
- ./segments:/data/segments:ro
|
||||
networks:
|
||||
- trails-brouter-internal
|
||||
# Scope logs to json-file so we don't leak to the host's default Loki
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
labels:
|
||||
trails.cool.service: "brouter"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -q -O- http://localhost:17777/ >/dev/null 2>&1 || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
|
||||
caddy:
|
||||
image: caddy:2-alpine
|
||||
container_name: trails-brouter-caddy
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
brouter:
|
||||
condition: service_healthy
|
||||
# Bind ONLY to the vSwitch IP on the host — the dedicated host's
|
||||
# public IP remains unaffected. UFW further restricts this to traffic
|
||||
# sourced from the flagship's private IP (10.0.0.2).
|
||||
ports:
|
||||
- "10.0.1.10:17777:17777"
|
||||
environment:
|
||||
BROUTER_AUTH_TOKEN: ${BROUTER_AUTH_TOKEN:?BROUTER_AUTH_TOKEN must be set}
|
||||
volumes:
|
||||
- ./Caddyfile:/etc/caddy/Caddyfile:ro
|
||||
- caddy-data:/data
|
||||
- caddy-config:/config
|
||||
networks:
|
||||
- trails-brouter-internal
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
labels:
|
||||
trails.cool.service: "brouter-caddy"
|
||||
|
||||
volumes:
|
||||
caddy-data:
|
||||
caddy-config:
|
||||
|
||||
networks:
|
||||
trails-brouter-internal:
|
||||
driver: bridge
|
||||
# Container-to-container only; no host-level exposure via this net
|
||||
80
infrastructure/brouter-host/download-segments.sh
Executable file
80
infrastructure/brouter-host/download-segments.sh
Executable file
|
|
@ -0,0 +1,80 @@
|
|||
#!/bin/bash
|
||||
# Download / refresh BRouter RD5 segments from brouter.de for planet-wide
|
||||
# coverage. Idempotent: re-running only fetches files that are new or
|
||||
# updated upstream (wget -N uses Last-Modified). First run takes hours
|
||||
# and pulls ~60–80 GB; subsequent runs are cheap.
|
||||
#
|
||||
# Usage:
|
||||
# ./download-segments.sh [dest_dir]
|
||||
# dest_dir defaults to ./segments relative to this script
|
||||
#
|
||||
# Runs safely as non-root; no privileged operations. Can be cron'd.
|
||||
#
|
||||
# After a successful run, restart the brouter container so it reloads
|
||||
# any updated segments:
|
||||
# docker compose restart brouter
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
DEST_DIR="${1:-$SCRIPT_DIR/segments}"
|
||||
BASE_URL="https://brouter.de/brouter/segments4"
|
||||
|
||||
mkdir -p "$DEST_DIR"
|
||||
|
||||
echo "Listing tiles at $BASE_URL/ ..."
|
||||
# Extract RD5 filenames from the Apache-style directory listing.
|
||||
# Pattern matches the standard brouter tile naming: W120_N40.rd5 etc.
|
||||
tiles=$(curl --fail --silent --show-error --location "$BASE_URL/" \
|
||||
| grep -oE '[WE][0-9]+_[NS][0-9]+\.rd5' \
|
||||
| sort -u)
|
||||
|
||||
if [ -z "$tiles" ]; then
|
||||
echo "ERROR: no tiles found at $BASE_URL/ (directory listing empty or blocked)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
total=$(printf '%s\n' "$tiles" | wc -l | tr -d ' ')
|
||||
echo "Found $total tiles. Destination: $DEST_DIR"
|
||||
echo
|
||||
|
||||
cd "$DEST_DIR"
|
||||
|
||||
i=0
|
||||
skipped=0
|
||||
failed=0
|
||||
while read -r tile; do
|
||||
[ -z "$tile" ] && continue
|
||||
i=$((i + 1))
|
||||
# -N: only fetch if remote is newer than local (Last-Modified)
|
||||
# -q: quiet; we print our own progress
|
||||
if wget --no-verbose --timestamping --tries=3 --timeout=60 "$BASE_URL/$tile" 2>&1 | grep -q 'not retrieving'; then
|
||||
skipped=$((skipped + 1))
|
||||
fi
|
||||
if [ ! -s "$tile" ]; then
|
||||
echo " [$i/$total] FAILED: $tile"
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
# Print heartbeat every 25 tiles so hour-long runs don't look hung
|
||||
if [ $((i % 25)) -eq 0 ]; then
|
||||
echo " [$i/$total] ... $skipped already-current, $failed failed so far"
|
||||
fi
|
||||
done <<< "$tiles"
|
||||
|
||||
echo
|
||||
echo "Done. Totals:"
|
||||
echo " attempted: $i"
|
||||
echo " already current: $skipped"
|
||||
echo " failed: $failed"
|
||||
echo
|
||||
echo "Destination size:"
|
||||
du -sh "$DEST_DIR"
|
||||
echo
|
||||
echo "Tile count on disk:"
|
||||
ls "$DEST_DIR"/*.rd5 2>/dev/null | wc -l
|
||||
|
||||
if [ "$failed" -gt 0 ]; then
|
||||
echo
|
||||
echo "WARNING: $failed downloads failed. Re-run the script to retry." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
|
@ -54,7 +54,16 @@ services:
|
|||
image: ghcr.io/trails-cool/planner:latest
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
BROUTER_URL: http://brouter:17777
|
||||
# BROUTER_URL overridable via SOPS: during the cutover to the
|
||||
# dedicated BRouter host, flip to `http://10.0.1.10:17777` without
|
||||
# touching this compose file. Default keeps the in-tree BRouter
|
||||
# for the soak window and local dev.
|
||||
BROUTER_URL: ${BROUTER_URL:-http://brouter:17777}
|
||||
# Shared secret the Planner attaches as X-BRouter-Auth on every
|
||||
# BRouter request. The Caddy sidecar on the dedicated BRouter
|
||||
# host enforces this header; for the in-tree flagship BRouter
|
||||
# it's an unused extra header. Set in SOPS secrets.app.env.
|
||||
BROUTER_AUTH_TOKEN: ${BROUTER_AUTH_TOKEN}
|
||||
# Ordered failover list for the Overpass proxy. The code defaults
|
||||
# to the same pair if unset; declaring it here makes the prod
|
||||
# upstream explicit and easy to reshuffle via env when a
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue