BRouter host compose + Planner auth + cd-brouter rewrite
Lands sections 3-5 of the relocate-brouter-to-dedicated-host change:
everything needed to run BRouter on the dedicated Hetzner Robot host
and have the Planner talk to it with the shared-secret header. Does
NOT flip the cutover — the flagship BRouter stays warm during soak.
## BRouter host compose (section 3)
New `infrastructure/brouter-host/` — a standalone compose project that
runs as the `trails` user on `ullrich.is`:
- `docker-compose.yml` — brouter + caddy sidecar. BRouter has no host
port; caddy binds only to `10.0.1.10:17777` (vSwitch IP). Every
service explicitly overrides the host's default Loki logging driver
to `json-file` so logs don't leak to the operator's personal Loki.
- `Caddyfile` — single-purpose reverse proxy that requires
`X-BRouter-Auth: ${BROUTER_AUTH_TOKEN}` on every request. `auto_https
off` (vSwitch-only); default access log format omits request
headers, so the token is never written to disk.
- `download-segments.sh` — crawls brouter.de, pulls planet-wide RD5
tiles via `wget -N` (incremental). Idempotent, safe to cron.
- `README.md` — one-shot provisioning + token rotation + rollback
notes.
`docker/brouter/Dockerfile` is patched to honor `JAVA_OPTS` (was
hardcoded `-Xmx1024M` in CMD). Default keeps the flagship's current
heap; compose on the dedicated host overrides to `-Xmx8g` for planet
scale on a 32 GB box.
## Planner shared-secret header (section 4)
`apps/planner/app/lib/brouter.ts`:
- Module-level guard: throws at startup in production if
`BROUTER_AUTH_TOKEN` is unset.
- `authHeaders()` helper (reads env at call time, so tests can
`vi.stubEnv` without module reset).
- Header attached on both `computeRoute` (per-segment) and
`computeSegmentGpx`.
3 new unit tests cover header attachment + the no-token path.
`infrastructure/docker-compose.yml` passes `BROUTER_AUTH_TOKEN` to
the Planner service, and makes `BROUTER_URL` overridable via SOPS so
the cutover is a one-variable flip.
## cd-brouter workflow (section 5)
Rewritten to deploy to the dedicated host:
- SSH as `trails@${BROUTER_DEPLOY_HOST}` on port
`${BROUTER_DEPLOY_SSH_PORT}` (2232) using
`${BROUTER_DEPLOY_SSH_KEY}`.
- Decrypts SOPS, extracts ONLY `BROUTER_AUTH_TOKEN` into a `.env`
file, scp'd alongside the compose project.
- `paths:` trigger now includes `infrastructure/brouter-host/**`.
- Segment download is NOT run here — first-time seed is a manual
operator step (multi-hour). Routine re-runs are cron-able on the
dedicated host.
- Grafana annotation step preserved (reaches flagship Grafana as
before).
## What's NOT here
- `brouter:` service on the flagship is intentionally left in place
(removed in section 7.5 after the 48 h soak window post-cutover).
- Observability (section 6) — Prometheus scrape + Loki shipping from
the dedicated host — comes in a follow-up PR.
- Cutover itself (section 7) — flip `BROUTER_URL`, verify, remove the
flagship brouter — is an operator action gated on first-time
provisioning + smoke testing.
## Verification
`pnpm typecheck && pnpm lint && pnpm test` all clean; planner build
passes (the CI regression from #286 was fixed in #290).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
12010b48c2
commit
c49047fd33
10 changed files with 500 additions and 55 deletions
84
.github/workflows/cd-brouter.yml
vendored
84
.github/workflows/cd-brouter.yml
vendored
|
|
@ -5,6 +5,7 @@ on:
|
|||
branches: [main]
|
||||
paths:
|
||||
- "docker/brouter/**"
|
||||
- "infrastructure/brouter-host/**"
|
||||
workflow_dispatch: {}
|
||||
|
||||
concurrency:
|
||||
|
|
@ -36,13 +37,63 @@ jobs:
|
|||
ghcr.io/trails-cool/brouter:${{ github.sha }}
|
||||
|
||||
deploy:
|
||||
name: Deploy BRouter
|
||||
name: Deploy BRouter to dedicated host
|
||||
needs: [build]
|
||||
runs-on: ubuntu-latest
|
||||
environment: infra
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Deploy via SSH
|
||||
- name: Decrypt shared secret
|
||||
run: |
|
||||
curl -sLO https://github.com/getsops/sops/releases/download/v3.9.4/sops-v3.9.4.linux.amd64
|
||||
chmod +x sops-v3.9.4.linux.amd64
|
||||
# Extract ONLY BROUTER_AUTH_TOKEN from secrets.app.env — the
|
||||
# rest of that file is app-only and has no business reaching
|
||||
# the BRouter host.
|
||||
SOPS_AGE_KEY="${{ secrets.AGE_SECRET_KEY }}" ./sops-v3.9.4.linux.amd64 -d infrastructure/secrets.app.env \
|
||||
| grep '^BROUTER_AUTH_TOKEN=' > infrastructure/brouter-host/.env
|
||||
chmod 0600 infrastructure/brouter-host/.env
|
||||
|
||||
- name: Copy compose project to dedicated host
|
||||
uses: appleboy/scp-action@v1
|
||||
with:
|
||||
host: ${{ secrets.BROUTER_DEPLOY_HOST }}
|
||||
username: trails
|
||||
port: ${{ secrets.BROUTER_DEPLOY_SSH_PORT }}
|
||||
key: ${{ secrets.BROUTER_DEPLOY_SSH_KEY }}
|
||||
source: "infrastructure/brouter-host/docker-compose.yml,infrastructure/brouter-host/Caddyfile,infrastructure/brouter-host/download-segments.sh,infrastructure/brouter-host/.env"
|
||||
target: /home/trails/brouter
|
||||
strip_components: 2
|
||||
|
||||
- name: Pull image and restart containers
|
||||
uses: appleboy/ssh-action@v1
|
||||
with:
|
||||
host: ${{ secrets.BROUTER_DEPLOY_HOST }}
|
||||
username: trails
|
||||
port: ${{ secrets.BROUTER_DEPLOY_SSH_PORT }}
|
||||
key: ${{ secrets.BROUTER_DEPLOY_SSH_KEY }}
|
||||
script: |
|
||||
set -euo pipefail
|
||||
cd /home/trails/brouter
|
||||
|
||||
chmod +x download-segments.sh
|
||||
|
||||
# Segment download is explicitly NOT run here — it's
|
||||
# multi-hour and idempotent. Seeding the segments directory
|
||||
# is a one-shot operator task (see brouter-host/README.md).
|
||||
# Run `~/brouter/download-segments.sh` by hand (or via cron)
|
||||
# to refresh.
|
||||
if [ ! -d segments ] || [ -z "$(ls -A segments 2>/dev/null)" ]; then
|
||||
echo "WARNING: segments/ is empty. BRouter will start but return 404 until segments are seeded."
|
||||
mkdir -p segments
|
||||
fi
|
||||
|
||||
docker compose pull
|
||||
docker compose up -d --remove-orphans
|
||||
docker compose ps
|
||||
|
||||
- name: Annotate deploy in flagship Grafana
|
||||
uses: appleboy/ssh-action@v1
|
||||
with:
|
||||
host: ${{ secrets.DEPLOY_HOST }}
|
||||
|
|
@ -50,34 +101,7 @@ jobs:
|
|||
key: ${{ secrets.DEPLOY_SSH_KEY }}
|
||||
script: |
|
||||
cd /opt/trails-cool
|
||||
|
||||
# Download BRouter segments for Europe (W10-E40, N35-N70)
|
||||
mkdir -p /opt/trails-cool/segments
|
||||
EUROPE_TILES="
|
||||
W10_N50 W10_N55 W10_N60 W10_N65
|
||||
W5_N35 W5_N40 W5_N45 W5_N50 W5_N55 W5_N60 W5_N65
|
||||
E0_N35 E0_N40 E0_N45 E0_N50 E0_N55 E0_N60 E0_N65 E0_N70
|
||||
E5_N35 E5_N40 E5_N45 E5_N50 E5_N55 E5_N60 E5_N65 E5_N70
|
||||
E10_N35 E10_N40 E10_N45 E10_N50 E10_N55 E10_N60 E10_N65 E10_N70
|
||||
E15_N35 E15_N40 E15_N45 E15_N50 E15_N55 E15_N60 E15_N65 E15_N70
|
||||
E20_N35 E20_N40 E20_N45 E20_N50 E20_N55 E20_N60 E20_N65 E20_N70
|
||||
E25_N35 E25_N40 E25_N45 E25_N50 E25_N55 E25_N60 E25_N65
|
||||
E30_N35 E30_N40 E30_N45 E30_N50 E30_N55 E30_N60 E30_N65
|
||||
E35_N40 E35_N45 E35_N50 E35_N55 E35_N60
|
||||
E40_N40 E40_N45 E40_N50 E40_N55
|
||||
"
|
||||
for tile in $EUROPE_TILES; do
|
||||
[ -f "/opt/trails-cool/segments/${tile}.rd5" ] || \
|
||||
wget -q "https://brouter.de/brouter/segments4/${tile}.rd5" -O "/opt/trails-cool/segments/${tile}.rd5" 2>/dev/null || \
|
||||
rm -f "/opt/trails-cool/segments/${tile}.rd5"
|
||||
done
|
||||
|
||||
docker pull ghcr.io/trails-cool/brouter:latest
|
||||
docker compose up -d brouter
|
||||
docker image prune -af
|
||||
|
||||
# Annotate deploy in Grafana
|
||||
GRAFANA_TOKEN=$(grep GRAFANA_SERVICE_TOKEN .env | cut -d= -f2- 2>/dev/null)
|
||||
GRAFANA_TOKEN=$(grep GRAFANA_SERVICE_TOKEN .env | cut -d= -f2-)
|
||||
if [ -n "$GRAFANA_TOKEN" ]; then
|
||||
docker compose exec -T grafana curl -sf -X POST \
|
||||
-H "Authorization: Bearer $GRAFANA_TOKEN" \
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, it, expect } from "vitest";
|
||||
import { mergeGeoJsonSegments } from "./brouter";
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
||||
import { computeRoute, computeSegmentGpx, mergeGeoJsonSegments } from "./brouter";
|
||||
|
||||
function makeSegment(coords: number[][], length: number, ascend: number) {
|
||||
return {
|
||||
|
|
@ -221,6 +221,83 @@ describe("highway tag extraction", () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe("X-BRouter-Auth header", () => {
|
||||
// Minimal valid response so computeRoute doesn't throw during merge.
|
||||
const stubGeoJson = JSON.stringify({
|
||||
type: "FeatureCollection",
|
||||
features: [
|
||||
{
|
||||
type: "Feature",
|
||||
properties: { "track-length": "100", "filtered ascend": "0", "total-time": "10" },
|
||||
geometry: { type: "LineString", coordinates: [[13.0, 52.0, 30], [13.1, 52.1, 40]] },
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
let fetchSpy: ReturnType<typeof vi.fn>;
|
||||
|
||||
beforeEach(() => {
|
||||
fetchSpy = vi.fn().mockResolvedValue(
|
||||
new Response(stubGeoJson, { status: 200, headers: { "content-type": "application/json" } }),
|
||||
);
|
||||
vi.stubGlobal("fetch", fetchSpy);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllEnvs();
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
it("attaches X-BRouter-Auth when BROUTER_AUTH_TOKEN is set (computeRoute)", async () => {
|
||||
vi.stubEnv("BROUTER_AUTH_TOKEN", "test-token-abc");
|
||||
|
||||
await computeRoute({
|
||||
waypoints: [
|
||||
{ lat: 52.0, lon: 13.0 },
|
||||
{ lat: 52.1, lon: 13.1 },
|
||||
],
|
||||
});
|
||||
|
||||
expect(fetchSpy).toHaveBeenCalledTimes(1);
|
||||
const [, init] = fetchSpy.mock.calls[0]!;
|
||||
expect(init?.headers).toMatchObject({ "X-BRouter-Auth": "test-token-abc" });
|
||||
});
|
||||
|
||||
it("omits X-BRouter-Auth when BROUTER_AUTH_TOKEN is unset (dev/test convenience)", async () => {
|
||||
vi.stubEnv("BROUTER_AUTH_TOKEN", "");
|
||||
|
||||
await computeRoute({
|
||||
waypoints: [
|
||||
{ lat: 52.0, lon: 13.0 },
|
||||
{ lat: 52.1, lon: 13.1 },
|
||||
],
|
||||
});
|
||||
|
||||
const [, init] = fetchSpy.mock.calls[0]!;
|
||||
expect(init?.headers).not.toHaveProperty("X-BRouter-Auth");
|
||||
});
|
||||
|
||||
it("attaches X-BRouter-Auth on computeSegmentGpx as well", async () => {
|
||||
vi.stubEnv("BROUTER_AUTH_TOKEN", "test-token-xyz");
|
||||
fetchSpy.mockResolvedValueOnce(
|
||||
new Response('<gpx><trk><trkseg><trkpt lat="52" lon="13"/></trkseg></trk></gpx>', {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/gpx+xml" },
|
||||
}),
|
||||
);
|
||||
|
||||
await computeSegmentGpx({
|
||||
waypoints: [
|
||||
{ lat: 52.0, lon: 13.0 },
|
||||
{ lat: 52.1, lon: 13.1 },
|
||||
],
|
||||
});
|
||||
|
||||
const [, init] = fetchSpy.mock.calls[0]!;
|
||||
expect(init?.headers).toMatchObject({ "X-BRouter-Auth": "test-token-xyz" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("waypoint to BRouter segments", () => {
|
||||
it("splits N waypoints into N-1 pairs", () => {
|
||||
const waypoints = [
|
||||
|
|
|
|||
|
|
@ -1,5 +1,23 @@
|
|||
const BROUTER_URL = process.env.BROUTER_URL ?? "http://localhost:17777";
|
||||
|
||||
// The BRouter host's Caddy sidecar requires every request to carry a
|
||||
// shared-secret header. Missing in production = every request 403s, so
|
||||
// crash loudly at startup rather than during the first route request.
|
||||
if (process.env.NODE_ENV === "production" && !process.env.BROUTER_AUTH_TOKEN) {
|
||||
throw new Error(
|
||||
"BROUTER_AUTH_TOKEN is required in production. The BRouter Caddy " +
|
||||
"sidecar rejects unauthenticated requests with 403. Check the " +
|
||||
"SOPS-encrypted infrastructure/secrets.app.env and the cd-apps " +
|
||||
"env wiring.",
|
||||
);
|
||||
}
|
||||
|
||||
// Read at call time so tests can stub via vi.stubEnv without module reset.
|
||||
function authHeaders(): HeadersInit {
|
||||
const token = process.env.BROUTER_AUTH_TOKEN;
|
||||
return token ? { "X-BRouter-Auth": token } : {};
|
||||
}
|
||||
|
||||
export interface NoGoArea {
|
||||
points: Array<{ lat: number; lon: number }>;
|
||||
}
|
||||
|
|
@ -77,7 +95,7 @@ export class BRouterError extends Error {
|
|||
}
|
||||
|
||||
async function fetchSegment(url: string): Promise<Record<string, unknown>> {
|
||||
const response = await fetch(url);
|
||||
const response = await fetch(url, { headers: authHeaders() });
|
||||
if (!response.ok) {
|
||||
const body = await response.text();
|
||||
throw new BRouterError(body.trim(), response.status);
|
||||
|
|
@ -310,7 +328,9 @@ export async function computeSegmentGpx(request: {
|
|||
const nogoParam = request.noGoAreas?.length ? noGoAreasToParam(request.noGoAreas) : undefined;
|
||||
if (nogoParam) params.set("polygons", nogoParam);
|
||||
|
||||
const resp = await fetch(`${BROUTER_URL}/brouter?${params}`);
|
||||
const resp = await fetch(`${BROUTER_URL}/brouter?${params}`, {
|
||||
headers: authHeaders(),
|
||||
});
|
||||
if (!resp.ok) {
|
||||
const body = await resp.text();
|
||||
throw new BRouterError(body.trim(), resp.status);
|
||||
|
|
|
|||
|
|
@ -34,10 +34,13 @@ RUN for f in /data/profiles/*.brf; do \
|
|||
USER app
|
||||
EXPOSE 17777
|
||||
|
||||
# JAVA_OPTS can be overridden at runtime (e.g., `-Xmx8g` for planet-scale).
|
||||
# The default keeps the flagship's single-instance footprint small.
|
||||
ENV JAVA_OPTS="-Xmx1024M -Xms256M -Xmn64M"
|
||||
|
||||
# BRouter server: <segmentdir> <profiledir> <customprofiledir> <port> <maxthreads>
|
||||
CMD ["java", "-Xmx1024M", "-Xms256M", "-Xmn64M", \
|
||||
"-DmaxRunningTime=300", \
|
||||
"-cp", "brouter.jar", \
|
||||
"btools.server.RouteServer", \
|
||||
"/data/segments", "/data/profiles", "/data/profiles", \
|
||||
"17777", "4"]
|
||||
# Shell form so $JAVA_OPTS expands at container start.
|
||||
CMD java $JAVA_OPTS -DmaxRunningTime=300 -cp brouter.jar \
|
||||
btools.server.RouteServer \
|
||||
/data/segments /data/profiles /data/profiles \
|
||||
17777 4
|
||||
|
|
|
|||
47
infrastructure/brouter-host/Caddyfile
Normal file
47
infrastructure/brouter-host/Caddyfile
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
# BRouter Caddy sidecar.
|
||||
#
|
||||
# Role: enforce the X-BRouter-Auth shared-secret header so the exposed
|
||||
# vSwitch port can only be used by the Planner, not by any other process
|
||||
# that happens to land on the private network.
|
||||
#
|
||||
# Note on auth hygiene: by default, Caddy's access log does NOT include
|
||||
# request headers, so the token value is not written to disk. If you
|
||||
# ever enable `log { format json }` with `fields_exclude`, make sure
|
||||
# `X-BRouter-Auth` is NOT added to the log's request headers set.
|
||||
|
||||
{
|
||||
# Bind admin socket to localhost only (default behavior, explicit here
|
||||
# as defense-in-depth).
|
||||
admin localhost:2019
|
||||
# Don't attempt TLS — we only listen on the private vSwitch and
|
||||
# terminate plain HTTP on :17777. No certs, no Let's Encrypt, no
|
||||
# leakage of this hostname to the ACME CAs.
|
||||
auto_https off
|
||||
}
|
||||
|
||||
:17777 {
|
||||
# Match requests carrying the correct shared secret.
|
||||
@authed header X-BRouter-Auth {$BROUTER_AUTH_TOKEN}
|
||||
|
||||
handle @authed {
|
||||
reverse_proxy brouter:17777 {
|
||||
# Don't forward the auth header upstream — BRouter doesn't
|
||||
# use it, and it's cleaner to contain the credential at
|
||||
# the proxy boundary.
|
||||
header_up -X-BRouter-Auth
|
||||
}
|
||||
}
|
||||
|
||||
# Everything else: blunt 403.
|
||||
handle {
|
||||
respond "Forbidden" 403
|
||||
}
|
||||
|
||||
# Access log to stdout (structured JSON). Container logs are captured
|
||||
# by Docker's json-file driver and shipped to Loki by the promtail
|
||||
# sidecar. Header values are NOT logged unless explicitly configured.
|
||||
log {
|
||||
output stdout
|
||||
format json
|
||||
}
|
||||
}
|
||||
94
infrastructure/brouter-host/README.md
Normal file
94
infrastructure/brouter-host/README.md
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
# BRouter host compose project
|
||||
|
||||
Runs on a dedicated Hetzner Robot server (currently `ullrich.is`,
|
||||
private IP `10.0.1.10` over vSwitch #80672), owned by the non-root
|
||||
`trails` user. Services:
|
||||
|
||||
- **brouter** — the BRouter Java server, planet-scale segments, 8 GB
|
||||
JVM heap, no public port.
|
||||
- **caddy** — thin sidecar enforcing the `X-BRouter-Auth` shared-secret
|
||||
header. Bound to `10.0.1.10:17777` (vSwitch IP only).
|
||||
|
||||
Public ingress is blocked at the host's UFW (port 17777 is only allowed
|
||||
on the VLAN interface from `10.0.0.2`, the flagship's vSwitch IP).
|
||||
|
||||
## One-time provisioning
|
||||
|
||||
Runs as the `trails` user on the dedicated host.
|
||||
|
||||
```bash
|
||||
# 1. Land the compose project
|
||||
cd ~
|
||||
git clone https://github.com/trails-cool/trails.git repo
|
||||
mkdir -p brouter
|
||||
cp -r repo/infrastructure/brouter-host/* brouter/
|
||||
cd brouter
|
||||
|
||||
# 2. Provide the shared secret (matches BROUTER_AUTH_TOKEN in SOPS)
|
||||
# The CD workflow normally writes this file; for manual bring-up,
|
||||
# do it yourself.
|
||||
cat > .env <<'EOF'
|
||||
BROUTER_AUTH_TOKEN=<paste value from sops -d infrastructure/secrets.app.env | grep BROUTER_AUTH_TOKEN>
|
||||
EOF
|
||||
chmod 0600 .env
|
||||
|
||||
# 3. Seed segments (multi-hour, ~60–80 GB)
|
||||
./download-segments.sh
|
||||
|
||||
# 4. Start services
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
|
||||
# 5. Smoke test from the flagship (over vSwitch)
|
||||
# Should return 200 with the token, 403 without.
|
||||
# ssh root@trails.cool 'curl -sSf -H "X-BRouter-Auth: <TOKEN>" http://10.0.1.10:17777/brouter?lonlats=... '
|
||||
```
|
||||
|
||||
## Subsequent deploys
|
||||
|
||||
The `cd-brouter` GitHub Actions workflow handles routine updates:
|
||||
it pulls the latest image, rewrites the compose file + Caddyfile from
|
||||
the repo, and restarts.
|
||||
|
||||
## Segment updates
|
||||
|
||||
Segments are refreshed by brouter.de weekly. To pull updates:
|
||||
|
||||
```bash
|
||||
./download-segments.sh
|
||||
docker compose restart brouter
|
||||
```
|
||||
|
||||
Schedule via cron if you want automatic updates (not wired in this repo
|
||||
yet).
|
||||
|
||||
## Token rotation
|
||||
|
||||
1. Regenerate: `openssl rand -base64 32`.
|
||||
2. Update SOPS: `sops infrastructure/secrets.app.env` (writer uses the
|
||||
`sops -d | append | sops -e` pattern via the CD workflow; editing
|
||||
directly works too).
|
||||
3. Merge the SOPS change to `main`.
|
||||
4. `cd-apps` redeploys the Planner (sends the new token outbound).
|
||||
5. `cd-brouter` redeploys Caddy (matches on the new token).
|
||||
6. Brief overlap window where Planner sends new token but Caddy still
|
||||
accepts old: both deploys should fire within a minute of each other,
|
||||
so a few 403s are the worst case.
|
||||
|
||||
## Rollback
|
||||
|
||||
If BRouter is misbehaving and the flagship BRouter is still warm
|
||||
(during the 48 h soak window post-cutover), flip `BROUTER_URL` in
|
||||
`infrastructure/secrets.app.env` back to `http://brouter:17777` and
|
||||
redeploy the Planner. After the soak window, see the change's
|
||||
design.md for the longer rollback path.
|
||||
|
||||
## Logging
|
||||
|
||||
The dedicated host's Docker daemon default logging driver is `loki`
|
||||
(the operator's personal Loki). Our compose file explicitly overrides
|
||||
each service to `json-file` so logs stay local; a `promtail` sidecar
|
||||
(section 6.3 of the relocate change) tails them and ships to
|
||||
trails.cool's Loki over the vSwitch. If you disable that sidecar, the
|
||||
BRouter logs will NOT flow to trails.cool's Grafana — they'll just
|
||||
accumulate locally and eventually rotate.
|
||||
78
infrastructure/brouter-host/docker-compose.yml
Normal file
78
infrastructure/brouter-host/docker-compose.yml
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
# BRouter host compose project — runs on the dedicated Hetzner Robot
|
||||
# server `ullrich.is` under the `trails` user. See README.md for first-time
|
||||
# provisioning notes.
|
||||
#
|
||||
# Exposed surface: Caddy listens on 10.0.1.10:17777 (vSwitch IP only).
|
||||
# BRouter itself is not published to the host — only reachable via the
|
||||
# internal Docker network from the Caddy sidecar.
|
||||
#
|
||||
# Logging: the host's default logging driver is `loki` (user's personal
|
||||
# Loki). Every service here explicitly overrides to `json-file` so logs
|
||||
# stay local and are picked up by the promtail sidecar (section 6.3) for
|
||||
# shipping to trails.cool's Loki.
|
||||
|
||||
services:
|
||||
brouter:
|
||||
image: ghcr.io/trails-cool/brouter:latest
|
||||
container_name: trails-brouter
|
||||
restart: unless-stopped
|
||||
# Planet-scale coverage: segments live on the host and are mounted in.
|
||||
# 8 GB heap for segment cache; -Xms generous because routing is
|
||||
# memory-heavy and we don't benefit from a slow JVM warmup.
|
||||
environment:
|
||||
JAVA_OPTS: "-Xmx8g -Xms512M"
|
||||
volumes:
|
||||
- ./segments:/data/segments:ro
|
||||
networks:
|
||||
- trails-brouter-internal
|
||||
# Scope logs to json-file so we don't leak to the host's default Loki
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
labels:
|
||||
trails.cool.service: "brouter"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -q -O- http://localhost:17777/ >/dev/null 2>&1 || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
|
||||
caddy:
|
||||
image: caddy:2-alpine
|
||||
container_name: trails-brouter-caddy
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
brouter:
|
||||
condition: service_healthy
|
||||
# Bind ONLY to the vSwitch IP on the host — the dedicated host's
|
||||
# public IP remains unaffected. UFW further restricts this to traffic
|
||||
# sourced from the flagship's private IP (10.0.0.2).
|
||||
ports:
|
||||
- "10.0.1.10:17777:17777"
|
||||
environment:
|
||||
BROUTER_AUTH_TOKEN: ${BROUTER_AUTH_TOKEN:?BROUTER_AUTH_TOKEN must be set}
|
||||
volumes:
|
||||
- ./Caddyfile:/etc/caddy/Caddyfile:ro
|
||||
- caddy-data:/data
|
||||
- caddy-config:/config
|
||||
networks:
|
||||
- trails-brouter-internal
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
labels:
|
||||
trails.cool.service: "brouter-caddy"
|
||||
|
||||
volumes:
|
||||
caddy-data:
|
||||
caddy-config:
|
||||
|
||||
networks:
|
||||
trails-brouter-internal:
|
||||
driver: bridge
|
||||
# Container-to-container only; no host-level exposure via this net
|
||||
80
infrastructure/brouter-host/download-segments.sh
Executable file
80
infrastructure/brouter-host/download-segments.sh
Executable file
|
|
@ -0,0 +1,80 @@
|
|||
#!/bin/bash
|
||||
# Download / refresh BRouter RD5 segments from brouter.de for planet-wide
|
||||
# coverage. Idempotent: re-running only fetches files that are new or
|
||||
# updated upstream (wget -N uses Last-Modified). First run takes hours
|
||||
# and pulls ~60–80 GB; subsequent runs are cheap.
|
||||
#
|
||||
# Usage:
|
||||
# ./download-segments.sh [dest_dir]
|
||||
# dest_dir defaults to ./segments relative to this script
|
||||
#
|
||||
# Runs safely as non-root; no privileged operations. Can be cron'd.
|
||||
#
|
||||
# After a successful run, restart the brouter container so it reloads
|
||||
# any updated segments:
|
||||
# docker compose restart brouter
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
DEST_DIR="${1:-$SCRIPT_DIR/segments}"
|
||||
BASE_URL="https://brouter.de/brouter/segments4"
|
||||
|
||||
mkdir -p "$DEST_DIR"
|
||||
|
||||
echo "Listing tiles at $BASE_URL/ ..."
|
||||
# Extract RD5 filenames from the Apache-style directory listing.
|
||||
# Pattern matches the standard brouter tile naming: W120_N40.rd5 etc.
|
||||
tiles=$(curl --fail --silent --show-error --location "$BASE_URL/" \
|
||||
| grep -oE '[WE][0-9]+_[NS][0-9]+\.rd5' \
|
||||
| sort -u)
|
||||
|
||||
if [ -z "$tiles" ]; then
|
||||
echo "ERROR: no tiles found at $BASE_URL/ (directory listing empty or blocked)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
total=$(printf '%s\n' "$tiles" | wc -l | tr -d ' ')
|
||||
echo "Found $total tiles. Destination: $DEST_DIR"
|
||||
echo
|
||||
|
||||
cd "$DEST_DIR"
|
||||
|
||||
i=0
|
||||
skipped=0
|
||||
failed=0
|
||||
while read -r tile; do
|
||||
[ -z "$tile" ] && continue
|
||||
i=$((i + 1))
|
||||
# -N: only fetch if remote is newer than local (Last-Modified)
|
||||
# -q: quiet; we print our own progress
|
||||
if wget --no-verbose --timestamping --tries=3 --timeout=60 "$BASE_URL/$tile" 2>&1 | grep -q 'not retrieving'; then
|
||||
skipped=$((skipped + 1))
|
||||
fi
|
||||
if [ ! -s "$tile" ]; then
|
||||
echo " [$i/$total] FAILED: $tile"
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
# Print heartbeat every 25 tiles so hour-long runs don't look hung
|
||||
if [ $((i % 25)) -eq 0 ]; then
|
||||
echo " [$i/$total] ... $skipped already-current, $failed failed so far"
|
||||
fi
|
||||
done <<< "$tiles"
|
||||
|
||||
echo
|
||||
echo "Done. Totals:"
|
||||
echo " attempted: $i"
|
||||
echo " already current: $skipped"
|
||||
echo " failed: $failed"
|
||||
echo
|
||||
echo "Destination size:"
|
||||
du -sh "$DEST_DIR"
|
||||
echo
|
||||
echo "Tile count on disk:"
|
||||
ls "$DEST_DIR"/*.rd5 2>/dev/null | wc -l
|
||||
|
||||
if [ "$failed" -gt 0 ]; then
|
||||
echo
|
||||
echo "WARNING: $failed downloads failed. Re-run the script to retry." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
|
@ -54,7 +54,16 @@ services:
|
|||
image: ghcr.io/trails-cool/planner:latest
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
BROUTER_URL: http://brouter:17777
|
||||
# BROUTER_URL overridable via SOPS: during the cutover to the
|
||||
# dedicated BRouter host, flip to `http://10.0.1.10:17777` without
|
||||
# touching this compose file. Default keeps the in-tree BRouter
|
||||
# for the soak window and local dev.
|
||||
BROUTER_URL: ${BROUTER_URL:-http://brouter:17777}
|
||||
# Shared secret the Planner attaches as X-BRouter-Auth on every
|
||||
# BRouter request. The Caddy sidecar on the dedicated BRouter
|
||||
# host enforces this header; for the in-tree flagship BRouter
|
||||
# it's an unused extra header. Set in SOPS secrets.app.env.
|
||||
BROUTER_AUTH_TOKEN: ${BROUTER_AUTH_TOKEN}
|
||||
# Ordered failover list for the Overpass proxy. The code defaults
|
||||
# to the same pair if unset; declaring it here makes the prod
|
||||
# upstream explicit and easy to reshuffle via env when a
|
||||
|
|
|
|||
|
|
@ -24,30 +24,43 @@
|
|||
- [ ] ~~2.2 Add `BROUTER_AUTH_TOKEN` to `infrastructure/secrets.infra.env` (SOPS)~~ **obsoleted** — after relocation, `cd-infra` no longer deploys BRouter and therefore doesn't need the token. `cd-brouter` reads it from `secrets.app.env` instead (see 5.1). Single source of truth.
|
||||
- [x] 2.3 Add `BROUTER_AUTH_TOKEN` to `infrastructure/secrets.app.env` (SOPS) for the Planner
|
||||
- Token added via `sops -d | append | sops -e`; round-trip decrypt confirms. Committed in this branch.
|
||||
- [ ] 2.4 Add GitHub Actions secrets: `BROUTER_DEPLOY_HOST`, `BROUTER_DEPLOY_SSH_KEY`, `BROUTER_DEPLOY_SSH_PORT`
|
||||
- [x] 2.4 Add GitHub Actions secrets: `BROUTER_DEPLOY_HOST`, `BROUTER_DEPLOY_SSH_KEY`, `BROUTER_DEPLOY_SSH_PORT`
|
||||
- Set via `gh secret set` from operator laptop: `BROUTER_DEPLOY_HOST=ullrich.is`, `BROUTER_DEPLOY_SSH_PORT=2232`, `BROUTER_DEPLOY_SSH_KEY` from `~/.ssh/trails-brouter-deploy`.
|
||||
- [ ] 2.5 Document the rotation runbook in `docs/deployment.md` (or equivalent)
|
||||
|
||||
## 3. BRouter host compose project
|
||||
|
||||
- [ ] 3.1 Create `infrastructure/brouter-host/docker-compose.yml` with services `brouter` (bound only to the internal Docker network) and `caddy` (published on the vSwitch IP, auth-enforcing)
|
||||
- [ ] 3.2 Create `infrastructure/brouter-host/Caddyfile` that requires `X-BRouter-Auth` equal to the configured token and forwards matching requests to `brouter:17777`; redact the header from access logs
|
||||
- [ ] 3.3 Set `JAVA_OPTS=-Xmx8g` (or equivalent BRouter env) on the `brouter` service
|
||||
- [ ] 3.4 Create `infrastructure/brouter-host/download-segments.sh` that fetches the planet RD5 tile list idempotently into `./segments/`
|
||||
- [ ] 3.5 Add a README in `infrastructure/brouter-host/` with one-shot provisioning notes (`git clone`, first segment download, first compose up)
|
||||
- [x] 3.1 Create `infrastructure/brouter-host/docker-compose.yml` with services `brouter` (bound only to the internal Docker network) and `caddy` (published on the vSwitch IP, auth-enforcing)
|
||||
- Compose has explicit `logging: driver: json-file` on each service to bypass the dedicated host's default `loki` logging driver. Caddy binds to `10.0.1.10:17777`; brouter has no published port.
|
||||
- [x] 3.2 Create `infrastructure/brouter-host/Caddyfile` that requires `X-BRouter-Auth` equal to the configured token and forwards matching requests to `brouter:17777`; redact the header from access logs
|
||||
- Header matcher + 403 fallback; `auto_https off` since vSwitch-only. Caddy default access log format does not include request headers, so token is not logged.
|
||||
- [x] 3.3 Set `JAVA_OPTS=-Xmx8g` (or equivalent BRouter env) on the `brouter` service
|
||||
- Also patched `docker/brouter/Dockerfile` to honor `JAVA_OPTS` (was hardcoded `-Xmx1024M` in CMD). Default env keeps flagship behavior unchanged.
|
||||
- [x] 3.4 Create `infrastructure/brouter-host/download-segments.sh` that fetches the planet RD5 tile list idempotently into `./segments/`
|
||||
- Crawls brouter.de directory listing, uses `wget -N` for Last-Modified-based incremental updates, prints heartbeat every 25 tiles.
|
||||
- [x] 3.5 Add a README in `infrastructure/brouter-host/` with one-shot provisioning notes (`git clone`, first segment download, first compose up)
|
||||
- Covers bring-up, segment refresh, token rotation, rollback. Paired with the CD workflow which handles routine updates.
|
||||
|
||||
## 4. Planner changes
|
||||
|
||||
- [ ] 4.1 Add `BROUTER_AUTH_TOKEN` env var to `apps/planner/app/lib/brouter.ts`; send `X-BRouter-Auth` on every fetch
|
||||
- [ ] 4.2 Fail the Planner startup with a clear error when `NODE_ENV=production` and `BROUTER_AUTH_TOKEN` is unset
|
||||
- [ ] 4.3 Update `infrastructure/docker-compose.yml` Planner service env to pass `BROUTER_AUTH_TOKEN` through from the SOPS env file
|
||||
- [ ] 4.4 Add a unit test covering the header-attachment path in `apps/planner/app/lib/brouter.ts`
|
||||
- [x] 4.1 Add `BROUTER_AUTH_TOKEN` env var to `apps/planner/app/lib/brouter.ts`; send `X-BRouter-Auth` on every fetch
|
||||
- `authHeaders()` helper reads env at call time (testable); attached to both `computeRoute` and `computeSegmentGpx` fetch sites.
|
||||
- [x] 4.2 Fail the Planner startup with a clear error when `NODE_ENV=production` and `BROUTER_AUTH_TOKEN` is unset
|
||||
- Module-level throw at import. Prod container fails fast; dev/test unaffected.
|
||||
- [x] 4.3 Update `infrastructure/docker-compose.yml` Planner service env to pass `BROUTER_AUTH_TOKEN` through from the SOPS env file
|
||||
- Also made `BROUTER_URL` overridable so cutover is a single SOPS edit away.
|
||||
- [x] 4.4 Add a unit test covering the header-attachment path in `apps/planner/app/lib/brouter.ts`
|
||||
- 3 new tests: token set → header attached, token unset → header omitted, covers both `computeRoute` and `computeSegmentGpx`.
|
||||
|
||||
## 5. CD workflow
|
||||
|
||||
- [ ] 5.1 Rewrite `.github/workflows/cd-brouter.yml` deploy job: SSH as `trails@${{ secrets.BROUTER_DEPLOY_HOST }}`, `cd ~trails/brouter`, `docker compose pull && docker compose up -d`
|
||||
- [ ] 5.2 Update workflow `paths:` trigger to include `infrastructure/brouter-host/**`
|
||||
- [ ] 5.3 Move the segment-download logic out of the workflow into the on-host `download-segments.sh`; workflow calls it but tolerates a long-running invocation (or skips on subsequent deploys if segments already present)
|
||||
- [ ] 5.4 Keep the Grafana annotation step, pointing at the flagship Grafana over its existing path
|
||||
- [x] 5.1 Rewrite `.github/workflows/cd-brouter.yml` deploy job: SSH as `trails@${{ secrets.BROUTER_DEPLOY_HOST }}`, `cd ~trails/brouter`, `docker compose pull && docker compose up -d`
|
||||
- SSH on port `BROUTER_DEPLOY_SSH_PORT` (2232), dedicated key `BROUTER_DEPLOY_SSH_KEY`.
|
||||
- [x] 5.2 Update workflow `paths:` trigger to include `infrastructure/brouter-host/**`
|
||||
- [x] 5.3 Move the segment-download logic out of the workflow into the on-host `download-segments.sh`; workflow calls it but tolerates a long-running invocation (or skips on subsequent deploys if segments already present)
|
||||
- Workflow does NOT call `download-segments.sh` — first-time seed is a manual operator step (per README and task 7.1); routine re-runs are cron-able on the dedicated host.
|
||||
- [x] 5.4 Keep the Grafana annotation step, pointing at the flagship Grafana over its existing path
|
||||
- Still uses `DEPLOY_HOST` + `DEPLOY_SSH_KEY` to reach the flagship for the annotation.
|
||||
- [ ] 5.5 Remove the `brouter:` service from `infrastructure/docker-compose.yml` on the flagship (deferred to cutover step 7.5)
|
||||
|
||||
## 6. Observability
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue