The 2026-06-07 disk-full outage (flagship 100%, postgres crash-looping
on its pidfile, prod + staging down): cd-apps DOES prune images after
deploys, but it had been failing early all day (migration bug), so its
prune never ran — while ~10 staging/preview deploys kept pulling fresh
images with no prune of their own.
- cd-staging: prune superseded layers (until=1h guard against racing
in-flight pulls) after persistent staging and preview deploys.
- disk-maintenance.yml: NEW daily scheduled prune (04:30 UTC) that
also FAILS when the disk is still ≥85% after pruning — a redundant
alert channel for exactly the case where the Grafana disk alert
drowns in other noise, as it did during the incident.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening from two incidents on 2026-06-06/07:
Schema drift (morning): drizzle-kit push exits 0 even when it aborts
on an interactive prompt it can't render in CI, so cd-staging's
set -euo pipefail never fired and a month of staging schema drift
accumulated silently until new code hit missing columns. All three
drizzle push call sites now tee output and fail the deploy on any
'Error:' line.
Production outage (overnight, ~9h): cd-infra and cd-apps had no
failure handling at all. A network-option change stopped postgres for
a network recreation that then deadlocked on containers from other
compose projects holding trails-shared; the script carried on, the
stack stayed down. Both scripts now run set -euo pipefail and gate on
container health at the end (postgres+journal for cd-infra,
journal+planner for cd-apps) so a deploy that leaves the stack down
is a red X, not a shrug.
docs/deployment.md gains the cross-project manual procedure for
network-changing deploys — the CD workflows only manage their own
compose project and cannot apply those safely.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Inbound Mastodon Follows are being rejected 401 (signature
verification) on staging and Fedify's diagnostics were invisible:
it logs through LogTape, which is silent until configured.
- Configure LogTape with a console sink for the 'fedify' category when
the federation instance is built; level via FEDERATION_LOG_LEVEL
(default info).
- Staging deploy sets FEDERATION_LOG_LEVEL=debug for the soak —
signature-verification detail per request; dial down once proven.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
social-federation rollout step 12.2 — inbound soak. Persistent staging
gets FEDERATION_ENABLED=true; PR previews keep the flag empty so
preview journals never emit federation traffic.
- docker-compose.staging.yml: journal accepts FEDERATION_ENABLED +
FEDERATION_KEY_ENCRYPTION_KEY (both default empty).
- cd-staging.yml: persistent staging env sets FEDERATION_ENABLED=true;
the encryption key arrives via the existing SOPS decryption.
- secrets.app.env: new FEDERATION_KEY_ENCRYPTION_KEY (SOPS-encrypted,
generated with openssl rand -hex 32).
Production is unaffected: the flagship compose doesn't pass these vars
and the flag defaults off in code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The five \`*.integration.test.ts\` files in apps/journal (explore, follow,
demo-bot, notifications, notifications-fanout — 31 tests total) were
gated behind \`EXPLORE_INTEGRATION=1\` / \`FOLLOW_INTEGRATION=1\` /
\`DEMO_BOT_INTEGRATION=1\` / \`NOTIFICATIONS_INTEGRATION=1\`. The unit-test
job doesn't have Postgres, so they correctly skipped there — but no
CI job set the env vars, so they were effectively dead code.
Added a step in the E2E job (which already has Postgres + schema
pushed) that flips all four gates and runs the integration files
serially (\`--no-file-parallelism\` — they share the schema and trip FK
constraints if run in parallel; ~2.5s sequential anyway).
Wiring them up surfaced two real issues, both fixed here:
1. **\`createRoute\` silently dropped \`input.visibility\`** — every
caller passing \`visibility: \"public\"\` (including the demo-bot)
was getting the column's \`private\` default. Spread now mirrors
\`createActivity\`'s pattern. Demo-bot routes have actually been
private in production all this time — they were rendering on the
home feed only via the \`activity_published\` fan-out from the
*activity*, not as visible *routes*.
2. **The demo-bot test's fetch stub was incomplete** — it stubbed
\`.text()\` but the planner-session preflight calls \`.json()\`. The
stub now branches on URL: \`/api/sessions\` returns
\`{ sessionId: 'test-session' }\` JSON, the BRouter call returns the
stub GPX text.
Full repo: pnpm typecheck, pnpm lint, pnpm test all green
(177 unit-test pass + 31 integration pass = 208 total, no skips).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Self-hosted instances no longer inherit the trails.cool flagship Sentry
DSNs. Code paths now read DSN strictly from env — unset = no Sentry
init, no events sent.
Flagship continues to report unchanged: cd-apps.yml and cd-staging.yml
inject the public DSNs as workflow env vars, which feed into:
- runtime via \`infrastructure/app.env\` / \`staging.env\`
(\`SENTRY_DSN_JOURNAL\` / \`SENTRY_DSN_PLANNER\` → compose env per service)
- build time via docker build-arg \`VITE_SENTRY_DSN\` (journal only;
planner has no client Sentry init).
Sentry DSNs are public-by-design (transmitted unencrypted from the
client JS bundle), so embedding them as plaintext workflow env vars
is no worse than the runtime exposure. Forks should replace these with
their own DSNs or remove the workflow env lines to ship Sentry-free.
Files changed:
- apps/journal/server.ts: \`process.env.SENTRY_DSN\` only, no fallback
- apps/planner/server.ts: same
- apps/journal/app/lib/sentry.client.ts: \`import.meta.env.VITE_SENTRY_DSN\`
only; falsy = skip init
- apps/journal/Dockerfile: new \`ARG VITE_SENTRY_DSN\` baked into build
- infrastructure/docker-compose.yml: pass \`SENTRY_DSN\` per service
- infrastructure/docker-compose.staging.yml: same
- .github/workflows/cd-apps.yml: workflow env + build-arg + app.env echo
- .github/workflows/cd-staging.yml: same
Full repo: pnpm typecheck, pnpm lint, pnpm test, journal build all green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Playwright runs the server via `react-router serve` with
NODE_ENV=production but against a local dev Postgres and local cookie
secrets. The guards added in 5a7bb76 refused to start under that
configuration. `E2E=true` (already set by the CI E2E job) is now the
explicit opt-out: in real production this env var is never set, so the
guard still bites.
pnpm test:e2e runs via react-router serve which boots with
NODE_ENV=production; the new requireSecret() guard refuses to start
without explicit values. CI now supplies CI-only throwaway secrets so
the guard still bites in real prod deploys without breaking the test
job.
- Add INTEGRATION_SECRET to journal service in docker-compose.yml with
:? guard so a missing value fails loudly at compose-up time
- Add INTEGRATION_SECRET to E2E test step in ci.yml via GitHub secret
(unit tests already set their own value in the test file)
- Archive openspec/changes/komoot-import → archive/2026-05-23-komoot-import
- Sync delta specs: new openspec/specs/komoot-import/spec.md,
updated openspec/specs/route-management/spec.md
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The segment format changed between BRouter 1.7.8 and 1.7.9. The old
cache key 'brouter-segment-E10_N50' served the v1.7.8 segment to the
v1.7.9 binary, causing 'lookup version mismatch (old rd5?)' errors on
every routing request.
Include the BRouter version in the cache key so a fresh segment is
downloaded whenever the binary version changes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The bind mount override (docker-compose.ci.yml) failed because file
permissions on the cached segment prevented the entrypoint from seeing
the file, causing a fresh download on every CI run.
Instead: create the named volume and copy the cached segment into it
before compose starts, using alpine with explicit chmod. The entrypoint
then finds the segment and skips the download, so BRouter starts in ~4s
instead of ~2min.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
BRouter needs time to parse the segment file into memory before it can
serve routing requests. 90s was insufficient; 120s matches the margin
the old explicit startup loop provided.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The routing probe healthcheck was too strict — BRouter is up but may
not have the segment loaded yet, causing all probes to fail. Use a
simple liveness check (curl exit 0 when server responds on any code)
so --wait unblocks as soon as the server is up. Add an explicit
routing readiness wait step in CI that actually confirms a route can
be computed before running E2E tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extend docker-compose.dev.yml with optional monitoring profile
(Prometheus, Grafana, Loki) via --profile monitoring
- Align dev PostgreSQL with production: pg_stat_statements + init scripts
- Add scripts/seed.ts with idempotent Berlin test data (user, route, activity)
- Add pnpm db:seed and pnpm dev:reset scripts
- Simplify CI e2e job: replace manual docker run + BRouter setup with
docker compose up --wait; add db:seed step
- Improve scripts/dev.sh: Docker check, --wait health checks, --monitoring
flag, seed step
- Add scripts/reset-dev.sh to wipe and restart the local stack
- Add .env.development.example with documented local defaults
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When visual tests fail on a PR:
- Writes diff images (base64 data URIs) to the job summary for inline viewing
- Posts a PR comment listing failing tests with a link to the job summary
- Uploads .vitest-attachments/ as an artifact (include-hidden-files: true)
The job now has pull-requests: write permission for posting the comment.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When visual tests fail on a PR, upload each diff PNG to GitHub's CDN
via the issue assets endpoint and post a comment with the images
embedded inline. Also fixes the artifact upload path to point at
.vitest-attachments/ where the actual/diff files live.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Switch checkout/setup-node/setup-pnpm to @v6 to match CI
- Replace node-version-file (.nvmrc doesn't exist) with node-version: 24
- Switch upload-artifact to @v7 to match CI
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Runs Vitest browser visual regression tests (drawElevationChart) on every
PR and push to main. Uses the same Playwright/Chromium cache as the e2e job.
On first run (no committed snapshots yet) the tests create the snapshots and
pass. On subsequent runs they compare against committed snapshots and fail on
visual regression. Failed runs upload a visual-snapshots-diff artifact so the
diff is visible in the Actions UI.
To update snapshots after an intentional visual change:
Actions → "Update visual snapshots" → Run workflow (or add the
`update-snapshots` label to the PR).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces two E2E=true-gated test endpoints:
- POST /api/e2e/seed — creates a test user + bare route, returns routeId + JWT
- GET /api/e2e/route/:id — returns { hasGeom } for post-callback assertions
Three new integration tests:
- valid GPX via callback stores geometry (hasGeom = true)
- invalid GPX (< 2 track points) returns 400, geometry not stored
- missing token returns 401
CI: E2E=true added to the "Run E2E tests" step so the seed endpoints
are enabled when react-router-serve runs during the Playwright job.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three changes:
1. Skip GH-Actions-only Dependabot PRs (dependabot/github_actions/*) in
deploy-preview — there is no app image to preview, and these PRs were
landing phantom containers with mismatched ports.
2. Use per-PR env files (staging-pr-{N}.env) instead of the shared
staging.env for preview deploys and teardowns. Concurrent SCP transfers
to the same filename were overwriting each other, causing wrong
JOURNAL_HOST_PORT / JOURNAL_IMAGE_TAG values to be used.
3. Serialize server-side deploy operations with a flock on
/tmp/trails-preview-deploy.lock (300s timeout). Eviction + compose up
must be atomic; without the lock, two simultaneous jobs could both see
"3 active previews" and both evict different projects, or one could
start compose up against a just-evicted env.
Triggered by a Dependabot batch today (PRs 371-373) that opened
simultaneously and produced a phantom trails-pr-371 container running
the pr-370 image on port 3940 while the Caddyfile expected 3942,
causing sustained 502s on pr-371.staging.trails.cool.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds @fission-ai/openspec ^1.3.1 as a root devDependency so local and CI
run the same lockfile-pinned version, and dependabot can bump it. CI now
runs `pnpm openspec validate --all --strict` instead of npx.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The npm package `openspec` is an unrelated 0.0.0 placeholder; the real
CLI ships as `@fission-ai/openspec`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a fast standalone job that runs `openspec validate --all --strict`
via npx, gating PRs on spec/change well-formedness.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The deploy job was creating a fresh comment on every push, so PRs
accumulated a wall of preview-URL comments. Now we look up the existing
comment by an HTML marker (`<!-- cd-staging:preview -->`) and edit it
in place — both on push and on teardown. Comment also gains a link to
the GitHub Actions run that produced the preview, plus the head SHA.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
So a port bump or compose edit doesn't sit unapplied until the next
apps/ push, and so persistent staging can be redeployed manually
without forcing a no-op apps/ change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Loki binds 10.0.0.2:3100 on the vSwitch interface so the BRouter host can
ship logs in. Persistent staging was trying to publish 0.0.0.0:3100 which
conflicts because Linux refuses 0.0.0.0:<P> when any specific-interface
:<P> is already in use. Move staging journal to 3110, planner to 3111.
PR previews are unaffected — they're already on 3200+2N.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two bugs in the staging-environments rollout:
1. Staging containers published on 127.0.0.1 are unreachable from the
production Caddy container, which connects via the Docker bridge IP
(host.docker.internal:host-gateway resolves to the bridge, not
loopback). Bind to 0.0.0.0 instead — Hetzner Cloud firewall blocks
ports 3000+ from the public internet, so it stays internal-only.
2. The trailing 'docker compose ps' diagnostic in deploy-staging /
deploy-preview was missing --env-file staging.env, so compose
failed env interpolation and the job exited non-zero even when the
actual deploy succeeded.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the staging-environments OpenSpec change. Persistent staging at
staging.trails.cool / planner.staging.trails.cool deploys from main; PR
opens get a journal-only preview at pr-<N>.staging.trails.cool that shares
the persistent planner. cd-staging.yml builds tagged images, manages
per-PR Postgres databases and Caddyfile snippets, evicts the oldest
preview at the cap of 3, and tears everything down on PR close.
staging-cleanup.yml runs weekly to sweep orphaned previews.
DNS records (staging + *.staging A/AAAA) already applied to production
via tofu.
Caddy approach: per-PR Caddyfile snippets imported from /etc/caddy/sites/
and reloaded on each PR event — no wildcard / on-demand TLS, no router
service. Production compose gains a trails-shared network for the staging
project to reach Postgres, and host.docker.internal on Caddy so it can
reverse-proxy staging containers published on the host loopback.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Re-pushing an edited route to Wahoo now updates the existing remote
route via PUT against the stored remote_id instead of POSTing a new
copy. external_id drops the version suffix and identifies the logical
route. sync_pushes is keyed by (user, route, provider) and tracks
last_pushed_version for the "local newer" UI state.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cd-apps already scp's `infrastructure/Caddyfile` to the server
alongside docker-compose.yml — but the running Caddy container
doesn't auto-pick-up config changes. cd-infra is the workflow that
calls `caddy reload`, and it only triggers on `infrastructure/`
paths. So any Caddyfile edit shipped through cd-apps (e.g. the
`lb_try_duration` block in PR #329, deployed as part of an apps
push) sits on disk unapplied until the next cd-infra run.
This was real today: PR #329 added `lb_try_duration 30s` to silence
deploy-time 502s. PR #331 (a vite.config edit, apps path) merged
right after and triggered cd-apps, which copied the new Caddyfile
but didn't reload Caddy. The very next planner restart in that
deploy promptly produced three 502s — the exact thing #329 was
supposed to prevent. We applied the reload manually via SSH after
the fact.
This commit makes that reload happen on every cd-apps deploy. The
operation is idempotent (Caddy validates first, swaps live, no
downtime) so doing it even when Caddyfile is unchanged costs
nothing. `|| true` keeps the deploy from failing if Caddy is
unhealthy at deploy time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The token lives in `secrets.infra.env`, which `cd-infra.yml` merges
together with `secrets.app.env` into the server's
`/opt/trails-cool/.env`. The cd-apps workflow's own `app.env`
intentionally does NOT carry the token (apps don't need it at
runtime), so the original `grep ... .env` was correct. My earlier
edit in this branch swapped the path to `app.env` and would have
broken the annotation hook the moment it actually worked.
Restored `.env` and updated the inline comment to make the file
ownership explicit (cd-infra populates it; cd-apps reads it).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The journal/planner deploy in cd-apps.yml does `docker compose up -d
journal planner`, which stops the old container and starts the new
one — Caddy keeps forwarding requests during the ~10–30s gap and
returns 502s. The caddy-502-rate alert (threshold > 0 for 2m)
correctly trips, every time.
Two production changes plus a long-broken workflow detail:
- infrastructure/Caddyfile — add `lb_try_duration 30s` /
`lb_try_interval 250ms` to the journal and planner reverse_proxy
blocks. Caddy now holds and retries the upstream for up to 30s
during a restart instead of 502'ing immediately. Real outages
(upstream unreachable longer than 30s) still 502 and the alert
still fires for those.
- infrastructure/grafana/provisioning/alerting/alerts.yml — add a
comment documenting why caddy-502-rate stays at threshold > 0:
with lb_try_duration in front of it, the alert no longer
conflates "deploy in flight" with "real outage."
- .github/workflows/cd-apps.yml — fix a long-silent bug: the
Grafana deploy-annotation step was reading
GRAFANA_SERVICE_TOKEN from `.env`, but the secrets file we scp
to /opt/trails-cool is named `app.env`. The token check failed
silently and the curl was being skipped on every deploy.
Switching to `app.env` so deploys actually annotate Grafana.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The old home was an h1, subtitle, and two auth buttons — visitors
arriving at trails.cool had no idea what it was, and self-hosted
instances had nothing interesting to land on.
The new home is one layout that serves both audiences:
- Hero: a product-describing h1 ("Federated outdoor journal") +
tagline. The site name already lives in the top banner and nav brand,
so the h1 carries the pitch instead of triplicating "trails.cool".
- Auth CTAs: Register (blue) + Sign In (outlined) get primary weight.
Planner demotes to a small "Or try the Planner without an account →"
link below them — it's a nice escape hatch but not the goal.
- Marketing cards: on the flagship only, a 2x2 grid of emoji-icon cards
for Planner / Journal / Federation / Ownership, matching the Planner
home's visual weight. `IS_FLAGSHIP=true` toggles it. Self-hosters get
a "Powered by trails.cool — about the project" link back to the
flagship instead, so they don't have to write their own marketing.
- Public feed: reverse-chronological list of the 20 most recent public
activities on the instance, with owner + distance + date. Empty state
for fresh installs.
Plumbing:
- `listRecentPublicActivities(limit)` in activities.server.ts joins
users so the feed can render "by <displayName>" in one query.
- `IS_FLAGSHIP` env wired through docker-compose.yml; cd-apps and
cd-infra both write `IS_FLAGSHIP=true` alongside `DOMAIN=trails.cool`,
so self-hosters (who deploy without these workflows) default to off.
Specs:
- `activity-feed`: new "Instance-wide public activity feed" requirement
so the visibility contract is pinned (public in feed, unlisted +
private are not).
- `journal-landing`: new small spec capturing the hero / marketing /
feed layout contract and the `IS_FLAGSHIP` toggle, so future
instance-branding or empty-state changes have a stable anchor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The postgres init script grants read permissions to `grafana_reader` on
`public`, `planner`, `journal`, and `pgboss` schemas. It's mounted into
the postgres container via bind mount from
`/opt/trails-cool/postgres/init-grafana-user.sql`, and cd-infra re-runs
it with `psql -f` after postgres comes up.
But `cd-infra.yml` only scp's `infrastructure/postgres/queries.yml` to
the flagship — not `init-grafana-user.sql`. The file on the server was
the pre-pg-boss-upgrade March version with no pgboss grants. After
pg-boss v12 landed and (re)created the `pgboss` schema, the
grafana_reader lost schema-level USAGE, the "Background job failures"
alert query started erroring with `permission denied for schema pgboss`,
and the alert fired as "Alerting (Error)" for hours.
Fix: add the init SQL to the scp source list so changes to it actually
reach the flagship. I also manually re-granted on the live postgres to
clear the firing alert; the next cd-infra deploy will now pick up future
edits to the init file too.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The compose healthcheck for both apps was `curl -sf http://localhost:.../health`,
but node:25-slim (Debian trixie-slim) ships neither curl nor wget, so
the check always reported the containers as `unhealthy` even while they
were happily serving 200s. Install curl in both Dockerfiles.
Separately, today's flagship BRouter cleanup (PR #297) left an orphan
`brouter` container behind because cd-infra's `docker compose up -d <list>`
doesn't pass `--remove-orphans`. I had to remove it by hand. Pass the
flag in both cd-infra and cd-apps so future service deletions clean up
on the next deploy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three fixes after the first cd-brouter run on the dedicated host:
1. **BRouter 1.7.8 → 1.7.9** (`docker/brouter/Dockerfile`). Planet RD5
segments on brouter.de are now version 11; 1.7.8's `lookups.dat`
is v10, causing `lookup version mismatch (old rd5?)
lookups.dat=10 E10_N45.rd5=11` on every route request.
2. **cd-brouter.yml: docker login to ghcr.io before pull**.
ghcr.io/trails-cool/brouter is private, and the dedicated host's
Docker daemon isn't logged in by default. Extract DEPLOY_GHCR_TOKEN
from SOPS at runner side, pass to the SSH step via envs, and
`docker login` before `docker compose pull`. Credential is
`::add-mask::`-ed so it doesn't show in logs.
3. **Drop custom healthcheck** on the brouter service. The image
strips wget/curl post-build, and /bin/sh in the base doesn't
support /dev/tcp, so there's no in-image way to do an HTTP probe.
Real health is observed via Caddy's upstream 502 behavior on
outage and the Planner-side `brouter_request_duration_seconds`
metric. caddy's `depends_on` drops from service_healthy to
service_started.
End-to-end verified on the dedicated host after applying the compose
fix manually:
- Caddy enforces auth: 403 without header, proxies with.
- BRouter 1.7.9 will resolve the segment-version error once the image
is rebuilt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands sections 6 and 8 of relocate-brouter-to-dedicated-host on top
of the host compose in #291.
## Observability (section 6)
- **Prometheus**: new `brouter-cadvisor` job scraping
`10.0.1.10:8080` over the vSwitch, labeled `host="brouter"`.
- **cAdvisor sidecar** on the dedicated host's compose
(`--docker_only --whitelisted_container_labels=trails.cool.service`)
so metrics only cover trails containers, never the operator's
unrelated workloads on the shared box.
- **Promtail sidecar** on the dedicated host, Docker SD with relabel-
drop on missing `trails.cool.service` label, pushing to flagship
Loki at `http://10.0.0.2:3100/loki/api/v1/push`.
- **Flagship compose**: Loki now publishes port 3100 on the vSwitch
IP only (10.0.0.2:3100) — Hetzner Cloud firewall blocks it from
the public internet.
- **Grafana dashboard**: `brouter.json` — scrape up/down, request
rate (from Planner-side `brouter_request_duration_seconds`),
p50/p95/p99, container memory/CPU, Loki logs panel.
- **Alert**: `brouter-scrape-down` fires on
`up{job="brouter-cadvisor"} < 1 for 2m`; `noDataState: Alerting`
so a total scrape failure still pages.
Operator needs one UFW rule on the dedicated host for the cAdvisor
port — documented in `infrastructure/brouter-host/README.md`.
## Documentation (section 8)
- `CLAUDE.md` — hosts table + updated deployment table with SSH
targets per workflow; BRouter host SSH is `-p 2232 trails@...`,
different key.
- `docs/architecture.md` — Hosting section rewritten to cover both
hosts, vSwitch boundary, and the observability-scoping rationale
for the shared dedicated host.
- `docs/deployment.md` (new) — full operator runbook: host layout,
first-time BRouter provisioning, SOPS rotation (including the
macOS `SOPS_AGE_KEY_FILE` gotcha), cutover procedure with
rollback, manual workflow triggers.
Task 8.4 (infrastructure/README.md) skipped: that file doesn't
exist and the ground is covered by brouter-host/README.md +
docs/deployment.md.
## Validation
- `docker compose config` on both the flagship and brouter-host
compose files — both validate.
- `pnpm typecheck`, `pnpm lint`, `pnpm test` — all clean (full
turbo cache hits; no code changes in this commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands sections 3-5 of the relocate-brouter-to-dedicated-host change:
everything needed to run BRouter on the dedicated Hetzner Robot host
and have the Planner talk to it with the shared-secret header. Does
NOT flip the cutover — the flagship BRouter stays warm during soak.
## BRouter host compose (section 3)
New `infrastructure/brouter-host/` — a standalone compose project that
runs as the `trails` user on `ullrich.is`:
- `docker-compose.yml` — brouter + caddy sidecar. BRouter has no host
port; caddy binds only to `10.0.1.10:17777` (vSwitch IP). Every
service explicitly overrides the host's default Loki logging driver
to `json-file` so logs don't leak to the operator's personal Loki.
- `Caddyfile` — single-purpose reverse proxy that requires
`X-BRouter-Auth: ${BROUTER_AUTH_TOKEN}` on every request. `auto_https
off` (vSwitch-only); default access log format omits request
headers, so the token is never written to disk.
- `download-segments.sh` — crawls brouter.de, pulls planet-wide RD5
tiles via `wget -N` (incremental). Idempotent, safe to cron.
- `README.md` — one-shot provisioning + token rotation + rollback
notes.
`docker/brouter/Dockerfile` is patched to honor `JAVA_OPTS` (was
hardcoded `-Xmx1024M` in CMD). Default keeps the flagship's current
heap; compose on the dedicated host overrides to `-Xmx8g` for planet
scale on a 32 GB box.
## Planner shared-secret header (section 4)
`apps/planner/app/lib/brouter.ts`:
- Module-level guard: throws at startup in production if
`BROUTER_AUTH_TOKEN` is unset.
- `authHeaders()` helper (reads env at call time, so tests can
`vi.stubEnv` without module reset).
- Header attached on both `computeRoute` (per-segment) and
`computeSegmentGpx`.
3 new unit tests cover header attachment + the no-token path.
`infrastructure/docker-compose.yml` passes `BROUTER_AUTH_TOKEN` to
the Planner service, and makes `BROUTER_URL` overridable via SOPS so
the cutover is a one-variable flip.
## cd-brouter workflow (section 5)
Rewritten to deploy to the dedicated host:
- SSH as `trails@${BROUTER_DEPLOY_HOST}` on port
`${BROUTER_DEPLOY_SSH_PORT}` (2232) using
`${BROUTER_DEPLOY_SSH_KEY}`.
- Decrypts SOPS, extracts ONLY `BROUTER_AUTH_TOKEN` into a `.env`
file, scp'd alongside the compose project.
- `paths:` trigger now includes `infrastructure/brouter-host/**`.
- Segment download is NOT run here — first-time seed is a manual
operator step (multi-hour). Routine re-runs are cron-able on the
dedicated host.
- Grafana annotation step preserved (reaches flagship Grafana as
before).
## What's NOT here
- `brouter:` service on the flagship is intentionally left in place
(removed in section 7.5 after the 48 h soak window post-cutover).
- Observability (section 6) — Prometheus scrape + Loki shipping from
the dedicated host — comes in a follow-up PR.
- Cutover itself (section 7) — flip `BROUTER_URL`, verify, remove the
flagship brouter — is an operator action gated on first-time
provisioning + smoke testing.
## Verification
`pnpm typecheck && pnpm lint && pnpm test` all clean; planner build
passes (the CI regression from #286 was fixed in #290).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Commit subject prefixed with "[github-actions]" so the audit trail
is obvious at a glance in `git log`.
- GH_TOKEN exposed as an env var on the commit step so the PAT is
also available to any `gh` invocations the step might grow, and the
token plumbing is visible at the site where the push happens — not
only up at the checkout step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous comment claimed `git push` reads the token via a git
credential helper wired into $RUNNER_TEMP. I don't actually know that's
how it works end to end — only that checkout stashes the token there
and that subsequent git ops in the same workspace end up authenticating
as the PAT. Describe just the observable contract, not an unverified
internal path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Current actions/checkout stores the token in $RUNNER_TEMP and wires a
git credential helper, rather than writing the token into .git/config's
http.extraheader. The mechanism that makes `git push` pick up the PAT
is unchanged at the caller's level; the comment was just wrong about
where the secret lives on disk.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
