The flagship Caddy ran its admin API on 0.0.0.0:2019, reachable by
every container on the Docker network. The admin API can rewrite
routes and proxy targets, so a journal/planner RCE could repoint
traffic with no further auth. (The brouter-host Caddyfile already
binds admin to localhost; the flagship didn't.)
The admin endpoint also served Prometheus metrics (global `metrics`
option → admin endpoint), and Prometheus scrapes caddy:2019
cross-container — so admin couldn't just move to loopback without
breaking metrics. Split them:
- admin localhost:2019 (loopback only). All reloads are in-container
(`docker compose exec caddy caddy reload`) so they use this endpoint
unaffected.
- a dedicated `:2020` server exposes the read-only metrics handler;
prometheus.yml now scrapes caddy:2020.
Validated with `caddy validate` + `caddy adapt` against caddy:2:
admin.listen=localhost:2019, srv on :2020 carries the metrics handler,
the :443 site is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Single-file bind mounts (./foo.yml:/etc/foo.yml) pin to the host file's
inode at container-create time. The CD pipeline scp's a replacement file
(new inode), so the running container keeps reading the OLD inode —
`docker compose up -d` won't recreate on a content-only change, and
neither restart/SIGHUP/`caddy reload` re-reads the new file. Net effect:
config-only infra PRs deployed "successfully" but never took effect
(confirmed with PR #500's prometheus.yml; the WAL retention work only
applied because #498 also changed docker-compose.yml, forcing a
recreate). Caddyfile changes had the same latent gap.
Switch the four single-file config mounts to DIRECTORY mounts, which
resolve children live so a reload/restart picks up the new file:
- prometheus, loki, promtail: mount ./<svc> dir (loki/promtail
--config.file paths updated to the real filenames).
- caddy: move Caddyfile into caddy/ and mount the dir; ./sites overlays
/etc/caddy/sites (caddy/sites/.gitkeep keeps the mountpoint). Container
path /etc/caddy/Caddyfile is unchanged, so every `caddy reload --config
/etc/caddy/Caddyfile` ref still works. scp source paths in cd-infra and
cd-apps updated to ship the caddy/ dir.
cd-infra now applies config-only changes after `up -d`: SIGHUP prometheus
(zero downtime), restart loki+promtail (no SIGHUP reload), caddy reload
(graceful). Mirrored prometheus/loki mounts in docker-compose.dev.yml.
Validated: docker compose config (both files), caddy validate from the
new path, read-only-parent + sites-overlay mount mechanics, workflow YAML.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the visibility that would have caught the corrupt-WAL incident,
plus a direct Overpass upstream alert.
- prometheus.yml: self-scrape Prometheus (localhost:9090) and Loki
(loki:3100). Prometheus scraped everything except itself, so TSDB
health (compaction failures, WAL corruption, head series, retention
deletions) was invisible.
- monitoring-health.json: new "Monitoring Health" dashboard — TSDB
compaction/WAL failures, retention deletions/hour, head series,
samples/s, block bytes vs size-retention limit, retention depth
(oldest-sample age), Loki ingestion rate + memory chunks.
- alerts.yml: prometheus-compaction-failing (any compaction failure or
WAL corruption in 1h) and overpass-upstream-unhealthy (>20% upstream
failure over 10m — sustained public-Overpass degradation, distinct
from the symptom-level Caddy-502 alert).
Validated: promtool check config, YAML parse, JSON parse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
overview.json was missing a comma between the "Health Status" and
"App Error Rate (from logs)" panel objects, making the whole file
invalid JSON. Grafana's file provisioner skips dashboards that fail to
parse, so the entire "trails.cool Overview" dashboard (5 panels) never
loaded — a silent gap in observability. Validated with json.load.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two cardinality/volume cuts to keep Prometheus storage sustainable
within the size budget:
- scrape_interval 5s -> 20s (evaluation_interval too). 5s tripled the
sample volume vs the 15s default for no benefit on a single-box
deployment; all rules use [5m] windows and for: >= 1m.
- Drop high-cardinality cAdvisor families we never chart or alert on
(container_fs_*, blkio_*, tasks_state, memory_failures_total,
memory_numa_*, network_tcp*/udp*) on both cadvisor jobs. Dashboards
only use container CPU, memory, and start_time. This was the bulk of
the ~12.4k active series.
Validated with `promtool check config`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prometheus size-based retention counts the WAL toward its budget. With
retention.size=1GB and a stuck/corrupt WAL that had grown to 2.1GB, the
total was permanently over budget, so Prometheus deleted every
freshly-compacted block the moment it was written. Net effect: only the
in-memory ~2h head block was ever queryable — metrics appeared to "start
this morning" no matter when you looked, and last night's Caddy 502
spike (planner /api/overpass upstream stall, 18:33-18:34 UTC) was
unobservable in Prometheus (logs survived in Loki).
The corrupt WAL was cleared out-of-band on the flagship. This raises the
size backstop to 6GB so compacted blocks survive to the intended 15d
time-based retention. Disk is 38G (43% used), so 6GB is safe headroom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Garmin Connect as the third connected-services provider (spec:
garmin-import). The interesting parts:
- Push-first ingestion: Garmin has no list endpoint. The webhook
normalizes ping (callbackURL) and push (inline) notification batches
into events; the slow work (authorized FIT download, FIT→GPX via the
shared converter, activity creation) runs in a garmin-import-activity
pg-boss job so the webhook answers fast. Callback URLs are validated
against Garmin's API host before any fetch (SSRF guard).
- History via backfill requests: /sync/import/garmin is a date-range
requester with honest async progress (no pick list — the concept
doesn't exist in a push model). Ranges chunk to Garmin's 90-day cap;
overlaps are free via sync_imports dedupe. Requests persist in
import_batches via two new nullable columns (range_start/range_end).
- OAuth2 + PKCE on the existing oauth credential kind. Design
correction from apply: the verifier rides a short-lived httpOnly
cookie scoped to the callback path — the state param is visible in
redirect URLs and must never carry it. Manifests opt in via pkce:true.
- Deregistration notifications flip the connection to 'revoked'
(row kept for audit, imports retained, re-connect prompt shown).
- Framework evolutions, all additive: parseWebhook returns
WebhookEvent[] (Garmin batches; Wahoo adapted), manifest gains
configured()/importUrl/pkce, importActivity accepts summary stats
for FIT-less imports, manager gains markRevoked.
- Env-gated: no GARMIN_CLIENT_ID → provider hidden on
/settings/connections. Privacy manifest entry (DE+EN). i18n en+de.
Rollout (§6) stays gated on the Garmin Developer Program application
(submitted 2026-06-07). Fixtures are doc-shaped; the staging soak
swaps in recorded payloads if shapes differ.
Gate: typecheck ✓ lint ✓ unit+integration ✓ e2e 70/72 + both known
flakes green isolated ✓ openspec validate ✓
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wiring follows the IS_FLAGSHIP pattern: the compose env defaults every
FEDERATION_* var to empty (off — the safe self-host default; all
federation surfaces 404), and cd-apps.yml appends
FEDERATION_ENABLED=true + FEDERATION_LOG_LEVEL=info to the flagship's
app.env. FEDERATION_KEY_ENCRYPTION_KEY was already in SOPS.
Rollout 12.2/12.3 soaked on staging against a real Mastodon
(2026-06-06/07); the trails↔trails Accept bug found by the §11 harness
is fixed and deployed. Rollback: drop the two echo lines, merge, rerun
cd-apps — instant off, follow rows persist.
First boot with the flag enqueues backfill-user-keypairs (idempotent)
and registers the federation jobs. Applied via manual cd-apps dispatch
after merge since workflow-file changes don't trigger the path filter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Caddy 2.9 deprecated the nested `servers { metrics }` form; every
deploy logs a deprecation warning. The global `metrics` option is the
replacement and emits the same Prometheus metrics on the admin
endpoint, so the existing scrape config is unaffected.
Validated with `caddy validate` against the caddy:2 image — config
valid, no deprecation warnings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to #466: staging and preview services attach only to the
externally-named trails-shared network (created by the production
compose project) — they have no per-project default network, so the
enable_ipv6 added to the staging file was dead config. Move the flag
to trails-shared where it takes effect.
Rollout requires recreating trails-shared on the flagship: detach
staging, previews, and production postgres, remove the network, re-up.
Production journal/planner are unaffected (they use the default
network).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The journal container could not reach v6-only fediverse instances
(e.g. social.ullrich.is) because the compose project network was
IPv4-only. The flagship host has working IPv6, and Docker >= 27
auto-allocates a ULA subnet with NAT66 masquerading when a network
sets enable_ipv6, so the fix is a one-line opt-in per network.
Applying it to an existing deployment requires recreating the
network (docker compose down && up -d) — a brief full-stack
restart on the flagship and the persistent staging project.
PR-preview networks are created fresh per PR and need nothing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Inbound Mastodon Follows are being rejected 401 (signature
verification) on staging and Fedify's diagnostics were invisible:
it logs through LogTape, which is silent until configured.
- Configure LogTape with a console sink for the 'fedify' category when
the federation instance is built; level via FEDERATION_LOG_LEVEL
(default info).
- Staging deploy sets FEDERATION_LOG_LEVEL=debug for the soak —
signature-verification detail per request; dial down once proven.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
social-federation rollout step 12.2 — inbound soak. Persistent staging
gets FEDERATION_ENABLED=true; PR previews keep the flag empty so
preview journals never emit federation traffic.
- docker-compose.staging.yml: journal accepts FEDERATION_ENABLED +
FEDERATION_KEY_ENCRYPTION_KEY (both default empty).
- cd-staging.yml: persistent staging env sets FEDERATION_ENABLED=true;
the encryption key arrives via the existing SOPS decryption.
- secrets.app.env: new FEDERATION_KEY_ENCRYPTION_KEY (SOPS-encrypted,
generated with openssl rand -hex 32).
Production is unaffected: the flagship compose doesn't pass these vars
and the flag defaults off in code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Self-hosted instances no longer inherit the trails.cool flagship Sentry
DSNs. Code paths now read DSN strictly from env — unset = no Sentry
init, no events sent.
Flagship continues to report unchanged: cd-apps.yml and cd-staging.yml
inject the public DSNs as workflow env vars, which feed into:
- runtime via \`infrastructure/app.env\` / \`staging.env\`
(\`SENTRY_DSN_JOURNAL\` / \`SENTRY_DSN_PLANNER\` → compose env per service)
- build time via docker build-arg \`VITE_SENTRY_DSN\` (journal only;
planner has no client Sentry init).
Sentry DSNs are public-by-design (transmitted unencrypted from the
client JS bundle), so embedding them as plaintext workflow env vars
is no worse than the runtime exposure. Forks should replace these with
their own DSNs or remove the workflow env lines to ship Sentry-free.
Files changed:
- apps/journal/server.ts: \`process.env.SENTRY_DSN\` only, no fallback
- apps/planner/server.ts: same
- apps/journal/app/lib/sentry.client.ts: \`import.meta.env.VITE_SENTRY_DSN\`
only; falsy = skip init
- apps/journal/Dockerfile: new \`ARG VITE_SENTRY_DSN\` baked into build
- infrastructure/docker-compose.yml: pass \`SENTRY_DSN\` per service
- infrastructure/docker-compose.staging.yml: same
- .github/workflows/cd-apps.yml: workflow env + build-arg + app.env echo
- .github/workflows/cd-staging.yml: same
Full repo: pnpm typecheck, pnpm lint, pnpm test, journal build all green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
infrastructure/docker-compose.yml had \`\${VAR:-default}\` fallbacks for
JWT_SECRET, SESSION_SECRET, and POSTGRES_PASSWORD that silently
substituted known-weak values (\`change-me-in-production\`, \`trails\`)
when the env wasn't set. A misconfigured prod deploy would happily come
up with these defaults — JWT/session forgery + a guessable Postgres
password.
Switched the four critical substitutions to \`\${VAR:?message}\` so
\`docker compose up\` fails loud instead. Mirrors the existing pattern
already used for BROUTER_URL/BROUTER_AUTH_TOKEN and (in staging) for
JWT_SECRET/SESSION_SECRET.
Sites fixed:
- journal: DATABASE_URL (POSTGRES_PASSWORD), JWT_SECRET, SESSION_SECRET
- planner: DATABASE_URL (POSTGRES_PASSWORD)
- postgres: POSTGRES_PASSWORD (container itself)
- exporter: DATA_SOURCE_NAME (POSTGRES_PASSWORD)
- staging: both DATABASE_URLs
Production already provides all three via SOPS-encrypted
secrets.app.env, so this is a defense-in-depth change with no behavior
change on a properly-configured deploy.
Updated infrastructure/.env.example to make SESSION_SECRET explicit
(was missing) and call out the trio as required.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Stage deletions of openspec/changes/komoot-import/ (files were moved
to archive via shell mv, not git mv, so deletions weren't staged)
- Include SOPS secrets.app.env update adding INTEGRATION_SECRET
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add INTEGRATION_SECRET to journal service in docker-compose.yml with
:? guard so a missing value fails loudly at compose-up time
- Add INTEGRATION_SECRET to E2E test step in ci.yml via GitHub secret
(unit tests already set their own value in the test file)
- Archive openspec/changes/komoot-import → archive/2026-05-23-komoot-import
- Sync delta specs: new openspec/specs/komoot-import/spec.md,
updated openspec/specs/route-management/spec.md
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Companion to PR #358 which moved the staging compose ports off 3100/3101
(Loki conflict on the vSwitch). The Caddyfile staging blocks have the
upstream ports baked in so they need bumping too — already applied
manually on the flagship to unblock staging; this lands it in-repo so it
survives the next cd-infra deploy.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two bugs in the staging-environments rollout:
1. Staging containers published on 127.0.0.1 are unreachable from the
production Caddy container, which connects via the Docker bridge IP
(host.docker.internal:host-gateway resolves to the bridge, not
loopback). Bind to 0.0.0.0 instead — Hetzner Cloud firewall blocks
ports 3000+ from the public internet, so it stays internal-only.
2. The trailing 'docker compose ps' diagnostic in deploy-staging /
deploy-preview was missing --env-file staging.env, so compose
failed env interpolation and the job exited non-zero even when the
actual deploy succeeded.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the staging-environments OpenSpec change. Persistent staging at
staging.trails.cool / planner.staging.trails.cool deploys from main; PR
opens get a journal-only preview at pr-<N>.staging.trails.cool that shares
the persistent planner. cd-staging.yml builds tagged images, manages
per-PR Postgres databases and Caddyfile snippets, evicts the oldest
preview at the cap of 3, and tears everything down on PR close.
staging-cleanup.yml runs weekly to sweep orphaned previews.
DNS records (staging + *.staging A/AAAA) already applied to production
via tofu.
Caddy approach: per-PR Caddyfile snippets imported from /etc/caddy/sites/
and reloaded on each PR event — no wildcard / on-demand TLS, no router
service. Production compose gains a trails-shared network for the staging
project to reach Postgres, and host.docker.internal on Caddy so it can
reverse-proxy staging containers published on the host loopback.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The journal/planner deploy in cd-apps.yml does `docker compose up -d
journal planner`, which stops the old container and starts the new
one — Caddy keeps forwarding requests during the ~10–30s gap and
returns 502s. The caddy-502-rate alert (threshold > 0 for 2m)
correctly trips, every time.
Two production changes plus a long-broken workflow detail:
- infrastructure/Caddyfile — add `lb_try_duration 30s` /
`lb_try_interval 250ms` to the journal and planner reverse_proxy
blocks. Caddy now holds and retries the upstream for up to 30s
during a restart instead of 502'ing immediately. Real outages
(upstream unreachable longer than 30s) still 502 and the alert
still fires for those.
- infrastructure/grafana/provisioning/alerting/alerts.yml — add a
comment documenting why caddy-502-rate stays at threshold > 0:
with lb_try_duration in front of it, the alert no longer
conflates "deploy in flight" with "real outage."
- .github/workflows/cd-apps.yml — fix a long-silent bug: the
Grafana deploy-annotation step was reading
GRAFANA_SERVICE_TOKEN from `.env`, but the secrets file we scp
to /opt/trails-cool is named `app.env`. The token check failed
silently and the curl was being skipped on every deploy.
Switching to `app.env` so deploys actually annotate Grafana.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The old home was an h1, subtitle, and two auth buttons — visitors
arriving at trails.cool had no idea what it was, and self-hosted
instances had nothing interesting to land on.
The new home is one layout that serves both audiences:
- Hero: a product-describing h1 ("Federated outdoor journal") +
tagline. The site name already lives in the top banner and nav brand,
so the h1 carries the pitch instead of triplicating "trails.cool".
- Auth CTAs: Register (blue) + Sign In (outlined) get primary weight.
Planner demotes to a small "Or try the Planner without an account →"
link below them — it's a nice escape hatch but not the goal.
- Marketing cards: on the flagship only, a 2x2 grid of emoji-icon cards
for Planner / Journal / Federation / Ownership, matching the Planner
home's visual weight. `IS_FLAGSHIP=true` toggles it. Self-hosters get
a "Powered by trails.cool — about the project" link back to the
flagship instead, so they don't have to write their own marketing.
- Public feed: reverse-chronological list of the 20 most recent public
activities on the instance, with owner + distance + date. Empty state
for fresh installs.
Plumbing:
- `listRecentPublicActivities(limit)` in activities.server.ts joins
users so the feed can render "by <displayName>" in one query.
- `IS_FLAGSHIP` env wired through docker-compose.yml; cd-apps and
cd-infra both write `IS_FLAGSHIP=true` alongside `DOMAIN=trails.cool`,
so self-hosters (who deploy without these workflows) default to off.
Specs:
- `activity-feed`: new "Instance-wide public activity feed" requirement
so the visibility contract is pinned (public in feed, unlisted +
private are not).
- `journal-landing`: new small spec capturing the hero / marketing /
feed layout contract and the `IS_FLAGSHIP` toggle, so future
instance-branding or empty-state changes have a stable anchor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The notification policy and contact points live in
`infrastructure/grafana/provisioning/alerting/alerts.yml`, so
UI-created contact points can't be attached to provisioned alerts.
Provision the Pushover integration instead.
Rather than add a second contact point and do the fan-out in the
notification tree (Grafana's tree only fires one receiver per match
without duplication workarounds), put both email + pushover under a
single `default` contact point. Every alert now fans out to both.
Secrets go to SOPS `secrets.infra.env`. The Grafana container reads
`PUSHOVER_API_TOKEN` and `PUSHOVER_USER_KEY_ULLRICH` from env; the
provisioning YAML references them via `$VAR` and Grafana substitutes
at startup, so no secret lands in git.
Priority 1 on firing (bypass quiet hours), 0 on resolve. User can
tweak in the YAML later.
Post-deploy: delete the UI-created "Pushover Ullrich" contact point
in Grafana to avoid a duplicate. It's unused (no policy references it)
but shows up in the contact-point list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Users land via GitHub OAuth gated by the trails-cool org allowlist, so
Viewer-only is more restrictive than we actually want — no Explore
access, no ad-hoc Loki/Prometheus queries. Promote the default role to
Editor so new sign-ins get it automatically.
Note: GF_USERS_AUTO_ASSIGN_ORG_ROLE only applies to new users on first
sign-in. Existing Viewers (including the single current user) need a
one-off promotion via the Grafana admin UI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After PR #293 flipped BROUTER_URL to the dedicated host and Grafana +
manual smoke tests confirmed clean routing (US route worked, request
rate/latency/memory/logs all green), there's no reason to keep the
in-tree flagship brouter container warm any longer.
- Remove the `brouter:` service and its `./segments` bind mount from
`infrastructure/docker-compose.yml`.
- Remove `depends_on: brouter` from the planner service.
- Tighten the BROUTER_URL and BROUTER_AUTH_TOKEN env wiring from
`${…:-default}` to `${…:?message}` so a missing SOPS value fails the
compose up loudly instead of silently pointing at the removed service
(or missing auth).
- Tick tasks 5.5, 7.5, 8.4, 9.3 in the OpenSpec change; 9.4 (archive)
is the last remaining step once this merges.
Post-merge operator step: `docker image prune -f` on the flagship to
reclaim the brouter image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a user rapidly moves waypoints in the Planner, BRouter's
thread-priority watchdog kills older in-flight requests to free a thread
for newer ones, surfacing to the user as "operation killed by
thread-priority-watchdog" errors.
Two fixes:
1. Bump the BRouter thread pool on the dedicated host from 4 to 16.
`BROUTER_THREADS` is now a Dockerfile ENV (default 4 for
flagship/CI/dev) overridden to 16 in `infrastructure/brouter-host/`.
The host has 32 GB of RAM and idle cores; 4 threads was a flagship-era
constraint that no longer applies.
2. Cancel the previous in-flight `/api/route` request when a new one
starts. `use-routing.ts` now keeps an `AbortController` in a ref,
aborts it at the top of each `computeRoute`, and bails quietly on
`AbortError` so the newer call drives state.
Together these eliminate both the "real" contention (at the BRouter
level) and the wasted work (BRouter computing a route the client no
longer cares about).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds BROUTER_URL=http://10.0.1.10:17777 (vSwitch) to SOPS so the Planner
starts routing through the dedicated host's Caddy sidecar instead of the
in-tree flagship BRouter. Flagship BRouter stays warm for the 48h
soak/rollback window.
Pre-flight from the flagship confirmed the new host answers 200 with the
auth header and 403 without.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three fixes after the first cd-brouter run on the dedicated host:
1. **BRouter 1.7.8 → 1.7.9** (`docker/brouter/Dockerfile`). Planet RD5
segments on brouter.de are now version 11; 1.7.8's `lookups.dat`
is v10, causing `lookup version mismatch (old rd5?)
lookups.dat=10 E10_N45.rd5=11` on every route request.
2. **cd-brouter.yml: docker login to ghcr.io before pull**.
ghcr.io/trails-cool/brouter is private, and the dedicated host's
Docker daemon isn't logged in by default. Extract DEPLOY_GHCR_TOKEN
from SOPS at runner side, pass to the SSH step via envs, and
`docker login` before `docker compose pull`. Credential is
`::add-mask::`-ed so it doesn't show in logs.
3. **Drop custom healthcheck** on the brouter service. The image
strips wget/curl post-build, and /bin/sh in the base doesn't
support /dev/tcp, so there's no in-image way to do an HTTP probe.
Real health is observed via Caddy's upstream 502 behavior on
outage and the Planner-side `brouter_request_duration_seconds`
metric. caddy's `depends_on` drops from service_healthy to
service_started.
End-to-end verified on the dedicated host after applying the compose
fix manually:
- Caddy enforces auth: 403 without header, proxies with.
- BRouter 1.7.9 will resolve the segment-version error once the image
is rebuilt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands sections 6 and 8 of relocate-brouter-to-dedicated-host on top
of the host compose in #291.
## Observability (section 6)
- **Prometheus**: new `brouter-cadvisor` job scraping
`10.0.1.10:8080` over the vSwitch, labeled `host="brouter"`.
- **cAdvisor sidecar** on the dedicated host's compose
(`--docker_only --whitelisted_container_labels=trails.cool.service`)
so metrics only cover trails containers, never the operator's
unrelated workloads on the shared box.
- **Promtail sidecar** on the dedicated host, Docker SD with relabel-
drop on missing `trails.cool.service` label, pushing to flagship
Loki at `http://10.0.0.2:3100/loki/api/v1/push`.
- **Flagship compose**: Loki now publishes port 3100 on the vSwitch
IP only (10.0.0.2:3100) — Hetzner Cloud firewall blocks it from
the public internet.
- **Grafana dashboard**: `brouter.json` — scrape up/down, request
rate (from Planner-side `brouter_request_duration_seconds`),
p50/p95/p99, container memory/CPU, Loki logs panel.
- **Alert**: `brouter-scrape-down` fires on
`up{job="brouter-cadvisor"} < 1 for 2m`; `noDataState: Alerting`
so a total scrape failure still pages.
Operator needs one UFW rule on the dedicated host for the cAdvisor
port — documented in `infrastructure/brouter-host/README.md`.
## Documentation (section 8)
- `CLAUDE.md` — hosts table + updated deployment table with SSH
targets per workflow; BRouter host SSH is `-p 2232 trails@...`,
different key.
- `docs/architecture.md` — Hosting section rewritten to cover both
hosts, vSwitch boundary, and the observability-scoping rationale
for the shared dedicated host.
- `docs/deployment.md` (new) — full operator runbook: host layout,
first-time BRouter provisioning, SOPS rotation (including the
macOS `SOPS_AGE_KEY_FILE` gotcha), cutover procedure with
rollback, manual workflow triggers.
Task 8.4 (infrastructure/README.md) skipped: that file doesn't
exist and the ground is covered by brouter-host/README.md +
docs/deployment.md.
## Validation
- `docker compose config` on both the flagship and brouter-host
compose files — both validate.
- `pnpm typecheck`, `pnpm lint`, `pnpm test` — all clean (full
turbo cache hits; no code changes in this commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands sections 3-5 of the relocate-brouter-to-dedicated-host change:
everything needed to run BRouter on the dedicated Hetzner Robot host
and have the Planner talk to it with the shared-secret header. Does
NOT flip the cutover — the flagship BRouter stays warm during soak.
## BRouter host compose (section 3)
New `infrastructure/brouter-host/` — a standalone compose project that
runs as the `trails` user on `ullrich.is`:
- `docker-compose.yml` — brouter + caddy sidecar. BRouter has no host
port; caddy binds only to `10.0.1.10:17777` (vSwitch IP). Every
service explicitly overrides the host's default Loki logging driver
to `json-file` so logs don't leak to the operator's personal Loki.
- `Caddyfile` — single-purpose reverse proxy that requires
`X-BRouter-Auth: ${BROUTER_AUTH_TOKEN}` on every request. `auto_https
off` (vSwitch-only); default access log format omits request
headers, so the token is never written to disk.
- `download-segments.sh` — crawls brouter.de, pulls planet-wide RD5
tiles via `wget -N` (incremental). Idempotent, safe to cron.
- `README.md` — one-shot provisioning + token rotation + rollback
notes.
`docker/brouter/Dockerfile` is patched to honor `JAVA_OPTS` (was
hardcoded `-Xmx1024M` in CMD). Default keeps the flagship's current
heap; compose on the dedicated host overrides to `-Xmx8g` for planet
scale on a 32 GB box.
## Planner shared-secret header (section 4)
`apps/planner/app/lib/brouter.ts`:
- Module-level guard: throws at startup in production if
`BROUTER_AUTH_TOKEN` is unset.
- `authHeaders()` helper (reads env at call time, so tests can
`vi.stubEnv` without module reset).
- Header attached on both `computeRoute` (per-segment) and
`computeSegmentGpx`.
3 new unit tests cover header attachment + the no-token path.
`infrastructure/docker-compose.yml` passes `BROUTER_AUTH_TOKEN` to
the Planner service, and makes `BROUTER_URL` overridable via SOPS so
the cutover is a one-variable flip.
## cd-brouter workflow (section 5)
Rewritten to deploy to the dedicated host:
- SSH as `trails@${BROUTER_DEPLOY_HOST}` on port
`${BROUTER_DEPLOY_SSH_PORT}` (2232) using
`${BROUTER_DEPLOY_SSH_KEY}`.
- Decrypts SOPS, extracts ONLY `BROUTER_AUTH_TOKEN` into a `.env`
file, scp'd alongside the compose project.
- `paths:` trigger now includes `infrastructure/brouter-host/**`.
- Segment download is NOT run here — first-time seed is a manual
operator step (multi-hour). Routine re-runs are cron-able on the
dedicated host.
- Grafana annotation step preserved (reaches flagship Grafana as
before).
## What's NOT here
- `brouter:` service on the flagship is intentionally left in place
(removed in section 7.5 after the 48 h soak window post-cutover).
- Observability (section 6) — Prometheus scrape + Loki shipping from
the dedicated host — comes in a follow-up PR.
- Cutover itself (section 7) — flip `BROUTER_URL`, verify, remove the
flagship brouter — is an operator action gated on first-time
provisioning + smoke testing.
## Verification
`pnpm typecheck && pnpm lint && pnpm test` all clean; planner build
passes (the CI regression from #286 was fixed in #290).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the shared-secret header value consumed by the Planner
(as X-BRouter-Auth outbound) and by the BRouter host's Caddy
sidecar (enforced inbound). Lives in secrets.app.env only;
cd-brouter.yml will also read from this file, making it the
single source of truth. secrets.infra.env is intentionally NOT
updated — cd-infra no longer touches BRouter after the move.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an OpenSpec change scoping the relocation of BRouter from the
co-located flagship (cx23, 4 GB RAM, 40 GB SSD, Europe segments only) to
a dedicated Hetzner Robot host in the same datacenter, with private
connectivity over Hetzner vSwitch #80672 (VLAN 4000).
This first PR only lays the network prerequisite:
- Terraform: a Hetzner Cloud Network (10.0.0.0/16) with a cloud subnet
(10.0.0.0/24) hosting the flagship at 10.0.0.2, and a vSwitch subnet
(10.0.1.0/24) bridged to Robot VLAN 4000. The dedicated host's VLAN
sub-interface (10.0.1.10 on enp4s0.4000) is configured out-of-band via
netplan and is not Terraform-managed.
- lifecycle { ignore_changes = [user_data] } on the flagship server to
prevent the Hetzner provider's post-1.45 user_data hash drift from
triggering a spurious full-server replacement on unrelated applies.
- OpenSpec change with proposal, design, specs (delta for
brouter-integration / infrastructure / observability /
security-hardening), and tasks; Section 1 (pre-flight) is checked off
with operator notes.
Verification: ping both directions across the vSwitch is 0% loss,
sub-ms latency; dedicated host's VLAN config persists across reboot
(verified ~60 s to restore private reachability).
Follow-up PRs will land the BRouter host compose project, Planner
shared-secret header, CD workflow retarget, observability wiring, and
the cutover.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Public Overpass instances go bad without warning — we just lived
through a day where overpass.private.coffee (our single upstream)
returned 504s after 60s while lz4.overpass-api.de served the same
query in 0.6s. Survive that by trying a list of upstreams in order
until one responds usefully.
Server changes:
- OVERPASS_URLS env — comma-separated list, tried in order. Falls
back to OVERPASS_URL for backward compat, then to a built-in
default pair (lz4.overpass-api.de, overpass-api.de).
- Per-upstream timeout of 10s via AbortSignal.timeout, so a saturated
instance fails over in seconds, not minutes.
- Client abort propagation via AbortSignal.any — if the browser
cancels (e.g. user panned the map), the in-flight upstream fetch is
aborted too. Stops wasting upstream capacity on requests nobody
wants.
- Rate-limited-body detection (Overpass returns 200 with a
`rate_limited` marker when throttling) triggers failover.
- Per-upstream labels on overpass_upstream_requests_total and
overpass_upstream_duration_seconds so the dashboard can break out
health + latency by instance. Histogram buckets extended to 30s.
Compose: OVERPASS_URLS passthrough with the default pair hard-wired,
overridable via SOPS.
Tests: 8 cases covering the new fetchWithFailover helper — happy
path, 5xx failover, rate_limited failover, network error failover,
timeout tagging, all-upstreams-fail, client abort stops the loop,
per-attempt latency observation.
Follow-ups left out of scope:
- Client-side debounce (UI concern).
- Moving `.observe()` after body read for accuracy in measurement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pg-boss v11 renamed every concatenated-lowercase timestamp column
(`createdon`, `completedon`) to snake_case (`created_on`,
`completed_on`). Our service-health dashboard and failed-job alert
still referenced the v10 names, so after the v12 upgrade those
queries error out silently.
Fix the three affected queries:
- alerts.yml: failed-job alert
- service-health.json: "Recent Failed Jobs" table
- service-health.json: "Job Queue — Completed/hour" time series
The `state` / `name` / `output` columns stay the same, and the `state`
enum still accepts `'failed'` / `'completed'` literals.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Nine panels at trails-demo-bot covering:
- Current synthetic route/activity counts + time-since-last-walk
- Gauge time series over the configured retention window
- Walks per day + distance distribution
- Loki log stream filtered to `demo-bot` messages
- Recent walks table with name, distance, ascent
Provisioned via the existing grafana/dashboards directory — no extra
wiring needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sets DEMO_BOT_ENABLED=true plus DEMO_BOT_RETENTION_DAYS and
DEMO_BOT_REGION in the SOPS env file so the journal worker starts
the generate + prune jobs and bootstraps the Bruno demo user on
next deploy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds DEMO_BOT_PERSONA env var (inline JSON or file:<path>) so
self-hosted instances can replace Bruno-in-Berlin with their own
demo account identity and content pools. No-op for trails.cool — the
built-in Bruno persona remains the default.
- DemoPersona type + Zod schema validation
- loadPersona() cached at boot; falls back to default on any failure
- ensureDemoUser throws DemoPersonaUsernameClashError when the
persona username is already a real user; server declines to
schedule demo jobs for that process
- isDemoUser flag moved from hardcoded "bruno" check to loader-supplied
boolean computed from the running persona
- docs/demo-persona.md explains the schema + operator flow
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a background demo user (`bruno`) plus two pg-boss jobs that generate
public trekking routes and activities in inner Berlin and prune them
after 14 days. Disabled by default everywhere; flip `DEMO_BOT_ENABLED`
only in prod.
- Schema: `synthetic` boolean on routes + activities
- `ensureDemoUser()` idempotent insert + badge on /users/bruno
- Generation gate: 07-21 Berlin local, p=0.12, 40-route cap in 14d
- Initial 4-walk backfill on first enable; retention via daily prune
- Prometheus gauges `demo_bot_synthetic_routes_total` / `_activities_total`
- Unit + DB-gated integration + E2E tests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original formulas used clamp_min(denominator, 1) to avoid divide-
by-zero, but that inflates the denominator when request rate is < 1/s.
Real example from prod: ~0.04 req/s, all successful, displayed as
1.02% health (red) instead of ~100%.
Switch to (numerator or vector(0)) / denominator:
- Missing "hit" series (no cache hits yet) → numerator resolves to 0
instead of empty set, so the stat shows 0% instead of "No data".
- Zero traffic (empty denominator) → whole expr is empty, Grafana
renders "No data", which is the correct behaviour for a health
stat when nothing's happening.
- Non-zero low traffic → ratio is true ratio, not artificially
depressed by the clamp.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds four metrics surfaced via the existing /metrics endpoint:
- overpass_cache_events_total{result=hit|miss|coalesced}
- overpass_cache_size (gauge)
- overpass_upstream_duration_seconds (histogram)
- overpass_upstream_requests_total{status} — covers 2xx/4xx/5xx and
an explicit "error" label for network-level failures
Grafana dashboards/planner.json gets a new row:
- Upstream health (5m 2xx success ratio, red <90%, green >99%)
- Cache hit ratio
- Cache size
- Upstream p95 latency
Plus timeseries panels for cache event breakdown, upstream status
over time, and p50/p95/p99 upstream latency.
The success-rate formula uses clamp_min(..., 1) on the denominator so
it doesn't NaN when traffic is zero.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds apps/planner/app/routes/api.overpass.ts — a same-origin, rate-limited
server-side proxy that forwards Overpass QL to the upstream endpoint
configured via OVERPASS_URL (default: overpass.private.coffee).
Motivation:
- private.coffee's best-practices require a meaningful User-Agent
identifying the project. Browsers cannot set User-Agent on fetch()
(forbidden header), so the request has to originate server-side.
- Collaborative sessions commonly have N clients pan/zoom the same map,
so the same bbox query arrives multiple times within seconds.
- Setting up the proxy now lets us swap OVERPASS_URL to a self-hosted
Overpass later without client changes.
What the proxy does:
- Sets User-Agent "trails.cool Planner (https://trails.cool; legal@trails.cool)"
- Enforces same-origin via the Origin header
- Rate-limits per client IP (120 req/min)
- In-memory LRU cache of upstream responses keyed on the form-encoded body
(TTL 10 min, max 200 entries)
- Coalesces concurrent misses for the same key so N simultaneous clients
in one session incur exactly one upstream call
Client change: apps/planner/app/lib/overpass.ts POSTs to /api/overpass
instead of iterating over public Overpass endpoints. Fallback list
removed; resilience is now the proxy's responsibility.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add @trails-cool/jobs package wrapping pg-boss for durable background job
execution using the existing PostgreSQL database. Wire up planner session
expiry as the first scheduled job (hourly, 7-day TTL) — this was previously
dead code that never ran. Add placeholder worker to journal for future
Komoot import and federation jobs.
Infrastructure: grant grafana_reader access to pgboss schema, add job queue
health panels to Service Health dashboard, add failed job alert.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix blind spot where Caddy 502 errors were invisible: the dashboards and
alerts queried caddy_http_response_duration_seconds (upstream responses only),
missing 502s that Caddy generates itself. Switch to
caddy_http_request_duration_seconds (server-level, all responses).
Add Journal Grafana dashboard with: 502 rate, response codes, request rate
by route, latency percentiles, container restarts/memory/CPU, Node.js event
loop lag and heap, and Loki log panels for errors and Caddy 5xx entries.
Add color coding to Caddy status code panel (green=2xx, blue=3xx, yellow=4xx,
red=5xx). Add log-based error rate panel to the overview dashboard.
New alerts: container restart loop, PostgreSQL connections > 80, application
crash log detection (Loki), and Caddy 502 rate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Switch journal Docker health check from `node -e "fetch(...)"` to `curl -sf`,
matching the planner. The Node.js process spawn was heavy and prone to timeout
under memory pressure, causing false health check failures and 502s.
Upgrade Promtail from static file scraping to Docker service discovery so logs
get `service` and `container` labels. Parse Pino JSON logs to extract the
`level` label, enabling Loki queries filtered by log level.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
