Public Overpass instances go bad without warning — we just lived
through a day where overpass.private.coffee (our single upstream)
returned 504s after 60s while lz4.overpass-api.de served the same
query in 0.6s. Survive that by trying a list of upstreams in order
until one responds usefully.
Server changes:
- OVERPASS_URLS env — comma-separated list, tried in order. Falls
back to OVERPASS_URL for backward compat, then to a built-in
default pair (lz4.overpass-api.de, overpass-api.de).
- Per-upstream timeout of 10s via AbortSignal.timeout, so a saturated
instance fails over in seconds, not minutes.
- Client abort propagation via AbortSignal.any — if the browser
cancels (e.g. user panned the map), the in-flight upstream fetch is
aborted too. Stops wasting upstream capacity on requests nobody
wants.
- Rate-limited-body detection (Overpass returns 200 with a
`rate_limited` marker when throttling) triggers failover.
- Per-upstream labels on overpass_upstream_requests_total and
overpass_upstream_duration_seconds so the dashboard can break out
health + latency by instance. Histogram buckets extended to 30s.
Compose: OVERPASS_URLS passthrough with the default pair hard-wired,
overridable via SOPS.
Tests: 8 cases covering the new fetchWithFailover helper — happy
path, 5xx failover, rate_limited failover, network error failover,
timeout tagging, all-upstreams-fail, client abort stops the loop,
per-attempt latency observation.
Follow-ups left out of scope:
- Client-side debounce (UI concern).
- Moving `.observe()` after body read for accuracy in measurement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds DEMO_BOT_PERSONA env var (inline JSON or file:<path>) so
self-hosted instances can replace Bruno-in-Berlin with their own
demo account identity and content pools. No-op for trails.cool — the
built-in Bruno persona remains the default.
- DemoPersona type + Zod schema validation
- loadPersona() cached at boot; falls back to default on any failure
- ensureDemoUser throws DemoPersonaUsernameClashError when the
persona username is already a real user; server declines to
schedule demo jobs for that process
- isDemoUser flag moved from hardcoded "bruno" check to loader-supplied
boolean computed from the running persona
- docs/demo-persona.md explains the schema + operator flow
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a background demo user (`bruno`) plus two pg-boss jobs that generate
public trekking routes and activities in inner Berlin and prune them
after 14 days. Disabled by default everywhere; flip `DEMO_BOT_ENABLED`
only in prod.
- Schema: `synthetic` boolean on routes + activities
- `ensureDemoUser()` idempotent insert + badge on /users/bruno
- Generation gate: 07-21 Berlin local, p=0.12, 40-route cap in 14d
- Initial 4-walk backfill on first enable; retention via daily prune
- Prometheus gauges `demo_bot_synthetic_routes_total` / `_activities_total`
- Unit + DB-gated integration + E2E tests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds apps/planner/app/routes/api.overpass.ts — a same-origin, rate-limited
server-side proxy that forwards Overpass QL to the upstream endpoint
configured via OVERPASS_URL (default: overpass.private.coffee).
Motivation:
- private.coffee's best-practices require a meaningful User-Agent
identifying the project. Browsers cannot set User-Agent on fetch()
(forbidden header), so the request has to originate server-side.
- Collaborative sessions commonly have N clients pan/zoom the same map,
so the same bbox query arrives multiple times within seconds.
- Setting up the proxy now lets us swap OVERPASS_URL to a self-hosted
Overpass later without client changes.
What the proxy does:
- Sets User-Agent "trails.cool Planner (https://trails.cool; legal@trails.cool)"
- Enforces same-origin via the Origin header
- Rate-limits per client IP (120 req/min)
- In-memory LRU cache of upstream responses keyed on the form-encoded body
(TTL 10 min, max 200 entries)
- Coalesces concurrent misses for the same key so N simultaneous clients
in one session incur exactly one upstream call
Client change: apps/planner/app/lib/overpass.ts POSTs to /api/overpass
instead of iterating over public Overpass endpoints. Fallback list
removed; resilience is now the proxy's responsibility.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switch journal Docker health check from `node -e "fetch(...)"` to `curl -sf`,
matching the planner. The Node.js process spawn was heavy and prone to timeout
under memory pressure, causing false health check failures and 502s.
Upgrade Promtail from static file scraping to Docker service discovery so logs
get `service` and `container` labels. Parse Pino JSON logs to extract the
`level` label, enabling Loki queries filtered by log level.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dashboard: switch error rate panel from app-level http_request_duration
(never observed) to Caddy's caddy_http_response_duration_seconds_count.
Journal: add custom server.ts (matching planner pattern) that observes
http_request_duration_seconds on every request. Replaces react-router-serve
with a custom Node HTTP server that handles /api/health, /api/metrics,
static assets, and request timing. Removes redundant React Router routes
for health and metrics.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Caddy Response Status Codes: caddy_http_responses_total doesn't exist,
use caddy_http_response_duration_seconds_count which has the code label
- pg_stat_statements: enable extension in init script, add custom queries
config for postgres-exporter, remove invalid datname filter from query
- Add postgres/queries.yml to cd-infra SCP sources
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Loki was running but had no log shipper — no labels or logs were
visible in Grafana. Adds Promtail to scrape all Docker container
logs and push them to Loki.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
WAHOO_CLIENT_ID, WAHOO_CLIENT_SECRET, and WAHOO_WEBHOOK_TOKEN were in
SOPS secrets but never passed through to the journal service in
docker-compose.yml, causing the OAuth authorize URL to have an empty
client_id.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without this, GitHub OAuth creates a new Grafana user on every login
instead of linking to the existing account by email.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Create grafana_reader role with SELECT-only access on all schemas
- Init script runs on postgres first boot (docker-entrypoint-initdb.d)
- Grafana PostgreSQL datasource provisioned with read-only credentials
- Enables SQL queries in dashboards (user count, routes, etc.)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Exporters:
- postgres_exporter: DB connections, transactions, cache hit ratio, query stats
- node_exporter: host CPU, memory, disk, network
- cAdvisor: per-container CPU and memory usage
PostgreSQL:
- Enable pg_stat_statements for query-level performance tracking
- Track index scans vs sequential scans, cache hit ratio
Dashboard (service-health.json):
- DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage
- Host: disk gauge, CPU, memory, network I/O, disk I/O
- BRouter: request latency p50/p95/p99, container CPU + memory
- All containers: CPU and memory comparison
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Secrets:
- Add .sops.yaml with age encryption config
- Add encrypted secrets.app.env (app secrets) and secrets.infra.env (Grafana OAuth)
- CD decrypts at deploy time with AGE_SECRET_KEY — all other secrets
move out of GitHub Actions into version-controlled encrypted files
Split CD:
- cd-apps.yml: triggered by apps/packages changes, builds Docker images, deploys apps
- cd-infra.yml: triggered by infrastructure/ changes, copies configs, restarts services
- Remove monolithic cd.yml
Grafana auth:
- GitHub OAuth (trails-cool org), disable login form
- Remove Caddy basic_auth block and all GRAFANA_* env vars/secrets
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add DOMAIN, GRAFANA_USER, GRAFANA_PASSWORD_HASH as environment vars
on the Caddy container so Caddyfile can reference them
- Fix Loki compactor: add delete_request_store required when retention is enabled
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add wildcard DNS record for *.internal.trails.cool (A + AAAA) so
internal services don't leak hostnames in individual DNS entries.
Move Grafana from grafana.trails.cool to grafana.internal.trails.cool.
Future internal services just need a Caddy block — no DNS changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Health endpoints:
- /api/health (Journal) and /health (Planner) with DB connectivity check
- Docker healthchecks updated to use app health endpoints
Structured logging:
- Pino with JSON output in production, pretty-print in dev
- Request logging middleware in Planner (method, path, status, duration)
- Replaced console.log/error with structured logger in email and auth flows
Prometheus metrics:
- prom-client with default Node.js metrics + custom histograms/gauges
- /metrics endpoints on both apps
- http_request_duration, planner_active_sessions, brouter_request_duration
Monitoring stack:
- Prometheus, Loki, Grafana containers in docker-compose
- Grafana provisioned with datasources, dashboards, and alert rules
- Caddy access logging (JSON to stdout for Loki)
- grafana.trails.cool with basic auth via Caddy
Dashboards and alerting:
- Overview: request rate, error rate, latency p50/p95/p99
- Planner: active sessions, connected clients, BRouter latency
- Infrastructure: memory, CPU, event loop lag
- Alerts: disk >80%, app down 2min, error rate >5%
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Transactional emails:
- Add nodemailer SMTP email module with dev-mode console logging
- Magic link template and welcome template with HTML + plain text
- Wire sendMagicLink into login flow, sendWelcome into registration
- Update privacy page and deploy docs for SMTP configuration
Planner features:
- No-go areas: draw polygons on map (leaflet-geoman), synced via Yjs,
passed to BRouter as nogos parameter, route recomputes on change
- Session notes: collaborative Y.Text textarea in sidebar tab
- Crash recovery: periodic localStorage save of Yjs state, restore on reconnect
- Rate limit session creation (10/IP/hour) in /new route
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- @sentry/vite-plugin uploads source maps during Docker build
- Release tagged with git SHA (SENTRY_RELEASE) on both client and server
- Environment set to production/development for filtering
- CD passes SENTRY_AUTH_TOKEN + SENTRY_RELEASE as Docker build args
- docker-compose passes SENTRY_RELEASE to runtime containers
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CD workflow:
- Uncommented deploy step
- Copy docker-compose.yml + Caddyfile to server via SCP
- Login to ghcr.io with DEPLOY_GHCR_TOKEN (classic PAT, read:packages)
- Download Germany BRouter segments on first deploy (~750MB)
- Pull images, restart services, verify
Infrastructure:
- BRouter segments as bind mount (./segments) not Docker volume
- Added ORIGIN, PLANNER_URL, SESSION_SECRET, NODE_ENV to Journal
- Added NODE_ENV to Planner
- Disabled Garage for now (no media storage yet)
Required secrets: DEPLOY_HOST, DEPLOY_SSH_KEY, DEPLOY_GHCR_TOKEN
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Terraform:
- Hetzner CX22 server with Docker, firewall (80/443/22)
- SSH key and output for server IP
Docker Compose:
- Caddy reverse proxy (auto HTTPS)
- Journal and Planner app containers
- BRouter routing engine container
- PostgreSQL + PostGIS
- Garage S3 storage
BRouter:
- Dockerfile based on eclipse-temurin:11-jre
- Segment download script for Germany (4 tiles, ~750MB)
CD Pipeline:
- Build & push Docker images to ghcr.io on main push
- Deploy step commented out (enable when server provisioned)
Scripts:
- PostgreSQL daily backup with 14-day retention
- Dockerfiles for both apps (multi-stage builds)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
