Dashboard: switch error rate panel from app-level http_request_duration
(never observed) to Caddy's caddy_http_response_duration_seconds_count.
Journal: add custom server.ts (matching planner pattern) that observes
http_request_duration_seconds on every request. Replaces react-router-serve
with a custom Node HTTP server that handles /api/health, /api/metrics,
static assets, and request timing. Removes redundant React Router routes
for health and metrics.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Caddy Response Status Codes: caddy_http_responses_total doesn't exist,
use caddy_http_response_duration_seconds_count which has the code label
- pg_stat_statements: enable extension in init script, add custom queries
config for postgres-exporter, remove invalid datname filter from query
- Add postgres/queries.yml to cd-infra SCP sources
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Loki was running but had no log shipper — no labels or logs were
visible in Grafana. Adds Promtail to scrape all Docker container
logs and push them to Loki.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
WAHOO_CLIENT_ID, WAHOO_CLIENT_SECRET, and WAHOO_WEBHOOK_TOKEN were in
SOPS secrets but never passed through to the journal service in
docker-compose.yml, causing the OAuth authorize URL to have an empty
client_id.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without this, GitHub OAuth creates a new Grafana user on every login
instead of linking to the existing account by email.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add SSH hardening to user_data for future server rebuilds:
- PasswordAuthentication no (key-only access)
- X11Forwarding no (headless server)
- AllowTcpForwarding no (no SSH tunnels needed)
- AllowAgentForwarding no (no agent forwarding needed)
- MaxAuthTries 3 (reduced from 6)
- ClientAliveInterval 300 + ClientAliveCountMax 2 (clean up dead sessions)
Already applied manually to the existing server via
/etc/ssh/sshd_config.d/99-hardening.conf.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The alert was firing with DatasourceNoData because zero 5xx responses
means the numerator returns empty (not 0). Fixed by:
- Adding `or vector(0)` so zero 5xx returns 0% instead of empty
- Setting `noDataState: OK` as a safety net
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Grafana file-based provisioning expects flat dashboard JSON, not the
API format with a "dashboard" wrapper. The 4 wrapped dashboards were
silently failing to load — only business.json (already flat) worked.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The type:dashboard field was causing Grafana to filter annotations
by dashboard scope. Our deploy annotations are global (no dashboardUID),
so the target.type:tags is the only filter needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The annotation config was missing the target block that tells
Grafana to filter by tags. Without it, the annotation query
was silently ignored.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The error rate formula (errors/total*100) produces NaN when there's
zero traffic (0/0). Grafana treats NaN as a firing condition.
Use clamp_min on the denominator to return 0% instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Planner's save-to-journal callback makes a cross-origin request to
trails.cool from planner.trails.cool. Add the Journal's origin to the
Planner's connect-src CSP directive.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The threshold expressions were missing the 'expression' field referencing
the query refId, causing 'no variable specified for refId C' errors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Panels:
- Stat cards: total users, routes, activities, active planner sessions
- Time series: cumulative user growth, route growth
- Bar charts: signups/day, routes/day, planner sessions/day
- Table: top 20 routes by distance
All queries use the grafana_reader PostgreSQL datasource.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Create grafana_reader role with SELECT-only access on all schemas
- Init script runs on postgres first boot (docker-entrypoint-initdb.d)
- Grafana PostgreSQL datasource provisioned with read-only credentials
- Enables SQL queries in dashboards (user count, routes, etc.)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Exporters:
- postgres_exporter: DB connections, transactions, cache hit ratio, query stats
- node_exporter: host CPU, memory, disk, network
- cAdvisor: per-container CPU and memory usage
PostgreSQL:
- Enable pg_stat_statements for query-level performance tracking
- Track index scans vs sequential scans, cache hit ratio
Dashboard (service-health.json):
- DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage
- Host: disk gauge, CPU, memory, network I/O, disk I/O
- BRouter: request latency p50/p95/p99, container CPU + memory
- All containers: CPU and memory comparison
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Secrets:
- Add .sops.yaml with age encryption config
- Add encrypted secrets.app.env (app secrets) and secrets.infra.env (Grafana OAuth)
- CD decrypts at deploy time with AGE_SECRET_KEY — all other secrets
move out of GitHub Actions into version-controlled encrypted files
Split CD:
- cd-apps.yml: triggered by apps/packages changes, builds Docker images, deploys apps
- cd-infra.yml: triggered by infrastructure/ changes, copies configs, restarts services
- Remove monolithic cd.yml
Grafana auth:
- GitHub OAuth (trails-cool org), disable login form
- Remove Caddy basic_auth block and all GRAFANA_* env vars/secrets
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Alert rules reference datasourceUid: prometheus/loki which requires
explicit UIDs in the provisioning config.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Grafana provisioning YAML doesn't support env var substitution.
Replace ${ALERT_EMAIL} with hardcoded admin@trails.cool.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add DOMAIN, GRAFANA_USER, GRAFANA_PASSWORD_HASH as environment vars
on the Caddy container so Caddyfile can reference them
- Fix Loki compactor: add delete_request_store required when retention is enabled
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add wildcard DNS record for *.internal.trails.cool (A + AAAA) so
internal services don't leak hostnames in individual DNS entries.
Move Grafana from grafana.trails.cool to grafana.internal.trails.cool.
Future internal services just need a Caddy block — no DNS changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Health endpoints:
- /api/health (Journal) and /health (Planner) with DB connectivity check
- Docker healthchecks updated to use app health endpoints
Structured logging:
- Pino with JSON output in production, pretty-print in dev
- Request logging middleware in Planner (method, path, status, duration)
- Replaced console.log/error with structured logger in email and auth flows
Prometheus metrics:
- prom-client with default Node.js metrics + custom histograms/gauges
- /metrics endpoints on both apps
- http_request_duration, planner_active_sessions, brouter_request_duration
Monitoring stack:
- Prometheus, Loki, Grafana containers in docker-compose
- Grafana provisioned with datasources, dashboards, and alert rules
- Caddy access logging (JSON to stdout for Loki)
- grafana.trails.cool with basic auth via Caddy
Dashboards and alerting:
- Overview: request rate, error rate, latency p50/p95/p99
- Planner: active sessions, connected clients, BRouter latency
- Infrastructure: memory, CPU, event loop lag
- Alerts: disk >80%, app down 2min, error rate >5%
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Transactional emails:
- Add nodemailer SMTP email module with dev-mode console logging
- Magic link template and welcome template with HTML + plain text
- Wire sendMagicLink into login flow, sendWelcome into registration
- Update privacy page and deploy docs for SMTP configuration
Planner features:
- No-go areas: draw polygons on map (leaflet-geoman), synced via Yjs,
passed to BRouter as nogos parameter, route recomputes on change
- Session notes: collaborative Y.Text textarea in sidebar tab
- Crash recovery: periodic localStorage save of Yjs state, restore on reconnect
- Rate limit session creation (10/IP/hour) in /new route
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
y-websocket creates a Web Worker from a blob URL. The CSP script-src
needs blob: and an explicit worker-src directive to allow this.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MX, SPF, DMARC, DKIM (4 keys), mail A record, and SRV records for
autodiscover, IMAP, JMAP, CalDAV, CardDAV, SMTP submission.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- @sentry/vite-plugin uploads source maps during Docker build
- Release tagged with git SHA (SENTRY_RELEASE) on both client and server
- Environment set to production/development for filtering
- CD passes SENTRY_AUTH_TOKEN + SENTRY_RELEASE as Docker build args
- docker-compose passes SENTRY_RELEASE to runtime containers
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CD workflow:
- Uncommented deploy step
- Copy docker-compose.yml + Caddyfile to server via SCP
- Login to ghcr.io with DEPLOY_GHCR_TOKEN (classic PAT, read:packages)
- Download Germany BRouter segments on first deploy (~750MB)
- Pull images, restart services, verify
Infrastructure:
- BRouter segments as bind mount (./segments) not Docker volume
- Added ORIGIN, PLANNER_URL, SESSION_SECRET, NODE_ENV to Journal
- Added NODE_ENV to Planner
- Disabled Garage for now (no media storage yet)
Required secrets: DEPLOY_HOST, DEPLOY_SSH_KEY, DEPLOY_GHCR_TOKEN
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- trails.cool and planner.trails.cool A records pointing to server
- Using hcloud_zone_rrset (native hcloud provider, no separate DNS provider)
- Bumped hcloud provider ~> 1.45 to ~> 1.60
- Server type updated to cx23
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Terraform:
- Hetzner CX22 server with Docker, firewall (80/443/22)
- SSH key and output for server IP
Docker Compose:
- Caddy reverse proxy (auto HTTPS)
- Journal and Planner app containers
- BRouter routing engine container
- PostgreSQL + PostGIS
- Garage S3 storage
BRouter:
- Dockerfile based on eclipse-temurin:11-jre
- Segment download script for Germany (4 tiles, ~750MB)
CD Pipeline:
- Build & push Docker images to ghcr.io on main push
- Deploy step commented out (enable when server provisioned)
Scripts:
- PostgreSQL daily backup with 14-day retention
- Dockerfiles for both apps (multi-stage builds)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
