Task group 4 of federation-hardening.
4.1 — FEDERATION.md at the repo root: actor discovery (WebFinger, actor,
NodeInfo), object/activity types with real JSON examples (Note, Create,
Delete, the narrow follow-graph inbox), addressing, HTTP-Signature
expectations, the two-layer dedup contract, durable delivery/retry
policy, and blocklist moderation semantics — precise enough for another
implementation to interoperate. Linked from README and docs/architecture.
4.2 — three prom-client metrics + a journal dashboard row:
- `federation_delivery_total{outcome}` — incremented in deliver-activity
(delivered/skipped/failed).
- `federation_inbox_dropped_total{reason}` — incremented at every inbox
drop (duplicate | blocked); this is the counter deferred from task 3.2.
- `federation_queue_depth` — gauge sampled at scrape time in
/api/metrics from PgBossMessageQueue.getDepth(); the restart-loss
regression detector.
Grafana journal.json gains a Federation row (delivery rate, queue depth,
inbox drops); the logs panels shift down to make room.
Verified: dashboard JSON valid; journal typecheck + lint clean; unit
suite 357 passing (route-template guard unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- planner dashboard: replace Overpass panels with POI index freshness +
serving panels (age, rows/category, import status, API request rate/errors)
- alerts: replace overpass-upstream-unhealthy with poi-index-stale (>6 weeks)
- architecture.md: POI data flow now the self-hosted index
- privacy manifest (DE+EN): Overpass is Journal-surface-backfill-only;
Planner POIs served same-origin from /api/pois
- self-host-overpass README + roadmap: superseded for POIs by poi-index
- poi-extract README: self-hoster story (optional pipeline, regional extract,
graceful empty index)
- map-core tsconfig: exclude *.sync.test.ts from tsc to keep it zero-dep
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the visibility that would have caught the corrupt-WAL incident,
plus a direct Overpass upstream alert.
- prometheus.yml: self-scrape Prometheus (localhost:9090) and Loki
(loki:3100). Prometheus scraped everything except itself, so TSDB
health (compaction failures, WAL corruption, head series, retention
deletions) was invisible.
- monitoring-health.json: new "Monitoring Health" dashboard — TSDB
compaction/WAL failures, retention deletions/hour, head series,
samples/s, block bytes vs size-retention limit, retention depth
(oldest-sample age), Loki ingestion rate + memory chunks.
- alerts.yml: prometheus-compaction-failing (any compaction failure or
WAL corruption in 1h) and overpass-upstream-unhealthy (>20% upstream
failure over 10m — sustained public-Overpass degradation, distinct
from the symptom-level Caddy-502 alert).
Validated: promtool check config, YAML parse, JSON parse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
overview.json was missing a comma between the "Health Status" and
"App Error Rate (from logs)" panel objects, making the whole file
invalid JSON. Grafana's file provisioner skips dashboards that fail to
parse, so the entire "trails.cool Overview" dashboard (5 panels) never
loaded — a silent gap in observability. Validated with json.load.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The journal/planner deploy in cd-apps.yml does `docker compose up -d
journal planner`, which stops the old container and starts the new
one — Caddy keeps forwarding requests during the ~10–30s gap and
returns 502s. The caddy-502-rate alert (threshold > 0 for 2m)
correctly trips, every time.
Two production changes plus a long-broken workflow detail:
- infrastructure/Caddyfile — add `lb_try_duration 30s` /
`lb_try_interval 250ms` to the journal and planner reverse_proxy
blocks. Caddy now holds and retries the upstream for up to 30s
during a restart instead of 502'ing immediately. Real outages
(upstream unreachable longer than 30s) still 502 and the alert
still fires for those.
- infrastructure/grafana/provisioning/alerting/alerts.yml — add a
comment documenting why caddy-502-rate stays at threshold > 0:
with lb_try_duration in front of it, the alert no longer
conflates "deploy in flight" with "real outage."
- .github/workflows/cd-apps.yml — fix a long-silent bug: the
Grafana deploy-annotation step was reading
GRAFANA_SERVICE_TOKEN from `.env`, but the secrets file we scp
to /opt/trails-cool is named `app.env`. The token check failed
silently and the curl was being skipped on every deploy.
Switching to `app.env` so deploys actually annotate Grafana.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The notification policy and contact points live in
`infrastructure/grafana/provisioning/alerting/alerts.yml`, so
UI-created contact points can't be attached to provisioned alerts.
Provision the Pushover integration instead.
Rather than add a second contact point and do the fan-out in the
notification tree (Grafana's tree only fires one receiver per match
without duplication workarounds), put both email + pushover under a
single `default` contact point. Every alert now fans out to both.
Secrets go to SOPS `secrets.infra.env`. The Grafana container reads
`PUSHOVER_API_TOKEN` and `PUSHOVER_USER_KEY_ULLRICH` from env; the
provisioning YAML references them via `$VAR` and Grafana substitutes
at startup, so no secret lands in git.
Priority 1 on firing (bypass quiet hours), 0 on resolve. User can
tweak in the YAML later.
Post-deploy: delete the UI-created "Pushover Ullrich" contact point
in Grafana to avoid a duplicate. It's unused (no policy references it)
but shows up in the contact-point list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands sections 6 and 8 of relocate-brouter-to-dedicated-host on top
of the host compose in #291.
## Observability (section 6)
- **Prometheus**: new `brouter-cadvisor` job scraping
`10.0.1.10:8080` over the vSwitch, labeled `host="brouter"`.
- **cAdvisor sidecar** on the dedicated host's compose
(`--docker_only --whitelisted_container_labels=trails.cool.service`)
so metrics only cover trails containers, never the operator's
unrelated workloads on the shared box.
- **Promtail sidecar** on the dedicated host, Docker SD with relabel-
drop on missing `trails.cool.service` label, pushing to flagship
Loki at `http://10.0.0.2:3100/loki/api/v1/push`.
- **Flagship compose**: Loki now publishes port 3100 on the vSwitch
IP only (10.0.0.2:3100) — Hetzner Cloud firewall blocks it from
the public internet.
- **Grafana dashboard**: `brouter.json` — scrape up/down, request
rate (from Planner-side `brouter_request_duration_seconds`),
p50/p95/p99, container memory/CPU, Loki logs panel.
- **Alert**: `brouter-scrape-down` fires on
`up{job="brouter-cadvisor"} < 1 for 2m`; `noDataState: Alerting`
so a total scrape failure still pages.
Operator needs one UFW rule on the dedicated host for the cAdvisor
port — documented in `infrastructure/brouter-host/README.md`.
## Documentation (section 8)
- `CLAUDE.md` — hosts table + updated deployment table with SSH
targets per workflow; BRouter host SSH is `-p 2232 trails@...`,
different key.
- `docs/architecture.md` — Hosting section rewritten to cover both
hosts, vSwitch boundary, and the observability-scoping rationale
for the shared dedicated host.
- `docs/deployment.md` (new) — full operator runbook: host layout,
first-time BRouter provisioning, SOPS rotation (including the
macOS `SOPS_AGE_KEY_FILE` gotcha), cutover procedure with
rollback, manual workflow triggers.
Task 8.4 (infrastructure/README.md) skipped: that file doesn't
exist and the ground is covered by brouter-host/README.md +
docs/deployment.md.
## Validation
- `docker compose config` on both the flagship and brouter-host
compose files — both validate.
- `pnpm typecheck`, `pnpm lint`, `pnpm test` — all clean (full
turbo cache hits; no code changes in this commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pg-boss v11 renamed every concatenated-lowercase timestamp column
(`createdon`, `completedon`) to snake_case (`created_on`,
`completed_on`). Our service-health dashboard and failed-job alert
still referenced the v10 names, so after the v12 upgrade those
queries error out silently.
Fix the three affected queries:
- alerts.yml: failed-job alert
- service-health.json: "Recent Failed Jobs" table
- service-health.json: "Job Queue — Completed/hour" time series
The `state` / `name` / `output` columns stay the same, and the `state`
enum still accepts `'failed'` / `'completed'` literals.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Nine panels at trails-demo-bot covering:
- Current synthetic route/activity counts + time-since-last-walk
- Gauge time series over the configured retention window
- Walks per day + distance distribution
- Loki log stream filtered to `demo-bot` messages
- Recent walks table with name, distance, ascent
Provisioned via the existing grafana/dashboards directory — no extra
wiring needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original formulas used clamp_min(denominator, 1) to avoid divide-
by-zero, but that inflates the denominator when request rate is < 1/s.
Real example from prod: ~0.04 req/s, all successful, displayed as
1.02% health (red) instead of ~100%.
Switch to (numerator or vector(0)) / denominator:
- Missing "hit" series (no cache hits yet) → numerator resolves to 0
instead of empty set, so the stat shows 0% instead of "No data".
- Zero traffic (empty denominator) → whole expr is empty, Grafana
renders "No data", which is the correct behaviour for a health
stat when nothing's happening.
- Non-zero low traffic → ratio is true ratio, not artificially
depressed by the clamp.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds four metrics surfaced via the existing /metrics endpoint:
- overpass_cache_events_total{result=hit|miss|coalesced}
- overpass_cache_size (gauge)
- overpass_upstream_duration_seconds (histogram)
- overpass_upstream_requests_total{status} — covers 2xx/4xx/5xx and
an explicit "error" label for network-level failures
Grafana dashboards/planner.json gets a new row:
- Upstream health (5m 2xx success ratio, red <90%, green >99%)
- Cache hit ratio
- Cache size
- Upstream p95 latency
Plus timeseries panels for cache event breakdown, upstream status
over time, and p50/p95/p99 upstream latency.
The success-rate formula uses clamp_min(..., 1) on the denominator so
it doesn't NaN when traffic is zero.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add @trails-cool/jobs package wrapping pg-boss for durable background job
execution using the existing PostgreSQL database. Wire up planner session
expiry as the first scheduled job (hourly, 7-day TTL) — this was previously
dead code that never ran. Add placeholder worker to journal for future
Komoot import and federation jobs.
Infrastructure: grant grafana_reader access to pgboss schema, add job queue
health panels to Service Health dashboard, add failed job alert.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix blind spot where Caddy 502 errors were invisible: the dashboards and
alerts queried caddy_http_response_duration_seconds (upstream responses only),
missing 502s that Caddy generates itself. Switch to
caddy_http_request_duration_seconds (server-level, all responses).
Add Journal Grafana dashboard with: 502 rate, response codes, request rate
by route, latency percentiles, container restarts/memory/CPU, Node.js event
loop lag and heap, and Loki log panels for errors and Caddy 5xx entries.
Add color coding to Caddy status code panel (green=2xx, blue=3xx, yellow=4xx,
red=5xx). Add log-based error rate panel to the overview dashboard.
New alerts: container restart loop, PostgreSQL connections > 80, application
crash log detection (Loki), and Caddy 502 rate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dashboard: switch error rate panel from app-level http_request_duration
(never observed) to Caddy's caddy_http_response_duration_seconds_count.
Journal: add custom server.ts (matching planner pattern) that observes
http_request_duration_seconds on every request. Replaces react-router-serve
with a custom Node HTTP server that handles /api/health, /api/metrics,
static assets, and request timing. Removes redundant React Router routes
for health and metrics.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Caddy Response Status Codes: caddy_http_responses_total doesn't exist,
use caddy_http_response_duration_seconds_count which has the code label
- pg_stat_statements: enable extension in init script, add custom queries
config for postgres-exporter, remove invalid datname filter from query
- Add postgres/queries.yml to cd-infra SCP sources
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The alert was firing with DatasourceNoData because zero 5xx responses
means the numerator returns empty (not 0). Fixed by:
- Adding `or vector(0)` so zero 5xx returns 0% instead of empty
- Setting `noDataState: OK` as a safety net
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Grafana file-based provisioning expects flat dashboard JSON, not the
API format with a "dashboard" wrapper. The 4 wrapped dashboards were
silently failing to load — only business.json (already flat) worked.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The type:dashboard field was causing Grafana to filter annotations
by dashboard scope. Our deploy annotations are global (no dashboardUID),
so the target.type:tags is the only filter needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The annotation config was missing the target block that tells
Grafana to filter by tags. Without it, the annotation query
was silently ignored.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The error rate formula (errors/total*100) produces NaN when there's
zero traffic (0/0). Grafana treats NaN as a firing condition.
Use clamp_min on the denominator to return 0% instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The threshold expressions were missing the 'expression' field referencing
the query refId, causing 'no variable specified for refId C' errors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Panels:
- Stat cards: total users, routes, activities, active planner sessions
- Time series: cumulative user growth, route growth
- Bar charts: signups/day, routes/day, planner sessions/day
- Table: top 20 routes by distance
All queries use the grafana_reader PostgreSQL datasource.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Create grafana_reader role with SELECT-only access on all schemas
- Init script runs on postgres first boot (docker-entrypoint-initdb.d)
- Grafana PostgreSQL datasource provisioned with read-only credentials
- Enables SQL queries in dashboards (user count, routes, etc.)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Exporters:
- postgres_exporter: DB connections, transactions, cache hit ratio, query stats
- node_exporter: host CPU, memory, disk, network
- cAdvisor: per-container CPU and memory usage
PostgreSQL:
- Enable pg_stat_statements for query-level performance tracking
- Track index scans vs sequential scans, cache hit ratio
Dashboard (service-health.json):
- DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage
- Host: disk gauge, CPU, memory, network I/O, disk I/O
- BRouter: request latency p50/p95/p99, container CPU + memory
- All containers: CPU and memory comparison
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Alert rules reference datasourceUid: prometheus/loki which requires
explicit UIDs in the provisioning config.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Grafana provisioning YAML doesn't support env var substitution.
Replace ${ALERT_EMAIL} with hardcoded admin@trails.cool.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Health endpoints:
- /api/health (Journal) and /health (Planner) with DB connectivity check
- Docker healthchecks updated to use app health endpoints
Structured logging:
- Pino with JSON output in production, pretty-print in dev
- Request logging middleware in Planner (method, path, status, duration)
- Replaced console.log/error with structured logger in email and auth flows
Prometheus metrics:
- prom-client with default Node.js metrics + custom histograms/gauges
- /metrics endpoints on both apps
- http_request_duration, planner_active_sessions, brouter_request_duration
Monitoring stack:
- Prometheus, Loki, Grafana containers in docker-compose
- Grafana provisioned with datasources, dashboards, and alert rules
- Caddy access logging (JSON to stdout for Loki)
- grafana.trails.cool with basic auth via Caddy
Dashboards and alerting:
- Overview: request rate, error rate, latency p50/p95/p99
- Planner: active sessions, connected clients, BRouter latency
- Infrastructure: memory, CPU, event loop lag
- Alerts: disk >80%, app down 2min, error rate >5%
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
