The notification policy and contact points live in
`infrastructure/grafana/provisioning/alerting/alerts.yml`, so
UI-created contact points can't be attached to provisioned alerts.
Provision the Pushover integration instead.
Rather than add a second contact point and do the fan-out in the
notification tree (Grafana's tree only fires one receiver per match
without duplication workarounds), put both email + pushover under a
single `default` contact point. Every alert now fans out to both.
Secrets go to SOPS `secrets.infra.env`. The Grafana container reads
`PUSHOVER_API_TOKEN` and `PUSHOVER_USER_KEY_ULLRICH` from env; the
provisioning YAML references them via `$VAR` and Grafana substitutes
at startup, so no secret lands in git.
Priority 1 on firing (bypass quiet hours), 0 on resolve. User can
tweak in the YAML later.
Post-deploy: delete the UI-created "Pushover Ullrich" contact point
in Grafana to avoid a duplicate. It's unused (no policy references it)
but shows up in the contact-point list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands sections 6 and 8 of relocate-brouter-to-dedicated-host on top
of the host compose in #291.
## Observability (section 6)
- **Prometheus**: new `brouter-cadvisor` job scraping
`10.0.1.10:8080` over the vSwitch, labeled `host="brouter"`.
- **cAdvisor sidecar** on the dedicated host's compose
(`--docker_only --whitelisted_container_labels=trails.cool.service`)
so metrics only cover trails containers, never the operator's
unrelated workloads on the shared box.
- **Promtail sidecar** on the dedicated host, Docker SD with relabel-
drop on missing `trails.cool.service` label, pushing to flagship
Loki at `http://10.0.0.2:3100/loki/api/v1/push`.
- **Flagship compose**: Loki now publishes port 3100 on the vSwitch
IP only (10.0.0.2:3100) — Hetzner Cloud firewall blocks it from
the public internet.
- **Grafana dashboard**: `brouter.json` — scrape up/down, request
rate (from Planner-side `brouter_request_duration_seconds`),
p50/p95/p99, container memory/CPU, Loki logs panel.
- **Alert**: `brouter-scrape-down` fires on
`up{job="brouter-cadvisor"} < 1 for 2m`; `noDataState: Alerting`
so a total scrape failure still pages.
Operator needs one UFW rule on the dedicated host for the cAdvisor
port — documented in `infrastructure/brouter-host/README.md`.
## Documentation (section 8)
- `CLAUDE.md` — hosts table + updated deployment table with SSH
targets per workflow; BRouter host SSH is `-p 2232 trails@...`,
different key.
- `docs/architecture.md` — Hosting section rewritten to cover both
hosts, vSwitch boundary, and the observability-scoping rationale
for the shared dedicated host.
- `docs/deployment.md` (new) — full operator runbook: host layout,
first-time BRouter provisioning, SOPS rotation (including the
macOS `SOPS_AGE_KEY_FILE` gotcha), cutover procedure with
rollback, manual workflow triggers.
Task 8.4 (infrastructure/README.md) skipped: that file doesn't
exist and the ground is covered by brouter-host/README.md +
docs/deployment.md.
## Validation
- `docker compose config` on both the flagship and brouter-host
compose files — both validate.
- `pnpm typecheck`, `pnpm lint`, `pnpm test` — all clean (full
turbo cache hits; no code changes in this commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pg-boss v11 renamed every concatenated-lowercase timestamp column
(`createdon`, `completedon`) to snake_case (`created_on`,
`completed_on`). Our service-health dashboard and failed-job alert
still referenced the v10 names, so after the v12 upgrade those
queries error out silently.
Fix the three affected queries:
- alerts.yml: failed-job alert
- service-health.json: "Recent Failed Jobs" table
- service-health.json: "Job Queue — Completed/hour" time series
The `state` / `name` / `output` columns stay the same, and the `state`
enum still accepts `'failed'` / `'completed'` literals.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Nine panels at trails-demo-bot covering:
- Current synthetic route/activity counts + time-since-last-walk
- Gauge time series over the configured retention window
- Walks per day + distance distribution
- Loki log stream filtered to `demo-bot` messages
- Recent walks table with name, distance, ascent
Provisioned via the existing grafana/dashboards directory — no extra
wiring needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original formulas used clamp_min(denominator, 1) to avoid divide-
by-zero, but that inflates the denominator when request rate is < 1/s.
Real example from prod: ~0.04 req/s, all successful, displayed as
1.02% health (red) instead of ~100%.
Switch to (numerator or vector(0)) / denominator:
- Missing "hit" series (no cache hits yet) → numerator resolves to 0
instead of empty set, so the stat shows 0% instead of "No data".
- Zero traffic (empty denominator) → whole expr is empty, Grafana
renders "No data", which is the correct behaviour for a health
stat when nothing's happening.
- Non-zero low traffic → ratio is true ratio, not artificially
depressed by the clamp.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds four metrics surfaced via the existing /metrics endpoint:
- overpass_cache_events_total{result=hit|miss|coalesced}
- overpass_cache_size (gauge)
- overpass_upstream_duration_seconds (histogram)
- overpass_upstream_requests_total{status} — covers 2xx/4xx/5xx and
an explicit "error" label for network-level failures
Grafana dashboards/planner.json gets a new row:
- Upstream health (5m 2xx success ratio, red <90%, green >99%)
- Cache hit ratio
- Cache size
- Upstream p95 latency
Plus timeseries panels for cache event breakdown, upstream status
over time, and p50/p95/p99 upstream latency.
The success-rate formula uses clamp_min(..., 1) on the denominator so
it doesn't NaN when traffic is zero.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add @trails-cool/jobs package wrapping pg-boss for durable background job
execution using the existing PostgreSQL database. Wire up planner session
expiry as the first scheduled job (hourly, 7-day TTL) — this was previously
dead code that never ran. Add placeholder worker to journal for future
Komoot import and federation jobs.
Infrastructure: grant grafana_reader access to pgboss schema, add job queue
health panels to Service Health dashboard, add failed job alert.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix blind spot where Caddy 502 errors were invisible: the dashboards and
alerts queried caddy_http_response_duration_seconds (upstream responses only),
missing 502s that Caddy generates itself. Switch to
caddy_http_request_duration_seconds (server-level, all responses).
Add Journal Grafana dashboard with: 502 rate, response codes, request rate
by route, latency percentiles, container restarts/memory/CPU, Node.js event
loop lag and heap, and Loki log panels for errors and Caddy 5xx entries.
Add color coding to Caddy status code panel (green=2xx, blue=3xx, yellow=4xx,
red=5xx). Add log-based error rate panel to the overview dashboard.
New alerts: container restart loop, PostgreSQL connections > 80, application
crash log detection (Loki), and Caddy 502 rate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dashboard: switch error rate panel from app-level http_request_duration
(never observed) to Caddy's caddy_http_response_duration_seconds_count.
Journal: add custom server.ts (matching planner pattern) that observes
http_request_duration_seconds on every request. Replaces react-router-serve
with a custom Node HTTP server that handles /api/health, /api/metrics,
static assets, and request timing. Removes redundant React Router routes
for health and metrics.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Caddy Response Status Codes: caddy_http_responses_total doesn't exist,
use caddy_http_response_duration_seconds_count which has the code label
- pg_stat_statements: enable extension in init script, add custom queries
config for postgres-exporter, remove invalid datname filter from query
- Add postgres/queries.yml to cd-infra SCP sources
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The alert was firing with DatasourceNoData because zero 5xx responses
means the numerator returns empty (not 0). Fixed by:
- Adding `or vector(0)` so zero 5xx returns 0% instead of empty
- Setting `noDataState: OK` as a safety net
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Grafana file-based provisioning expects flat dashboard JSON, not the
API format with a "dashboard" wrapper. The 4 wrapped dashboards were
silently failing to load — only business.json (already flat) worked.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The type:dashboard field was causing Grafana to filter annotations
by dashboard scope. Our deploy annotations are global (no dashboardUID),
so the target.type:tags is the only filter needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The annotation config was missing the target block that tells
Grafana to filter by tags. Without it, the annotation query
was silently ignored.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The error rate formula (errors/total*100) produces NaN when there's
zero traffic (0/0). Grafana treats NaN as a firing condition.
Use clamp_min on the denominator to return 0% instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The threshold expressions were missing the 'expression' field referencing
the query refId, causing 'no variable specified for refId C' errors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Panels:
- Stat cards: total users, routes, activities, active planner sessions
- Time series: cumulative user growth, route growth
- Bar charts: signups/day, routes/day, planner sessions/day
- Table: top 20 routes by distance
All queries use the grafana_reader PostgreSQL datasource.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Create grafana_reader role with SELECT-only access on all schemas
- Init script runs on postgres first boot (docker-entrypoint-initdb.d)
- Grafana PostgreSQL datasource provisioned with read-only credentials
- Enables SQL queries in dashboards (user count, routes, etc.)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Exporters:
- postgres_exporter: DB connections, transactions, cache hit ratio, query stats
- node_exporter: host CPU, memory, disk, network
- cAdvisor: per-container CPU and memory usage
PostgreSQL:
- Enable pg_stat_statements for query-level performance tracking
- Track index scans vs sequential scans, cache hit ratio
Dashboard (service-health.json):
- DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage
- Host: disk gauge, CPU, memory, network I/O, disk I/O
- BRouter: request latency p50/p95/p99, container CPU + memory
- All containers: CPU and memory comparison
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Alert rules reference datasourceUid: prometheus/loki which requires
explicit UIDs in the provisioning config.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Grafana provisioning YAML doesn't support env var substitution.
Replace ${ALERT_EMAIL} with hardcoded admin@trails.cool.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Health endpoints:
- /api/health (Journal) and /health (Planner) with DB connectivity check
- Docker healthchecks updated to use app health endpoints
Structured logging:
- Pino with JSON output in production, pretty-print in dev
- Request logging middleware in Planner (method, path, status, duration)
- Replaced console.log/error with structured logger in email and auth flows
Prometheus metrics:
- prom-client with default Node.js metrics + custom histograms/gauges
- /metrics endpoints on both apps
- http_request_duration, planner_active_sessions, brouter_request_duration
Monitoring stack:
- Prometheus, Loki, Grafana containers in docker-compose
- Grafana provisioned with datasources, dashboards, and alert rules
- Caddy access logging (JSON to stdout for Loki)
- grafana.trails.cool with basic auth via Caddy
Dashboards and alerting:
- Overview: request rate, error rate, latency p50/p95/p99
- Planner: active sessions, connected clients, BRouter latency
- Infrastructure: memory, CPU, event loop lag
- Alerts: disk >80%, app down 2min, error rate >5%
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
