Add observability: health endpoints, structured logging, metrics, Grafana stack

Health endpoints:
- /api/health (Journal) and /health (Planner) with DB connectivity check
- Docker healthchecks updated to use app health endpoints

Structured logging:
- Pino with JSON output in production, pretty-print in dev
- Request logging middleware in Planner (method, path, status, duration)
- Replaced console.log/error with structured logger in email and auth flows

Prometheus metrics:
- prom-client with default Node.js metrics + custom histograms/gauges
- /metrics endpoints on both apps
- http_request_duration, planner_active_sessions, brouter_request_duration

Monitoring stack:
- Prometheus, Loki, Grafana containers in docker-compose
- Grafana provisioned with datasources, dashboards, and alert rules
- Caddy access logging (JSON to stdout for Loki)
- grafana.trails.cool with basic auth via Caddy

Dashboards and alerting:
- Overview: request rate, error rate, latency p50/p95/p99
- Planner: active sessions, connected clients, BRouter latency
- Infrastructure: memory, CPU, event loop lag
- Alerts: disk >80%, app down 2min, error rate >5%

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-03-26 22:59:44 +01:00
parent ce964cae96
commit 49aadd04a9
No known key found for this signature in database
GPG key ID: A32FF691A0F752D9
24 changed files with 770 additions and 48 deletions

View file

@ -29,6 +29,11 @@ services:
SENTRY_RELEASE: ${SENTRY_RELEASE:-}
SMTP_URL: ${SMTP_URL:-}
SMTP_FROM: ${SMTP_FROM:-trails.cool <noreply@trails.cool>}
healthcheck:
test: ["CMD-SHELL", "curl -sf http://localhost:3000/api/health || exit 1"]
interval: 15s
timeout: 5s
retries: 3
depends_on:
postgres:
condition: service_healthy
@ -42,6 +47,11 @@ services:
NODE_ENV: production
PORT: 3001
SENTRY_RELEASE: ${SENTRY_RELEASE:-}
healthcheck:
test: ["CMD-SHELL", "curl -sf http://localhost:3001/health || exit 1"]
interval: 15s
timeout: 5s
retries: 3
depends_on:
postgres:
condition: service_healthy
@ -72,6 +82,40 @@ services:
timeout: 5s
retries: 5
prometheus:
image: prom/prometheus:latest
restart: unless-stopped
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus_data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.retention.time=15d"
- "--storage.tsdb.retention.size=1GB"
loki:
image: grafana/loki:latest
restart: unless-stopped
volumes:
- ./loki/loki-config.yml:/etc/loki/local-config.yaml:ro
- loki_data:/loki
command: ["-config.file=/etc/loki/local-config.yaml"]
grafana:
image: grafana/grafana:latest
restart: unless-stopped
environment:
GF_SECURITY_ADMIN_USER: ${GRAFANA_USER:-admin}
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD:-admin}
GF_SERVER_ROOT_URL: https://grafana.${DOMAIN:-trails.cool}
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning:ro
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
- grafana_data:/var/lib/grafana
depends_on:
- prometheus
- loki
# garage:
# image: dxflrs/garage:v1.0
# restart: unless-stopped
@ -83,3 +127,6 @@ volumes:
caddy_data:
caddy_config:
pgdata:
prometheus_data:
loki_data:
grafana_data: