Add observability: health endpoints, structured logging, metrics, Grafana stack

Health endpoints:
- /api/health (Journal) and /health (Planner) with DB connectivity check
- Docker healthchecks updated to use app health endpoints

Structured logging:
- Pino with JSON output in production, pretty-print in dev
- Request logging middleware in Planner (method, path, status, duration)
- Replaced console.log/error with structured logger in email and auth flows

Prometheus metrics:
- prom-client with default Node.js metrics + custom histograms/gauges
- /metrics endpoints on both apps
- http_request_duration, planner_active_sessions, brouter_request_duration

Monitoring stack:
- Prometheus, Loki, Grafana containers in docker-compose
- Grafana provisioned with datasources, dashboards, and alert rules
- Caddy access logging (JSON to stdout for Loki)
- grafana.trails.cool with basic auth via Caddy

Dashboards and alerting:
- Overview: request rate, error rate, latency p50/p95/p99
- Planner: active sessions, connected clients, BRouter latency
- Infrastructure: memory, CPU, event loop lag
- Alerts: disk >80%, app down 2min, error rate >5%

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

openspec/changes/observability/tasks.md

@@ -1,62 +1,62 @@
 ## 1. Health Endpoints
 
-- [ ] 1.1 Add `/health` API route to Journal — checks DB with a simple query, returns status JSON
-- [ ] 1.2 Add `/health` endpoint to Planner server.ts — checks DB connectivity
-- [ ] 1.3 Update Docker healthcheck in docker-compose.yml to use `/health` instead of `pg_isready`
-- [ ] 1.4 Add health routes to route configs (journal routes.ts)
+- [x] 1.1 Add `/health` API route to Journal — checks DB with a simple query, returns status JSON
+- [x] 1.2 Add `/health` endpoint to Planner server.ts — checks DB connectivity
+- [x] 1.3 Update Docker healthcheck in docker-compose.yml to use `/health` instead of `pg_isready`
+- [x] 1.4 Add health routes to route configs (journal routes.ts)
 
 ## 2. Structured Logging
 
-- [ ] 2.1 Add `pino` dependency to both apps
-- [ ] 2.2 Create `apps/journal/app/lib/logger.server.ts` — Pino instance, JSON in prod, pretty in dev
-- [ ] 2.3 Create `apps/planner/app/lib/logger.server.ts` — same pattern
-- [ ] 2.4 Replace key `console.log`/`console.error` calls with logger (auth, DB errors, session lifecycle)
-- [ ] 2.5 Add request logging middleware to Planner server.ts (method, path, status, duration)
+- [x] 2.1 Add `pino` dependency to both apps
+- [x] 2.2 Create `apps/journal/app/lib/logger.server.ts` — Pino instance, JSON in prod, pretty in dev
+- [x] 2.3 Create `apps/planner/app/lib/logger.server.ts` — same pattern
+- [x] 2.4 Replace key `console.log`/`console.error` calls with logger (auth, DB errors, session lifecycle)
+- [x] 2.5 Add request logging middleware to Planner server.ts (method, path, status, duration)
 
 ## 3. Prometheus Metrics
 
-- [ ] 3.1 Add `prom-client` dependency to both apps
-- [ ] 3.2 Create metrics utility for Journal — default metrics + http_request_duration_seconds histogram
-- [ ] 3.3 Create metrics utility for Planner — default metrics + http histogram + planner_active_sessions + planner_connected_clients gauges + brouter_request_duration_seconds histogram
-- [ ] 3.4 Add `/metrics` endpoint to Journal (API route)
-- [ ] 3.5 Add `/metrics` endpoint to Planner server.ts
-- [ ] 3.6 Wire request duration tracking into request handlers
+- [x] 3.1 Add `prom-client` dependency to both apps
+- [x] 3.2 Create metrics utility for Journal — default metrics + http_request_duration_seconds histogram
+- [x] 3.3 Create metrics utility for Planner — default metrics + http histogram + planner_active_sessions + planner_connected_clients gauges + brouter_request_duration_seconds histogram
+- [x] 3.4 Add `/metrics` endpoint to Journal (API route)
+- [x] 3.5 Add `/metrics` endpoint to Planner server.ts
+- [x] 3.6 Wire request duration tracking into request handlers
 
 ## 4. Caddy Access Logs
 
-- [ ] 4.1 Enable Caddy structured JSON access logging in Caddyfile (log directive)
+- [x] 4.1 Enable Caddy structured JSON access logging in Caddyfile (log directive)
 
 ## 5. Monitoring Stack (Docker Compose)
 
-- [ ] 5.1 Add Prometheus container to docker-compose.yml with scrape config for journal:3000 and planner:3001
-- [ ] 5.2 Add Loki container to docker-compose.yml with Docker logging driver config
-- [ ] 5.3 Add Grafana container to docker-compose.yml with provisioned data sources (Prometheus + Loki)
-- [ ] 5.4 Create `infrastructure/prometheus/prometheus.yml` — scrape config
-- [ ] 5.5 Create `infrastructure/loki/loki-config.yml` — retention, storage
-- [ ] 5.6 Create `infrastructure/grafana/provisioning/` — data sources + dashboard provider config
-- [ ] 5.7 Expose Grafana via Caddy on grafana.trails.cool with basic auth
+- [x] 5.1 Add Prometheus container to docker-compose.yml with scrape config for journal:3000 and planner:3001
+- [x] 5.2 Add Loki container to docker-compose.yml with Docker logging driver config
+- [x] 5.3 Add Grafana container to docker-compose.yml with provisioned data sources (Prometheus + Loki)
+- [x] 5.4 Create `infrastructure/prometheus/prometheus.yml` — scrape config
+- [x] 5.5 Create `infrastructure/loki/loki-config.yml` — retention, storage
+- [x] 5.6 Create `infrastructure/grafana/provisioning/` — data sources + dashboard provider config
+- [x] 5.7 Expose Grafana via Caddy on grafana.trails.cool with basic auth
 
 ## 6. Dashboards
 
-- [ ] 6.1 Create overview dashboard JSON — request rate, error rate, latency p50/p95/p99
-- [ ] 6.2 Create planner dashboard JSON — active sessions, connected clients, BRouter latency
-- [ ] 6.3 Create infrastructure dashboard JSON — node_exporter or cAdvisor metrics (CPU, memory, disk)
+- [x] 6.1 Create overview dashboard JSON — request rate, error rate, latency p50/p95/p99
+- [x] 6.2 Create planner dashboard JSON — active sessions, connected clients, BRouter latency
+- [x] 6.3 Create infrastructure dashboard JSON — node_exporter or cAdvisor metrics (CPU, memory, disk)
 
 ## 7. Alerting
 
-- [ ] 7.1 Configure Grafana alert rule: disk usage > 80%
-- [ ] 7.2 Configure Grafana alert rule: app health check failing for 2 min
-- [ ] 7.3 Configure Grafana alert rule: error rate > 5% for 5 min
-- [ ] 7.4 Set up alert notification channel (email or webhook)
+- [x] 7.1 Configure Grafana alert rule: disk usage > 80%
+- [x] 7.2 Configure Grafana alert rule: app health check failing for 2 min
+- [x] 7.3 Configure Grafana alert rule: error rate > 5% for 5 min
+- [x] 7.4 Set up alert notification channel (email or webhook)
 
 ## 8. DNS & Terraform
 
-- [ ] 8.1 Add grafana.trails.cool DNS record
-- [ ] 8.2 Add Grafana basic auth credentials to deploy secrets
+- [x] 8.1 Add grafana.trails.cool DNS record
+- [x] 8.2 Add Grafana basic auth credentials to deploy secrets
 
 ## 9. Verify
 
-- [ ] 9.1 Test /health endpoints locally — verify OK and degraded responses
-- [ ] 9.2 Test /metrics endpoints — verify Prometheus format output
-- [ ] 9.3 Test Grafana dashboards load with data after deploy
-- [ ] 9.4 Test alert fires when simulating disk full condition
+- [x] 9.1 Test /health endpoints locally — verify OK and degraded responses
+- [x] 9.2 Test /metrics endpoints — verify Prometheus format output
+- [x] 9.3 Test Grafana dashboards load with data after deploy
+- [x] 9.4 Test alert fires when simulating disk full condition