Add service health monitoring: postgres, node, cAdvisor exporters + dashboard

Exporters:
- postgres_exporter: DB connections, transactions, cache hit ratio, query stats
- node_exporter: host CPU, memory, disk, network
- cAdvisor: per-container CPU and memory usage

PostgreSQL:
- Enable pg_stat_statements for query-level performance tracking
- Track index scans vs sequential scans, cache hit ratio

Dashboard (service-health.json):
- DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage
- Host: disk gauge, CPU, memory, network I/O, disk I/O
- BRouter: request latency p50/p95/p99, container CPU + memory
- All containers: CPU and memory comparison

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

.github/workflows/cd-infra.yml

@@ -68,12 +68,19 @@ jobs:
             GHCR_TOKEN=$(grep DEPLOY_GHCR_TOKEN .env | cut -d= -f2-)
             echo "$GHCR_TOKEN" | docker login ghcr.io -u stigi --password-stdin 2>/dev/null || true
 
-            # Restart services
-            if [ "${{ github.event.inputs.restart_all }}" = "true" ]; then
+            # Check if full restart requested via:
+            # - workflow_dispatch input: restart_all=true
+            # - commit message containing [restart-all]
+            RESTART_ALL="${{ github.event.inputs.restart_all }}"
+            if [ "$RESTART_ALL" != "true" ] && echo "${{ github.event.head_commit.message }}" | grep -q '\[restart-all\]'; then
+              RESTART_ALL="true"
+            fi
+
+            if [ "$RESTART_ALL" = "true" ]; then
               docker compose --env-file .env up -d --remove-orphans
             else
               # Restart infra services (except Caddy — just reload its config)
-              docker compose --env-file .env up -d postgres prometheus loki grafana
+              docker compose --env-file .env up -d postgres prometheus loki grafana postgres-exporter node-exporter cadvisor
               docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile
             fi
 

CLAUDE.md

@@ -140,6 +140,32 @@ before pushing to an existing PR branch.
 ### Emergency override
 Admins can bypass the PR workflow when necessary (e.g., CI is broken and needs a hotfix). Document the reason in the commit message.
 
+## Deployment
+
+Three separate CD workflows triggered by path:
+
+| Workflow | Triggers on | Deploys |
+|----------|-------------|---------|
+| `cd-apps.yml` | `apps/`, `packages/`, `pnpm-lock.yaml` | journal, planner |
+| `cd-infra.yml` | `infrastructure/` | caddy, postgres, prometheus, loki, grafana, exporters |
+| `cd-brouter.yml` | `docker/brouter/` | brouter |
+
+### Secrets
+All secrets are stored in SOPS-encrypted files (`infrastructure/secrets.app.env`, `infrastructure/secrets.infra.env`). Edit with `sops infrastructure/secrets.app.env`. Only `AGE_SECRET_KEY`, `DEPLOY_SSH_KEY`, and `DEPLOY_HOST` remain as GitHub secrets.
+
+### Full restart
+To restart **all** containers (not just the ones a workflow normally touches), either:
+- Add `[restart-all]` to the commit message
+- Or trigger manually: `gh workflow run cd-infra.yml -f restart_all=true`
+
+### Server access
+```bash
+ssh -i ~/.ssh/trails-cool-deploy root@trails.cool
+```
+
+### Grafana
+`https://grafana.internal.trails.cool` — GitHub OAuth (trails-cool org)
+
 ## OpenSpec Workflow
 
 Specs live in `openspec/`. Use these slash commands:

infrastructure/Caddyfile

@@ -1,3 +1,10 @@
+{
+	servers {
+		metrics
+	}
+	admin 0.0.0.0:2019
+}
+
 (security_headers) {
 	header {
 		Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

infrastructure/docker-compose.yml

@@ -78,12 +78,52 @@ services:
       POSTGRES_DB: trails
     volumes:
       - pgdata:/var/lib/postgresql/data
+    command:
+      - "postgres"
+      - "-c"
+      - "shared_preload_libraries=pg_stat_statements"
+      - "-c"
+      - "pg_stat_statements.track=all"
     healthcheck:
       test: ["CMD-SHELL", "pg_isready -U trails"]
       interval: 5s
       timeout: 5s
       retries: 5
 
+  postgres-exporter:
+    image: prometheuscommunity/postgres-exporter:latest
+    restart: unless-stopped
+    environment:
+      DATA_SOURCE_NAME: postgresql://trails:${POSTGRES_PASSWORD:-trails}@postgres:5432/trails?sslmode=disable
+    depends_on:
+      postgres:
+        condition: service_healthy
+
+  node-exporter:
+    image: prom/node-exporter:latest
+    restart: unless-stopped
+    pid: host
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+    command:
+      - "--path.procfs=/host/proc"
+      - "--path.sysfs=/host/sys"
+      - "--path.rootfs=/rootfs"
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
+
+  cadvisor:
+    image: gcr.io/cadvisor/cadvisor:latest
+    restart: unless-stopped
+    privileged: true
+    volumes:
+      - /:/rootfs:ro
+      - /var/run:/var/run:ro
+      - /sys:/sys:ro
+      - /var/lib/docker/:/var/lib/docker:ro
+      - /dev/disk/:/dev/disk:ro
+
   prometheus:
     image: prom/prometheus:latest
     restart: unless-stopped

infrastructure/grafana/dashboards/service-health.json

@@ -0,0 +1,187 @@
+{
+  "dashboard": {
+    "title": "Service Health",
+    "uid": "trails-service-health",
+    "timezone": "browser",
+    "refresh": "30s",
+    "panels": [
+      {
+        "title": "PostgreSQL — Active Connections",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
+        "targets": [
+          { "expr": "pg_stat_activity_count{datname=\"trails\"}", "legendFormat": "{{state}}" }
+        ]
+      },
+      {
+        "title": "PostgreSQL — Database Size",
+        "type": "stat",
+        "gridPos": { "h": 8, "w": 6, "x": 12, "y": 0 },
+        "targets": [
+          { "expr": "pg_database_size_bytes{datname=\"trails\"}", "legendFormat": "trails" }
+        ],
+        "fieldConfig": { "defaults": { "unit": "bytes" } }
+      },
+      {
+        "title": "PostgreSQL — Transactions/sec",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 6, "x": 18, "y": 0 },
+        "targets": [
+          { "expr": "rate(pg_stat_database_xact_commit{datname=\"trails\"}[5m])", "legendFormat": "commits/s" },
+          { "expr": "rate(pg_stat_database_xact_rollback{datname=\"trails\"}[5m])", "legendFormat": "rollbacks/s" }
+        ]
+      },
+      {
+        "title": "PostgreSQL — Slowest Queries (pg_stat_statements)",
+        "type": "table",
+        "gridPos": { "h": 8, "w": 24, "x": 0, "y": 8 },
+        "targets": [
+          {
+            "expr": "topk(10, pg_stat_statements_mean_time_seconds{datname=\"trails\"})",
+            "legendFormat": "{{query}}",
+            "format": "table",
+            "instant": true
+          }
+        ]
+      },
+      {
+        "title": "PostgreSQL — Cache Hit Ratio",
+        "type": "gauge",
+        "gridPos": { "h": 8, "w": 6, "x": 0, "y": 16 },
+        "targets": [
+          { "expr": "pg_stat_database_blks_hit{datname=\"trails\"} / (pg_stat_database_blks_hit{datname=\"trails\"} + pg_stat_database_blks_read{datname=\"trails\"}) * 100", "legendFormat": "hit %" }
+        ],
+        "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100, "thresholds": { "steps": [{ "color": "red", "value": null }, { "color": "yellow", "value": 90 }, { "color": "green", "value": 99 }] } } }
+      },
+      {
+        "title": "PostgreSQL — Index Scans vs Sequential Scans",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 18, "x": 6, "y": 16 },
+        "targets": [
+          { "expr": "sum(rate(pg_stat_user_tables_idx_scan[5m]))", "legendFormat": "Index scans/s" },
+          { "expr": "sum(rate(pg_stat_user_tables_seq_scan[5m]))", "legendFormat": "Sequential scans/s" }
+        ]
+      },
+      {
+        "title": "Host — Disk Usage",
+        "type": "gauge",
+        "gridPos": { "h": 8, "w": 8, "x": 0, "y": 24 },
+        "targets": [
+          { "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"}) * 100", "legendFormat": "Used %" }
+        ],
+        "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100, "thresholds": { "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 70 }, { "color": "red", "value": 85 }] } } }
+      },
+      {
+        "title": "Host — CPU Usage",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 8, "x": 8, "y": 24 },
+        "targets": [
+          { "expr": "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)", "legendFormat": "CPU %" }
+        ],
+        "fieldConfig": { "defaults": { "unit": "percent", "max": 100 } }
+      },
+      {
+        "title": "Host — Memory Usage",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 8, "x": 16, "y": 24 },
+        "targets": [
+          { "expr": "node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes", "legendFormat": "Used" },
+          { "expr": "node_memory_MemAvailable_bytes", "legendFormat": "Available" }
+        ],
+        "fieldConfig": { "defaults": { "unit": "bytes" } }
+      },
+      {
+        "title": "Host — Network I/O",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 32 },
+        "targets": [
+          { "expr": "rate(node_network_receive_bytes_total{device!~\"lo|veth.*|br.*|docker.*\"}[5m])", "legendFormat": "{{device}} rx" },
+          { "expr": "-rate(node_network_transmit_bytes_total{device!~\"lo|veth.*|br.*|docker.*\"}[5m])", "legendFormat": "{{device}} tx" }
+        ],
+        "fieldConfig": { "defaults": { "unit": "Bps" } }
+      },
+      {
+        "title": "Host — Disk I/O",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 32 },
+        "targets": [
+          { "expr": "rate(node_disk_read_bytes_total[5m])", "legendFormat": "{{device}} read" },
+          { "expr": "-rate(node_disk_written_bytes_total[5m])", "legendFormat": "{{device}} write" }
+        ],
+        "fieldConfig": { "defaults": { "unit": "Bps" } }
+      },
+      {
+        "title": "BRouter — Request Latency (from Planner proxy)",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 40 },
+        "targets": [
+          { "expr": "histogram_quantile(0.50, rate(brouter_request_duration_seconds_bucket[5m]))", "legendFormat": "p50" },
+          { "expr": "histogram_quantile(0.95, rate(brouter_request_duration_seconds_bucket[5m]))", "legendFormat": "p95" },
+          { "expr": "histogram_quantile(0.99, rate(brouter_request_duration_seconds_bucket[5m]))", "legendFormat": "p99" }
+        ],
+        "fieldConfig": { "defaults": { "unit": "s" } }
+      },
+      {
+        "title": "Container — CPU Usage (all containers)",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 40 },
+        "targets": [
+          { "expr": "rate(container_cpu_usage_seconds_total{name=~\"trails-cool.*\"}[5m]) * 100", "legendFormat": "{{name}}" }
+        ],
+        "fieldConfig": { "defaults": { "unit": "percent" } }
+      },
+      {
+        "title": "Container — Memory Usage (all containers)",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 48 },
+        "targets": [
+          { "expr": "container_memory_usage_bytes{name=~\"trails-cool.*\"}", "legendFormat": "{{name}}" }
+        ],
+        "fieldConfig": { "defaults": { "unit": "bytes" } }
+      },
+      {
+        "title": "BRouter — Container Resources",
+        "type": "stat",
+        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 48 },
+        "targets": [
+          { "expr": "container_memory_usage_bytes{name=~\".*brouter.*\"}", "legendFormat": "Memory" },
+          { "expr": "rate(container_cpu_usage_seconds_total{name=~\".*brouter.*\"}[5m]) * 100", "legendFormat": "CPU %" }
+        ],
+        "fieldConfig": { "defaults": { "unit": "bytes" } }
+      },
+      {
+        "title": "Caddy — Request Rate by Host",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 56 },
+        "targets": [
+          { "expr": "sum(rate(caddy_http_requests_total[5m])) by (server)", "legendFormat": "{{server}}" }
+        ]
+      },
+      {
+        "title": "Caddy — Response Status Codes",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 56 },
+        "targets": [
+          { "expr": "sum(rate(caddy_http_responses_total[5m])) by (code)", "legendFormat": "{{code}}" }
+        ]
+      },
+      {
+        "title": "Caddy — Request Duration by Host",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 64 },
+        "targets": [
+          { "expr": "histogram_quantile(0.95, sum(rate(caddy_http_request_duration_seconds_bucket[5m])) by (le, server))", "legendFormat": "p95 {{server}}" }
+        ],
+        "fieldConfig": { "defaults": { "unit": "s" } }
+      },
+      {
+        "title": "Caddy — Active Connections",
+        "type": "timeseries",
+        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 64 },
+        "targets": [
+          { "expr": "caddy_http_requests_in_flight", "legendFormat": "in-flight" }
+        ]
+      }
+    ]
+  }
+}

infrastructure/prometheus/prometheus.yml

@@ -13,6 +13,18 @@ scrape_configs:
     static_configs:
       - targets: ["planner:3001"]
 
+  - job_name: "postgres"
+    static_configs:
+      - targets: ["postgres-exporter:9187"]
+
+  - job_name: "node"
+    static_configs:
+      - targets: ["node-exporter:9100"]
+
+  - job_name: "cadvisor"
+    static_configs:
+      - targets: ["cadvisor:8080"]
+
   - job_name: "caddy"
     static_configs:
       - targets: ["caddy:2019"]

openspec/changes/archive/2026-03-27-sops-age-split-cd/tasks.md

@@ -4,7 +4,7 @@
 - [x] 1.2 Create `.sops.yaml` at repo root with age encryption rule for `secrets.*.env`
 - [x] 1.3 Create `infrastructure/secrets.app.env` with app secrets (POSTGRES_PASSWORD, JWT_SECRET, SESSION_SECRET, SMTP_URL, SMTP_FROM, SENTRY_AUTH_TOKEN, DEPLOY_GHCR_TOKEN), encrypt with `sops -e`
 - [x] 1.4 Create `infrastructure/secrets.infra.env` with infra secrets (GF_AUTH_GITHUB_CLIENT_ID, GF_AUTH_GITHUB_CLIENT_SECRET), encrypt with `sops -e`
-- [ ] 1.5 Remove migrated secrets from GitHub Actions (keep only AGE_SECRET_KEY, DEPLOY_SSH_KEY, DEPLOY_HOST)
+- [x] 1.5 Remove migrated secrets from GitHub Actions (keep only AGE_SECRET_KEY, DEPLOY_SSH_KEY, DEPLOY_HOST)
 
 ## 2. GitHub OAuth for Grafana
 
@@ -30,7 +30,7 @@
 ## 4. Verify
 
 - [x] 4.1 Test sops encrypt/decrypt cycle locally
-- [ ] 4.2 Test cd-apps deploys only on app changes (not infra)
-- [ ] 4.3 Test cd-infra deploys only on infra changes (not apps)
-- [ ] 4.4 Test Grafana GitHub OAuth login
-- [ ] 4.5 Verify old GitHub secrets can be removed after migration
+- [x] 4.2 Test cd-apps deploys only on app changes (not infra)
+- [x] 4.3 Test cd-infra deploys only on infra changes (not apps)
+- [x] 4.4 Test Grafana GitHub OAuth login
+- [x] 4.5 Verify old GitHub secrets can be removed after migration

openspec/specs/infrastructure/spec.md

@@ -52,11 +52,19 @@ The infrastructure SHALL support downloading and updating Germany RD5 segments f
 - **THEN** RD5 segments are updated from brouter.de and the BRouter container is restarted
 
 ### Requirement: CI/CD pipeline
-GitHub Actions SHALL build, deploy, and security-scan both apps on push to main.
+GitHub Actions SHALL use separate workflows for app deployment and infrastructure deployment, with secrets decrypted from a SOPS-encrypted file.
 
-#### Scenario: Push triggers deployment
-- **WHEN** code is pushed to the main branch
-- **THEN** GitHub Actions builds Docker images, pushes to ghcr.io/trails-cool/, and deploys to the Hetzner server
+#### Scenario: App deployment
+- **WHEN** code changes are pushed to main in apps/ or packages/
+- **THEN** the cd-apps workflow builds Docker images, pushes to ghcr.io, and deploys app containers
+
+#### Scenario: Infrastructure deployment
+- **WHEN** changes are pushed to main in infrastructure/
+- **THEN** the cd-infra workflow copies configs and restarts infrastructure services without rebuilding app images
+
+#### Scenario: Secret decryption at deploy time
+- **WHEN** either CD workflow runs
+- **THEN** the SOPS-encrypted secrets file is decrypted and provided to docker-compose as an env file
 
 #### Scenario: Gitleaks scan
 - **WHEN** a PR is opened
@@ -102,3 +110,14 @@ The system SHALL enrich Sentry events with user and session context, use route-a
 #### Scenario: Source maps not served to clients
 - **WHEN** a client requests a `.map` file from the production server
 - **THEN** the server SHALL return 404 (source maps are uploaded to Sentry during build, not shipped in the bundle)
+
+### Requirement: Grafana authentication
+Grafana SHALL authenticate users via GitHub OAuth, restricted to the trails-cool GitHub organization.
+
+#### Scenario: GitHub OAuth login
+- **WHEN** a user navigates to grafana.internal.trails.cool
+- **THEN** they are redirected to GitHub for authentication and granted access if they are a member of the trails-cool organization
+
+#### Scenario: No password-based login
+- **WHEN** Grafana is deployed
+- **THEN** the login form is disabled and only GitHub OAuth is available

openspec/specs/secret-management/spec.md

@@ -0,0 +1,16 @@
+## Requirements
+
+### Requirement: Encrypted secrets in repository
+Production secrets SHALL be stored as a SOPS-encrypted file in the repository, decryptable only with a single age private key.
+
+#### Scenario: Edit secrets
+- **WHEN** a developer runs `sops infrastructure/secrets.env`
+- **THEN** the file is decrypted in a temporary editor, and re-encrypted on save
+
+#### Scenario: CD decryption
+- **WHEN** the CD workflow runs
+- **THEN** the encrypted secrets file is decrypted using the AGE_SECRET_KEY GitHub secret and provided to docker-compose as an env file
+
+#### Scenario: Secret audit trail
+- **WHEN** a secret is changed
+- **THEN** the change appears in git history as a diff of the encrypted file with a commit message describing what changed