Add service health monitoring: postgres, node, cAdvisor exporters + dashboard

Exporters: - postgres_exporter: DB connections, transactions, cache hit ratio, query stats - node_exporter: host CPU, memory, disk, network - cAdvisor: per-container CPU and memory usage PostgreSQL: - Enable pg_stat_statements for query-level performance tracking - Track index scans vs sequential scans, cache hit ratio Dashboard (service-health.json): - DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage - Host: disk gauge, CPU, memory, network I/O, disk I/O - BRouter: request latency p50/p95/p99, container CPU + memory - All containers: CPU and memory comparison Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 19:01:15 +01:00
17 changed files with 859 additions and 140 deletions
--- a/.github/workflows/cd-infra.yml
+++ b/.github/workflows/cd-infra.yml
@ -68,12 +68,19 @@ jobs:
            GHCR_TOKEN=$(grep DEPLOY_GHCR_TOKEN .env | cut -d= -f2-)
            echo "$GHCR_TOKEN" | docker login ghcr.io -u stigi --password-stdin 2>/dev/null || true

-            # Restart services
-            if [ "${{ github.event.inputs.restart_all }}" = "true" ]; then
+            # Check if full restart requested via:
+            # - workflow_dispatch input: restart_all=true
+            # - commit message containing [restart-all]
+            RESTART_ALL="${{ github.event.inputs.restart_all }}"
+            if [ "$RESTART_ALL" != "true" ] && echo "${{ github.event.head_commit.message }}" | grep -q '\[restart-all\]'; then
+              RESTART_ALL="true"
+            fi
+
+            if [ "$RESTART_ALL" = "true" ]; then
              docker compose --env-file .env up -d --remove-orphans
            else
              # Restart infra services (except Caddy — just reload its config)
-              docker compose --env-file .env up -d postgres prometheus loki grafana
+              docker compose --env-file .env up -d postgres prometheus loki grafana postgres-exporter node-exporter cadvisor
              docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile
            fi

--- a/CLAUDE.md
+++ b/CLAUDE.md
@ -140,6 +140,32 @@ before pushing to an existing PR branch.
 ### Emergency override
 Admins can bypass the PR workflow when necessary (e.g., CI is broken and needs a hotfix). Document the reason in the commit message.

+## Deployment
+
+Three separate CD workflows triggered by path:
+
+| Workflow | Triggers on | Deploys |
+|----------|-------------|---------|
+| `cd-apps.yml` | `apps/`, `packages/`, `pnpm-lock.yaml` | journal, planner |
+| `cd-infra.yml` | `infrastructure/` | caddy, postgres, prometheus, loki, grafana, exporters |
+| `cd-brouter.yml` | `docker/brouter/` | brouter |
+
+### Secrets
+All secrets are stored in SOPS-encrypted files (`infrastructure/secrets.app.env`, `infrastructure/secrets.infra.env`). Edit with `sops infrastructure/secrets.app.env`. Only `AGE_SECRET_KEY`, `DEPLOY_SSH_KEY`, and `DEPLOY_HOST` remain as GitHub secrets.
+
+### Full restart
+To restart **all** containers (not just the ones a workflow normally touches), either:
+- Add `[restart-all]` to the commit message
+- Or trigger manually: `gh workflow run cd-infra.yml -f restart_all=true`
+
+### Server access
+```bash
+ssh -i ~/.ssh/trails-cool-deploy root@trails.cool
+```
+
+### Grafana
+`https://grafana.internal.trails.cool` — GitHub OAuth (trails-cool org)
+
 ## OpenSpec Workflow

 Specs live in `openspec/`. Use these slash commands:
--- a/infrastructure/Caddyfile
+++ b/infrastructure/Caddyfile
@ -1,3 +1,10 @@
+{
+	servers {
+		metrics
+	}
+	admin 0.0.0.0:2019
+}
+
 (security_headers) {
 	header {
 		Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
--- a/infrastructure/docker-compose.yml
+++ b/infrastructure/docker-compose.yml
@ -78,12 +78,52 @@ services:
      POSTGRES_DB: trails
    volumes:
      - pgdata:/var/lib/postgresql/data
+    command:
+      - "postgres"
+      - "-c"
+      - "shared_preload_libraries=pg_stat_statements"
+      - "-c"
+      - "pg_stat_statements.track=all"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U trails"]
      interval: 5s
      timeout: 5s
      retries: 5

+  postgres-exporter:
+    image: prometheuscommunity/postgres-exporter:latest
+    restart: unless-stopped
+    environment:
+      DATA_SOURCE_NAME: postgresql://trails:${POSTGRES_PASSWORD:-trails}@postgres:5432/trails?sslmode=disable
+    depends_on:
+      postgres:
+        condition: service_healthy
+
+  node-exporter:
+    image: prom/node-exporter:latest
+    restart: unless-stopped
+    pid: host
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+    command:
+      - "--path.procfs=/host/proc"
+      - "--path.sysfs=/host/sys"
+      - "--path.rootfs=/rootfs"
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
+
+  cadvisor:
+    image: gcr.io/cadvisor/cadvisor:latest
+    restart: unless-stopped
+    privileged: true
+    volumes:
+      - /:/rootfs:ro
+      - /var/run:/var/run:ro
+      - /sys:/sys:ro
+      - /var/lib/docker/:/var/lib/docker:ro
+      - /dev/disk/:/dev/disk:ro
+
  prometheus:
    image: prom/prometheus:latest
    restart: unless-stopped
@ -114,6 +154,8 @@ services:
      GF_AUTH_GITHUB_ALLOWED_ORGANIZATIONS: trails-cool
      GF_AUTH_GITHUB_SCOPES: user:email,read:org
      GF_AUTH_DISABLE_LOGIN_FORM: "true"
+      GF_AUTH_GITHUB_ROLE_ATTRIBUTE_PATH: "contains(groups[*], '@trails-cool/owners') && 'GrafanaAdmin' || 'Editor'"
+      GF_AUTH_GITHUB_ALLOW_ASSIGN_GRAFANA_ADMIN: "true"
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
--- a/infrastructure/grafana/dashboards/infrastructure.json
+++ b/infrastructure/grafana/dashboards/infrastructure.json
@ -1,45 +1,87 @@
 {
-  "dashboard": {
-    "title": "Infrastructure",
-    "uid": "trails-infra",
-    "timezone": "browser",
-    "refresh": "30s",
-    "panels": [
-      {
-        "title": "Process Memory (RSS)",
-        "type": "timeseries",
-        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
-        "targets": [
-          { "expr": "process_resident_memory_bytes", "legendFormat": "{{job}}" }
-        ],
-        "fieldConfig": { "defaults": { "unit": "bytes" } }
+  "title": "Infrastructure",
+  "uid": "trails-infra",
+  "timezone": "browser",
+  "refresh": "30s",
+  "panels": [
+    {
+      "title": "Process Memory (RSS)",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
      },
-      {
-        "title": "CPU Usage",
-        "type": "timeseries",
-        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
-        "targets": [
-          { "expr": "rate(process_cpu_seconds_total[5m])", "legendFormat": "{{job}}" }
-        ],
-        "fieldConfig": { "defaults": { "unit": "percentunit" } }
-      },
-      {
-        "title": "Event Loop Lag",
-        "type": "timeseries",
-        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
-        "targets": [
-          { "expr": "nodejs_eventloop_lag_seconds", "legendFormat": "{{job}}" }
-        ],
-        "fieldConfig": { "defaults": { "unit": "s" } }
-      },
-      {
-        "title": "Open File Descriptors",
-        "type": "timeseries",
-        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
-        "targets": [
-          { "expr": "process_open_fds", "legendFormat": "{{job}}" }
-        ]
+      "targets": [
+        {
+          "expr": "process_resident_memory_bytes",
+          "legendFormat": "{{job}}"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        }
      }
-    ]
-  }
-}
+    },
+    {
+      "title": "CPU Usage",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "targets": [
+        {
+          "expr": "rate(process_cpu_seconds_total[5m])",
+          "legendFormat": "{{job}}"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        }
+      }
+    },
+    {
+      "title": "Event Loop Lag",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 8
+      },
+      "targets": [
+        {
+          "expr": "nodejs_eventloop_lag_seconds",
+          "legendFormat": "{{job}}"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        }
+      }
+    },
+    {
+      "title": "Open File Descriptors",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 8
+      },
+      "targets": [
+        {
+          "expr": "process_open_fds",
+          "legendFormat": "{{job}}"
+        }
+      ]
+    }
+  ]
+}
--- a/infrastructure/grafana/dashboards/overview.json
+++ b/infrastructure/grafana/dashboards/overview.json
@ -1,57 +1,121 @@
 {
-  "dashboard": {
-    "title": "trails.cool Overview",
-    "uid": "trails-overview",
-    "timezone": "browser",
-    "refresh": "30s",
-    "panels": [
-      {
-        "title": "Request Rate",
-        "type": "timeseries",
-        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
-        "targets": [
-          {
-            "expr": "sum(rate(http_request_duration_seconds_count[5m])) by (job)",
-            "legendFormat": "{{job}}"
-          }
-        ]
+  "title": "trails.cool Overview",
+  "uid": "trails-overview",
+  "timezone": "browser",
+  "refresh": "30s",
+  "panels": [
+    {
+      "title": "Request Rate",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
      },
-      {
-        "title": "Error Rate",
-        "type": "timeseries",
-        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
-        "targets": [
-          {
-            "expr": "sum(rate(http_request_duration_seconds_count{status=~\"5..\"}[5m])) / sum(rate(http_request_duration_seconds_count[5m])) * 100",
-            "legendFormat": "Error %"
-          }
-        ],
-        "fieldConfig": {
-          "defaults": { "unit": "percent", "thresholds": { "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 5 }] } }
+      "targets": [
+        {
+          "expr": "sum(rate(http_request_duration_seconds_count[5m])) by (job)",
+          "legendFormat": "{{job}}"
        }
+      ]
+    },
+    {
+      "title": "Error Rate",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 0
      },
-      {
-        "title": "Latency p50 / p95 / p99",
-        "type": "timeseries",
-        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
-        "targets": [
-          { "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p50" },
-          { "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p95" },
-          { "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p99" }
-        ],
-        "fieldConfig": { "defaults": { "unit": "s" } }
-      },
-      {
-        "title": "Health Status",
-        "type": "stat",
-        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
-        "targets": [
-          { "expr": "up", "legendFormat": "{{job}}" }
-        ],
-        "fieldConfig": {
-          "defaults": { "mappings": [{ "type": "value", "options": { "0": { "text": "DOWN", "color": "red" }, "1": { "text": "UP", "color": "green" } } }] }
+      "targets": [
+        {
+          "expr": "sum(rate(http_request_duration_seconds_count{status=~\"5..\"}[5m])) / sum(rate(http_request_duration_seconds_count[5m])) * 100",
+          "legendFormat": "Error %"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "thresholds": {
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 5
+              }
+            ]
+          }
        }
      }
-    ]
-  }
-}
+    },
+    {
+      "title": "Latency p50 / p95 / p99",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 8
+      },
+      "targets": [
+        {
+          "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))",
+          "legendFormat": "p50"
+        },
+        {
+          "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))",
+          "legendFormat": "p95"
+        },
+        {
+          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))",
+          "legendFormat": "p99"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        }
+      }
+    },
+    {
+      "title": "Health Status",
+      "type": "stat",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 8
+      },
+      "targets": [
+        {
+          "expr": "up",
+          "legendFormat": "{{job}}"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [
+            {
+              "type": "value",
+              "options": {
+                "0": {
+                  "text": "DOWN",
+                  "color": "red"
+                },
+                "1": {
+                  "text": "UP",
+                  "color": "green"
+                }
+              }
+            }
+          ]
+        }
+      }
+    }
+  ]
+}
--- a/infrastructure/grafana/dashboards/planner.json
+++ b/infrastructure/grafana/dashboards/planner.json
@ -1,40 +1,81 @@
 {
-  "dashboard": {
-    "title": "Planner",
-    "uid": "trails-planner",
-    "timezone": "browser",
-    "refresh": "30s",
-    "panels": [
-      {
-        "title": "Active Sessions",
-        "type": "stat",
-        "gridPos": { "h": 6, "w": 6, "x": 0, "y": 0 },
-        "targets": [{ "expr": "planner_active_sessions", "legendFormat": "Sessions" }]
+  "title": "Planner",
+  "uid": "trails-planner",
+  "timezone": "browser",
+  "refresh": "30s",
+  "panels": [
+    {
+      "title": "Active Sessions",
+      "type": "stat",
+      "gridPos": {
+        "h": 6,
+        "w": 6,
+        "x": 0,
+        "y": 0
      },
-      {
-        "title": "Connected Clients",
-        "type": "stat",
-        "gridPos": { "h": 6, "w": 6, "x": 6, "y": 0 },
-        "targets": [{ "expr": "planner_connected_clients", "legendFormat": "Clients" }]
+      "targets": [
+        {
+          "expr": "planner_active_sessions",
+          "legendFormat": "Sessions"
+        }
+      ]
+    },
+    {
+      "title": "Connected Clients",
+      "type": "stat",
+      "gridPos": {
+        "h": 6,
+        "w": 6,
+        "x": 6,
+        "y": 0
      },
-      {
-        "title": "BRouter Latency",
-        "type": "timeseries",
-        "gridPos": { "h": 6, "w": 12, "x": 12, "y": 0 },
-        "targets": [
-          { "expr": "histogram_quantile(0.50, rate(brouter_request_duration_seconds_bucket[5m]))", "legendFormat": "p50" },
-          { "expr": "histogram_quantile(0.95, rate(brouter_request_duration_seconds_bucket[5m]))", "legendFormat": "p95" }
-        ],
-        "fieldConfig": { "defaults": { "unit": "s" } }
+      "targets": [
+        {
+          "expr": "planner_connected_clients",
+          "legendFormat": "Clients"
+        }
+      ]
+    },
+    {
+      "title": "BRouter Latency",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 6,
+        "w": 12,
+        "x": 12,
+        "y": 0
      },
-      {
-        "title": "Request Rate by Route",
-        "type": "timeseries",
-        "gridPos": { "h": 8, "w": 24, "x": 0, "y": 6 },
-        "targets": [
-          { "expr": "sum(rate(http_request_duration_seconds_count{job=\"planner\"}[5m])) by (route)", "legendFormat": "{{route}}" }
-        ]
+      "targets": [
+        {
+          "expr": "histogram_quantile(0.50, rate(brouter_request_duration_seconds_bucket[5m]))",
+          "legendFormat": "p50"
+        },
+        {
+          "expr": "histogram_quantile(0.95, rate(brouter_request_duration_seconds_bucket[5m]))",
+          "legendFormat": "p95"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        }
      }
-    ]
-  }
-}
+    },
+    {
+      "title": "Request Rate by Route",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 6
+      },
+      "targets": [
+        {
+          "expr": "sum(rate(http_request_duration_seconds_count{job=\"planner\"}[5m])) by (route)",
+          "legendFormat": "{{route}}"
+        }
+      ]
+    }
+  ]
+}
--- a/infrastructure/grafana/dashboards/service-health.json
+++ b/infrastructure/grafana/dashboards/service-health.json
@ -0,0 +1,443 @@
+{
+  "title": "Service Health",
+  "uid": "trails-service-health",
+  "timezone": "browser",
+  "refresh": "30s",
+  "panels": [
+    {
+      "title": "PostgreSQL \u2014 Active Connections",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "targets": [
+        {
+          "expr": "pg_stat_activity_count{datname=\"trails\"}",
+          "legendFormat": "{{state}}"
+        }
+      ]
+    },
+    {
+      "title": "PostgreSQL \u2014 Database Size",
+      "type": "stat",
+      "gridPos": {
+        "h": 8,
+        "w": 6,
+        "x": 12,
+        "y": 0
+      },
+      "targets": [
+        {
+          "expr": "pg_database_size_bytes{datname=\"trails\"}",
+          "legendFormat": "trails"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        }
+      }
+    },
+    {
+      "title": "PostgreSQL \u2014 Transactions/sec",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 6,
+        "x": 18,
+        "y": 0
+      },
+      "targets": [
+        {
+          "expr": "rate(pg_stat_database_xact_commit{datname=\"trails\"}[5m])",
+          "legendFormat": "commits/s"
+        },
+        {
+          "expr": "rate(pg_stat_database_xact_rollback{datname=\"trails\"}[5m])",
+          "legendFormat": "rollbacks/s"
+        }
+      ]
+    },
+    {
+      "title": "PostgreSQL \u2014 Slowest Queries (pg_stat_statements)",
+      "type": "table",
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 8
+      },
+      "targets": [
+        {
+          "expr": "topk(10, pg_stat_statements_mean_time_seconds{datname=\"trails\"})",
+          "legendFormat": "{{query}}",
+          "format": "table",
+          "instant": true
+        }
+      ]
+    },
+    {
+      "title": "PostgreSQL \u2014 Cache Hit Ratio",
+      "type": "gauge",
+      "gridPos": {
+        "h": 8,
+        "w": 6,
+        "x": 0,
+        "y": 16
+      },
+      "targets": [
+        {
+          "expr": "pg_stat_database_blks_hit{datname=\"trails\"} / (pg_stat_database_blks_hit{datname=\"trails\"} + pg_stat_database_blks_read{datname=\"trails\"}) * 100",
+          "legendFormat": "hit %"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "thresholds": {
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 90
+              },
+              {
+                "color": "green",
+                "value": 99
+              }
+            ]
+          }
+        }
+      }
+    },
+    {
+      "title": "PostgreSQL \u2014 Index Scans vs Sequential Scans",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 18,
+        "x": 6,
+        "y": 16
+      },
+      "targets": [
+        {
+          "expr": "sum(rate(pg_stat_user_tables_idx_scan[5m]))",
+          "legendFormat": "Index scans/s"
+        },
+        {
+          "expr": "sum(rate(pg_stat_user_tables_seq_scan[5m]))",
+          "legendFormat": "Sequential scans/s"
+        }
+      ]
+    },
+    {
+      "title": "Host \u2014 Disk Usage",
+      "type": "gauge",
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 0,
+        "y": 24
+      },
+      "targets": [
+        {
+          "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"}) * 100",
+          "legendFormat": "Used %"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "thresholds": {
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 70
+              },
+              {
+                "color": "red",
+                "value": 85
+              }
+            ]
+          }
+        }
+      }
+    },
+    {
+      "title": "Host \u2014 CPU Usage",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 8,
+        "y": 24
+      },
+      "targets": [
+        {
+          "expr": "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
+          "legendFormat": "CPU %"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "max": 100
+        }
+      }
+    },
+    {
+      "title": "Host \u2014 Memory Usage",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 16,
+        "y": 24
+      },
+      "targets": [
+        {
+          "expr": "node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes",
+          "legendFormat": "Used"
+        },
+        {
+          "expr": "node_memory_MemAvailable_bytes",
+          "legendFormat": "Available"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        }
+      }
+    },
+    {
+      "title": "Host \u2014 Network I/O",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 32
+      },
+      "targets": [
+        {
+          "expr": "rate(node_network_receive_bytes_total{device!~\"lo|veth.*|br.*|docker.*\"}[5m])",
+          "legendFormat": "{{device}} rx"
+        },
+        {
+          "expr": "-rate(node_network_transmit_bytes_total{device!~\"lo|veth.*|br.*|docker.*\"}[5m])",
+          "legendFormat": "{{device}} tx"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "Bps"
+        }
+      }
+    },
+    {
+      "title": "Host \u2014 Disk I/O",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 32
+      },
+      "targets": [
+        {
+          "expr": "rate(node_disk_read_bytes_total[5m])",
+          "legendFormat": "{{device}} read"
+        },
+        {
+          "expr": "-rate(node_disk_written_bytes_total[5m])",
+          "legendFormat": "{{device}} write"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "Bps"
+        }
+      }
+    },
+    {
+      "title": "BRouter \u2014 Request Latency (from Planner proxy)",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 40
+      },
+      "targets": [
+        {
+          "expr": "histogram_quantile(0.50, rate(brouter_request_duration_seconds_bucket[5m]))",
+          "legendFormat": "p50"
+        },
+        {
+          "expr": "histogram_quantile(0.95, rate(brouter_request_duration_seconds_bucket[5m]))",
+          "legendFormat": "p95"
+        },
+        {
+          "expr": "histogram_quantile(0.99, rate(brouter_request_duration_seconds_bucket[5m]))",
+          "legendFormat": "p99"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        }
+      }
+    },
+    {
+      "title": "Container \u2014 CPU Usage (all containers)",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 40
+      },
+      "targets": [
+        {
+          "expr": "rate(container_cpu_usage_seconds_total{name=~\"trails-cool.*\"}[5m]) * 100",
+          "legendFormat": "{{name}}"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent"
+        }
+      }
+    },
+    {
+      "title": "Container \u2014 Memory Usage (all containers)",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 48
+      },
+      "targets": [
+        {
+          "expr": "container_memory_usage_bytes{name=~\"trails-cool.*\"}",
+          "legendFormat": "{{name}}"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        }
+      }
+    },
+    {
+      "title": "BRouter \u2014 Container Resources",
+      "type": "stat",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 48
+      },
+      "targets": [
+        {
+          "expr": "container_memory_usage_bytes{name=~\".*brouter.*\"}",
+          "legendFormat": "Memory"
+        },
+        {
+          "expr": "rate(container_cpu_usage_seconds_total{name=~\".*brouter.*\"}[5m]) * 100",
+          "legendFormat": "CPU %"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        }
+      }
+    },
+    {
+      "title": "Caddy \u2014 Request Rate by Host",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 56
+      },
+      "targets": [
+        {
+          "expr": "sum(rate(caddy_http_requests_total[5m])) by (server)",
+          "legendFormat": "{{server}}"
+        }
+      ]
+    },
+    {
+      "title": "Caddy \u2014 Response Status Codes",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 56
+      },
+      "targets": [
+        {
+          "expr": "sum(rate(caddy_http_responses_total[5m])) by (code)",
+          "legendFormat": "{{code}}"
+        }
+      ]
+    },
+    {
+      "title": "Caddy \u2014 Request Duration by Host",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 64
+      },
+      "targets": [
+        {
+          "expr": "histogram_quantile(0.95, sum(rate(caddy_http_request_duration_seconds_bucket[5m])) by (le, server))",
+          "legendFormat": "p95 {{server}}"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        }
+      }
+    },
+    {
+      "title": "Caddy \u2014 Active Connections",
+      "type": "timeseries",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 64
+      },
+      "targets": [
+        {
+          "expr": "caddy_http_requests_in_flight",
+          "legendFormat": "in-flight"
+        }
+      ]
+    }
+  ]
+}
--- a/infrastructure/prometheus/prometheus.yml
+++ b/infrastructure/prometheus/prometheus.yml
@ -13,6 +13,18 @@ scrape_configs:
    static_configs:
      - targets: ["planner:3001"]

+  - job_name: "postgres"
+    static_configs:
+      - targets: ["postgres-exporter:9187"]
+
+  - job_name: "node"
+    static_configs:
+      - targets: ["node-exporter:9100"]
+
+  - job_name: "cadvisor"
+    static_configs:
+      - targets: ["cadvisor:8080"]
+
  - job_name: "caddy"
    static_configs:
      - targets: ["caddy:2019"]
--- a/openspec/changes/archive/2026-03-27-sops-age-split-cd/.openspec.yaml
+++ b/openspec/changes/archive/2026-03-27-sops-age-split-cd/.openspec.yaml
--- a/openspec/changes/archive/2026-03-27-sops-age-split-cd/design.md
+++ b/openspec/changes/archive/2026-03-27-sops-age-split-cd/design.md
--- a/openspec/changes/archive/2026-03-27-sops-age-split-cd/proposal.md
+++ b/openspec/changes/archive/2026-03-27-sops-age-split-cd/proposal.md
--- a/openspec/changes/archive/2026-03-27-sops-age-split-cd/specs/infrastructure/spec.md
+++ b/openspec/changes/archive/2026-03-27-sops-age-split-cd/specs/infrastructure/spec.md
--- a/openspec/changes/archive/2026-03-27-sops-age-split-cd/specs/secret-management/spec.md
+++ b/openspec/changes/archive/2026-03-27-sops-age-split-cd/specs/secret-management/spec.md
--- a/openspec/changes/archive/2026-03-27-sops-age-split-cd/tasks.md
+++ b/openspec/changes/archive/2026-03-27-sops-age-split-cd/tasks.md
@ -4,7 +4,7 @@
 - [x] 1.2 Create `.sops.yaml` at repo root with age encryption rule for `secrets.*.env`
 - [x] 1.3 Create `infrastructure/secrets.app.env` with app secrets (POSTGRES_PASSWORD, JWT_SECRET, SESSION_SECRET, SMTP_URL, SMTP_FROM, SENTRY_AUTH_TOKEN, DEPLOY_GHCR_TOKEN), encrypt with `sops -e`
 - [x] 1.4 Create `infrastructure/secrets.infra.env` with infra secrets (GF_AUTH_GITHUB_CLIENT_ID, GF_AUTH_GITHUB_CLIENT_SECRET), encrypt with `sops -e`
- [ ] 1.5 Remove migrated secrets from GitHub Actions (keep only AGE_SECRET_KEY, DEPLOY_SSH_KEY, DEPLOY_HOST)
+- [x] 1.5 Remove migrated secrets from GitHub Actions (keep only AGE_SECRET_KEY, DEPLOY_SSH_KEY, DEPLOY_HOST)

 ## 2. GitHub OAuth for Grafana

@ -30,7 +30,7 @@
 ## 4. Verify

 - [x] 4.1 Test sops encrypt/decrypt cycle locally
- [ ] 4.2 Test cd-apps deploys only on app changes (not infra)
- [ ] 4.3 Test cd-infra deploys only on infra changes (not apps)
- [ ] 4.4 Test Grafana GitHub OAuth login
- [ ] 4.5 Verify old GitHub secrets can be removed after migration
+- [x] 4.2 Test cd-apps deploys only on app changes (not infra)
+- [x] 4.3 Test cd-infra deploys only on infra changes (not apps)
+- [x] 4.4 Test Grafana GitHub OAuth login
+- [x] 4.5 Verify old GitHub secrets can be removed after migration
--- a/openspec/specs/infrastructure/spec.md
+++ b/openspec/specs/infrastructure/spec.md
@ -52,11 +52,19 @@ The infrastructure SHALL support downloading and updating Germany RD5 segments f
 - **THEN** RD5 segments are updated from brouter.de and the BRouter container is restarted

 ### Requirement: CI/CD pipeline
-GitHub Actions SHALL build, deploy, and security-scan both apps on push to main.
+GitHub Actions SHALL use separate workflows for app deployment and infrastructure deployment, with secrets decrypted from a SOPS-encrypted file.

-#### Scenario: Push triggers deployment
- **WHEN** code is pushed to the main branch
- **THEN** GitHub Actions builds Docker images, pushes to ghcr.io/trails-cool/, and deploys to the Hetzner server
+#### Scenario: App deployment
+- **WHEN** code changes are pushed to main in apps/ or packages/
+- **THEN** the cd-apps workflow builds Docker images, pushes to ghcr.io, and deploys app containers
+
+#### Scenario: Infrastructure deployment
+- **WHEN** changes are pushed to main in infrastructure/
+- **THEN** the cd-infra workflow copies configs and restarts infrastructure services without rebuilding app images
+
+#### Scenario: Secret decryption at deploy time
+- **WHEN** either CD workflow runs
+- **THEN** the SOPS-encrypted secrets file is decrypted and provided to docker-compose as an env file

 #### Scenario: Gitleaks scan
 - **WHEN** a PR is opened
@ -102,3 +110,14 @@ The system SHALL enrich Sentry events with user and session context, use route-a
 #### Scenario: Source maps not served to clients
 - **WHEN** a client requests a `.map` file from the production server
 - **THEN** the server SHALL return 404 (source maps are uploaded to Sentry during build, not shipped in the bundle)
+
+### Requirement: Grafana authentication
+Grafana SHALL authenticate users via GitHub OAuth, restricted to the trails-cool GitHub organization.
+
+#### Scenario: GitHub OAuth login
+- **WHEN** a user navigates to grafana.internal.trails.cool
+- **THEN** they are redirected to GitHub for authentication and granted access if they are a member of the trails-cool organization
+
+#### Scenario: No password-based login
+- **WHEN** Grafana is deployed
+- **THEN** the login form is disabled and only GitHub OAuth is available
--- a/openspec/specs/secret-management/spec.md
+++ b/openspec/specs/secret-management/spec.md
@ -0,0 +1,16 @@
+## Requirements
+
+### Requirement: Encrypted secrets in repository
+Production secrets SHALL be stored as a SOPS-encrypted file in the repository, decryptable only with a single age private key.
+
+#### Scenario: Edit secrets
+- **WHEN** a developer runs `sops infrastructure/secrets.env`
+- **THEN** the file is decrypted in a temporary editor, and re-encrypted on save
+
+#### Scenario: CD decryption
+- **WHEN** the CD workflow runs
+- **THEN** the encrypted secrets file is decrypted using the AGE_SECRET_KEY GitHub secret and provided to docker-compose as an env file
+
+#### Scenario: Secret audit trail
+- **WHEN** a secret is changed
+- **THEN** the change appears in git history as a diff of the encrypted file with a commit message describing what changed