Add service health monitoring: postgres, node, cAdvisor exporters + dashboard

Exporters:
- postgres_exporter: DB connections, transactions, cache hit ratio, query stats
- node_exporter: host CPU, memory, disk, network
- cAdvisor: per-container CPU and memory usage

PostgreSQL:
- Enable pg_stat_statements for query-level performance tracking
- Track index scans vs sequential scans, cache hit ratio

Dashboard (service-health.json):
- DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage
- Host: disk gauge, CPU, memory, network I/O, disk I/O
- BRouter: request latency p50/p95/p99, container CPU + memory
- All containers: CPU and memory comparison

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-03-27 18:44:14 +01:00
parent f87aca6188
commit 424e692ee0
No known key found for this signature in database
GPG key ID: A32FF691A0F752D9
14 changed files with 326 additions and 12 deletions

View file

@ -140,6 +140,32 @@ before pushing to an existing PR branch.
### Emergency override
Admins can bypass the PR workflow when necessary (e.g., CI is broken and needs a hotfix). Document the reason in the commit message.
## Deployment
Three separate CD workflows triggered by path:
| Workflow | Triggers on | Deploys |
|----------|-------------|---------|
| `cd-apps.yml` | `apps/`, `packages/`, `pnpm-lock.yaml` | journal, planner |
| `cd-infra.yml` | `infrastructure/` | caddy, postgres, prometheus, loki, grafana, exporters |
| `cd-brouter.yml` | `docker/brouter/` | brouter |
### Secrets
All secrets are stored in SOPS-encrypted files (`infrastructure/secrets.app.env`, `infrastructure/secrets.infra.env`). Edit with `sops infrastructure/secrets.app.env`. Only `AGE_SECRET_KEY`, `DEPLOY_SSH_KEY`, and `DEPLOY_HOST` remain as GitHub secrets.
### Full restart
To restart **all** containers (not just the ones a workflow normally touches), either:
- Add `[restart-all]` to the commit message
- Or trigger manually: `gh workflow run cd-infra.yml -f restart_all=true`
### Server access
```bash
ssh -i ~/.ssh/trails-cool-deploy root@trails.cool
```
### Grafana
`https://grafana.internal.trails.cool` — GitHub OAuth (trails-cool org)
## OpenSpec Workflow
Specs live in `openspec/`. Use these slash commands: