Add service health monitoring: postgres, node, cAdvisor exporters + dashboard

Exporters:
- postgres_exporter: DB connections, transactions, cache hit ratio, query stats
- node_exporter: host CPU, memory, disk, network
- cAdvisor: per-container CPU and memory usage

PostgreSQL:
- Enable pg_stat_statements for query-level performance tracking
- Track index scans vs sequential scans, cache hit ratio

Dashboard (service-health.json):
- DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage
- Host: disk gauge, CPU, memory, network I/O, disk I/O
- BRouter: request latency p50/p95/p99, container CPU + memory
- All containers: CPU and memory comparison

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-03-27 18:44:14 +01:00
parent f87aca6188
commit 424e692ee0
No known key found for this signature in database
GPG key ID: A32FF691A0F752D9
14 changed files with 326 additions and 12 deletions

View file

@ -1,44 +0,0 @@
## Why
Secrets are scattered across GitHub Actions secrets with no version control,
no audit trail, and painful manual management (the Grafana password hash saga).
The CD pipeline is monolithic — changing a Grafana dashboard rebuilds both app
Docker images. And Grafana authentication requires managing bcrypt hashes and
basic auth layers.
## What Changes
- **SOPS + age for secrets**: Encrypt a `.env.production` file in the repo.
CD decrypts at deploy time with a single age private key stored as one
GitHub secret. All other secrets move from GitHub Actions secrets into the
encrypted file — version-controlled, diffable, auditable.
- **Split CD into apps vs infra**: Two workflows triggered by path filters.
App changes (apps/, packages/) build Docker images and deploy. Infra changes
(infrastructure/) copy configs and restart services. No unnecessary rebuilds.
- **GitHub OAuth for Grafana**: Replace Caddy basic auth + Grafana login with
GitHub OAuth. One login, restricted to the trails-cool GitHub org. Remove
GRAFANA_PASSWORD_HASH, GRAFANA_USER, GRAFANA_PASSWORD secrets entirely.
## Capabilities
### New Capabilities
- `secret-management`: SOPS + age encrypted secrets in the repository with
CD decryption
### Modified Capabilities
- `infrastructure`: Split CD workflows, GitHub OAuth for Grafana, remove
Caddy basic auth for Grafana
## Impact
- **Files**: New `.env.production.enc` (encrypted), `.sops.yaml` config,
split `cd-apps.yml` and `cd-infra.yml` workflows, updated docker-compose.yml
and Caddyfile
- **Dependencies**: `sops` and `age` CLI tools in CD runner (install step)
- **GitHub secrets**: Reduced from ~10 secrets to 2 (AGE_SECRET_KEY +
DEPLOY_SSH_KEY). Everything else moves into the encrypted env file.
- **Grafana**: GitHub OAuth app registration needed (Client ID + Secret go
into the SOPS-encrypted file)
- **Caddy**: Remove basic auth block for grafana.internal, just proxy through