Add service health monitoring: postgres, node, cAdvisor exporters + dashboard
Exporters: - postgres_exporter: DB connections, transactions, cache hit ratio, query stats - node_exporter: host CPU, memory, disk, network - cAdvisor: per-container CPU and memory usage PostgreSQL: - Enable pg_stat_statements for query-level performance tracking - Track index scans vs sequential scans, cache hit ratio Dashboard (service-health.json): - DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage - Host: disk gauge, CPU, memory, network I/O, disk I/O - BRouter: request latency p50/p95/p99, container CPU + memory - All containers: CPU and memory comparison Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
f87aca6188
commit
424e692ee0
14 changed files with 326 additions and 12 deletions
|
|
@ -1,44 +0,0 @@
|
|||
## Why
|
||||
|
||||
Secrets are scattered across GitHub Actions secrets with no version control,
|
||||
no audit trail, and painful manual management (the Grafana password hash saga).
|
||||
The CD pipeline is monolithic — changing a Grafana dashboard rebuilds both app
|
||||
Docker images. And Grafana authentication requires managing bcrypt hashes and
|
||||
basic auth layers.
|
||||
|
||||
## What Changes
|
||||
|
||||
- **SOPS + age for secrets**: Encrypt a `.env.production` file in the repo.
|
||||
CD decrypts at deploy time with a single age private key stored as one
|
||||
GitHub secret. All other secrets move from GitHub Actions secrets into the
|
||||
encrypted file — version-controlled, diffable, auditable.
|
||||
- **Split CD into apps vs infra**: Two workflows triggered by path filters.
|
||||
App changes (apps/, packages/) build Docker images and deploy. Infra changes
|
||||
(infrastructure/) copy configs and restart services. No unnecessary rebuilds.
|
||||
- **GitHub OAuth for Grafana**: Replace Caddy basic auth + Grafana login with
|
||||
GitHub OAuth. One login, restricted to the trails-cool GitHub org. Remove
|
||||
GRAFANA_PASSWORD_HASH, GRAFANA_USER, GRAFANA_PASSWORD secrets entirely.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `secret-management`: SOPS + age encrypted secrets in the repository with
|
||||
CD decryption
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- `infrastructure`: Split CD workflows, GitHub OAuth for Grafana, remove
|
||||
Caddy basic auth for Grafana
|
||||
|
||||
## Impact
|
||||
|
||||
- **Files**: New `.env.production.enc` (encrypted), `.sops.yaml` config,
|
||||
split `cd-apps.yml` and `cd-infra.yml` workflows, updated docker-compose.yml
|
||||
and Caddyfile
|
||||
- **Dependencies**: `sops` and `age` CLI tools in CD runner (install step)
|
||||
- **GitHub secrets**: Reduced from ~10 secrets to 2 (AGE_SECRET_KEY +
|
||||
DEPLOY_SSH_KEY). Everything else moves into the encrypted env file.
|
||||
- **Grafana**: GitHub OAuth app registration needed (Client ID + Secret go
|
||||
into the SOPS-encrypted file)
|
||||
- **Caddy**: Remove basic auth block for grafana.internal, just proxy through
|
||||
Loading…
Add table
Add a link
Reference in a new issue