Add service health monitoring: postgres, node, cAdvisor exporters + dashboard
Exporters: - postgres_exporter: DB connections, transactions, cache hit ratio, query stats - node_exporter: host CPU, memory, disk, network - cAdvisor: per-container CPU and memory usage PostgreSQL: - Enable pg_stat_statements for query-level performance tracking - Track index scans vs sequential scans, cache hit ratio Dashboard (service-health.json): - DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage - Host: disk gauge, CPU, memory, network I/O, disk I/O - BRouter: request latency p50/p95/p99, container CPU + memory - All containers: CPU and memory comparison Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
f87aca6188
commit
424e692ee0
14 changed files with 326 additions and 12 deletions
|
|
@ -4,7 +4,7 @@
|
|||
- [x] 1.2 Create `.sops.yaml` at repo root with age encryption rule for `secrets.*.env`
|
||||
- [x] 1.3 Create `infrastructure/secrets.app.env` with app secrets (POSTGRES_PASSWORD, JWT_SECRET, SESSION_SECRET, SMTP_URL, SMTP_FROM, SENTRY_AUTH_TOKEN, DEPLOY_GHCR_TOKEN), encrypt with `sops -e`
|
||||
- [x] 1.4 Create `infrastructure/secrets.infra.env` with infra secrets (GF_AUTH_GITHUB_CLIENT_ID, GF_AUTH_GITHUB_CLIENT_SECRET), encrypt with `sops -e`
|
||||
- [ ] 1.5 Remove migrated secrets from GitHub Actions (keep only AGE_SECRET_KEY, DEPLOY_SSH_KEY, DEPLOY_HOST)
|
||||
- [x] 1.5 Remove migrated secrets from GitHub Actions (keep only AGE_SECRET_KEY, DEPLOY_SSH_KEY, DEPLOY_HOST)
|
||||
|
||||
## 2. GitHub OAuth for Grafana
|
||||
|
||||
|
|
@ -30,7 +30,7 @@
|
|||
## 4. Verify
|
||||
|
||||
- [x] 4.1 Test sops encrypt/decrypt cycle locally
|
||||
- [ ] 4.2 Test cd-apps deploys only on app changes (not infra)
|
||||
- [ ] 4.3 Test cd-infra deploys only on infra changes (not apps)
|
||||
- [ ] 4.4 Test Grafana GitHub OAuth login
|
||||
- [ ] 4.5 Verify old GitHub secrets can be removed after migration
|
||||
- [x] 4.2 Test cd-apps deploys only on app changes (not infra)
|
||||
- [x] 4.3 Test cd-infra deploys only on infra changes (not apps)
|
||||
- [x] 4.4 Test Grafana GitHub OAuth login
|
||||
- [x] 4.5 Verify old GitHub secrets can be removed after migration
|
||||
|
|
@ -52,11 +52,19 @@ The infrastructure SHALL support downloading and updating Germany RD5 segments f
|
|||
- **THEN** RD5 segments are updated from brouter.de and the BRouter container is restarted
|
||||
|
||||
### Requirement: CI/CD pipeline
|
||||
GitHub Actions SHALL build, deploy, and security-scan both apps on push to main.
|
||||
GitHub Actions SHALL use separate workflows for app deployment and infrastructure deployment, with secrets decrypted from a SOPS-encrypted file.
|
||||
|
||||
#### Scenario: Push triggers deployment
|
||||
- **WHEN** code is pushed to the main branch
|
||||
- **THEN** GitHub Actions builds Docker images, pushes to ghcr.io/trails-cool/, and deploys to the Hetzner server
|
||||
#### Scenario: App deployment
|
||||
- **WHEN** code changes are pushed to main in apps/ or packages/
|
||||
- **THEN** the cd-apps workflow builds Docker images, pushes to ghcr.io, and deploys app containers
|
||||
|
||||
#### Scenario: Infrastructure deployment
|
||||
- **WHEN** changes are pushed to main in infrastructure/
|
||||
- **THEN** the cd-infra workflow copies configs and restarts infrastructure services without rebuilding app images
|
||||
|
||||
#### Scenario: Secret decryption at deploy time
|
||||
- **WHEN** either CD workflow runs
|
||||
- **THEN** the SOPS-encrypted secrets file is decrypted and provided to docker-compose as an env file
|
||||
|
||||
#### Scenario: Gitleaks scan
|
||||
- **WHEN** a PR is opened
|
||||
|
|
@ -102,3 +110,14 @@ The system SHALL enrich Sentry events with user and session context, use route-a
|
|||
#### Scenario: Source maps not served to clients
|
||||
- **WHEN** a client requests a `.map` file from the production server
|
||||
- **THEN** the server SHALL return 404 (source maps are uploaded to Sentry during build, not shipped in the bundle)
|
||||
|
||||
### Requirement: Grafana authentication
|
||||
Grafana SHALL authenticate users via GitHub OAuth, restricted to the trails-cool GitHub organization.
|
||||
|
||||
#### Scenario: GitHub OAuth login
|
||||
- **WHEN** a user navigates to grafana.internal.trails.cool
|
||||
- **THEN** they are redirected to GitHub for authentication and granted access if they are a member of the trails-cool organization
|
||||
|
||||
#### Scenario: No password-based login
|
||||
- **WHEN** Grafana is deployed
|
||||
- **THEN** the login form is disabled and only GitHub OAuth is available
|
||||
|
|
|
|||
16
openspec/specs/secret-management/spec.md
Normal file
16
openspec/specs/secret-management/spec.md
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
## Requirements
|
||||
|
||||
### Requirement: Encrypted secrets in repository
|
||||
Production secrets SHALL be stored as a SOPS-encrypted file in the repository, decryptable only with a single age private key.
|
||||
|
||||
#### Scenario: Edit secrets
|
||||
- **WHEN** a developer runs `sops infrastructure/secrets.env`
|
||||
- **THEN** the file is decrypted in a temporary editor, and re-encrypted on save
|
||||
|
||||
#### Scenario: CD decryption
|
||||
- **WHEN** the CD workflow runs
|
||||
- **THEN** the encrypted secrets file is decrypted using the AGE_SECRET_KEY GitHub secret and provided to docker-compose as an env file
|
||||
|
||||
#### Scenario: Secret audit trail
|
||||
- **WHEN** a secret is changed
|
||||
- **THEN** the change appears in git history as a diff of the encrypted file with a commit message describing what changed
|
||||
Loading…
Add table
Add a link
Reference in a new issue