Add observability and security-hardening proposals, gitignore settings.local

Two new OpenSpec changes:
- observability (30 tasks): health endpoints, Prometheus, Grafana+Loki,
  structured logging, dashboards, alerting
- security-hardening (24 tasks): Caddy headers, scanner blocking,
  gitleaks, pnpm audit, dependabot, non-root Docker, fail2ban

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-03-25 09:48:08 +01:00
parent 734b022004
commit 7d20dbb12f
No known key found for this signature in database
GPG key ID: A32FF691A0F752D9
13 changed files with 564 additions and 0 deletions

1
.gitignore vendored
View file

@ -13,3 +13,4 @@ test-results/
playwright-report/ playwright-report/
playwright-results.json playwright-results.json
.claude/worktrees/ .claude/worktrees/
.claude/settings.local.json

View file

@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-03-25

View file

@ -0,0 +1,92 @@
## Context
The Hetzner CX21 runs Journal, Planner, BRouter, PostgreSQL, and Caddy. There
is no monitoring — the disk-full outage was only discovered when a user
reported errors. Sentry covers application errors but not infrastructure health,
performance trends, or log aggregation.
## Goals / Non-Goals
**Goals:**
- Health endpoints for uptime monitoring
- Structured JSON logging for searchability
- Prometheus metrics for request latency, DB, sessions, BRouter
- Grafana dashboards for at-a-glance status
- Loki for centralized log aggregation
- Alerts for disk full, app down, high error rate
**Non-Goals:**
- Distributed tracing (Sentry already does this)
- Custom business metrics (user signups, route counts — later)
- External uptime monitoring service (can add later)
- Monitoring for self-hosted instances (flagship only)
## Decisions
### D1: Grafana + Prometheus + Loki in Docker Compose
Add all three as services in the existing docker-compose.yml. They share the
Docker network with the app containers. Prometheus scrapes `/metrics` from
Journal and Planner. Loki collects logs via the Docker logging driver.
Grafana is only accessible via SSH tunnel (`ssh -L 3100:localhost:3100`) or
Caddy with basic auth on a subdomain (e.g., `grafana.trails.cool`).
### D2: Pino for structured logging
Replace `console.log`/`console.error` with Pino. JSON output in production,
pretty-print in dev. Pino is the standard Node.js structured logger — fast,
zero-dep in production, and Loki-compatible.
Create a shared logging utility in `packages/logging/` or keep it simple
with per-app `lib/logger.server.ts`.
### D3: prom-client for Prometheus metrics
Use `prom-client` to expose a `/metrics` endpoint on each app. Default Node.js
metrics (event loop lag, heap, GC) plus custom:
- `http_request_duration_seconds` (histogram, by route + method + status)
- `planner_active_sessions` (gauge)
- `planner_connected_clients` (gauge)
- `brouter_request_duration_seconds` (histogram)
- `db_pool_active_connections` (gauge)
### D4: Health endpoint checking DB connectivity
`GET /health` returns `{ status: "ok", db: "connected" }` or
`{ status: "degraded", db: "unreachable" }` with appropriate HTTP status.
Used by Docker healthcheck and external monitoring.
### D5: Caddy access logs to stdout (Loki picks them up)
Enable Caddy's `log` directive. Structured JSON access logs go to stdout,
Docker sends them to Loki via the logging driver. No sidecar needed.
### D6: Grafana provisioned dashboards
Ship dashboard JSON files in `infrastructure/grafana/dashboards/`. Grafana
auto-loads them via provisioning config. Dashboards:
- **Overview**: Request rate, error rate, latency p50/p95/p99
- **Planner**: Active sessions, connected clients, BRouter latency
- **Infrastructure**: CPU, memory, disk, DB connections
### D7: Alert rules via Grafana
Configure alert rules in provisioned dashboard or as code:
- Disk usage > 80% (fires 15 min)
- Any app returns 0 healthy responses for 2 min
- Error rate > 5% for 5 min
- DB connection pool exhausted
Notifications via email (Resend, once transactional-emails is implemented)
or webhook.
## Risks / Trade-offs
- **Memory pressure** → Grafana + Prometheus + Loki add ~500MB. CX21 has 4GB,
currently using ~2GB. Tight but workable. Monitor and upgrade to CX22 if
needed.
- **Disk usage** → Prometheus retention default 15 days, Loki retention
configurable. Set conservative limits (1GB each).
- **Complexity** → Three new services to maintain. Mitigated by using official
Docker images with minimal config.

View file

@ -0,0 +1,47 @@
## Why
We just had a production outage caused by disk full — and the only way to
diagnose it was SSH + manual docker commands. There are no health endpoints,
no structured logs, no metrics, and no dashboards. When things break, we're
flying blind. The architecture doc specifies a full Grafana + Prometheus + Loki
stack for the flagship instance.
## What Changes
- **Health endpoints**: `/health` on both apps returning service status + DB
connectivity
- **Structured logging**: JSON logs from both apps (not plain text console.log)
- **Prometheus metrics**: Request latency, active sessions, DB pool stats,
BRouter response times, exposed via `/metrics` endpoint
- **Grafana + Prometheus + Loki stack**: Self-hosted on the Hetzner server via
Docker Compose, scraping app metrics and collecting container logs
- **Dashboards**: Pre-configured Grafana dashboards for request latency, error
rates, active Planner sessions, DB performance, disk usage
- **Alerting**: Grafana alerts for disk usage > 80%, app down, high error rate
- **Caddy access logging**: Enable structured access logs for request visibility
## Capabilities
### New Capabilities
- `observability`: Health endpoints, Prometheus metrics, structured logging,
Grafana dashboards, and alerting for the flagship instance
### Modified Capabilities
- `infrastructure`: Add Grafana, Prometheus, Loki containers to Docker Compose.
Configure Caddy access logging.
## Impact
- **Infrastructure**: 3 new Docker containers (Grafana, Prometheus, Loki) +
config files. Adds ~500MB RAM usage.
- **Server**: CX21 (4GB RAM) might be tight. May need CX22 (8GB) if memory
is an issue.
- **Files**: New Prometheus/Grafana/Loki configs, modified docker-compose.yml,
new `/health` and `/metrics` routes, logging utility
- **Dependencies**: `prom-client` for Prometheus metrics, `pino` for structured
logging
- **Security**: Grafana dashboard behind basic auth or restricted to localhost
+ SSH tunnel. Metrics endpoint not publicly accessible.
- **Ports**: Grafana on 3100 (internal), Prometheus on 9090 (internal)

View file

@ -0,0 +1,15 @@
## MODIFIED Requirements
### Requirement: Docker Compose deployment
All services SHALL be deployed via Docker Compose, including Grafana, Prometheus, and Loki for the flagship instance.
#### Scenario: Monitoring stack starts
- **WHEN** `docker compose up -d` is run
- **THEN** Grafana, Prometheus, and Loki containers start alongside the application containers
### Requirement: Caddy access logging
Caddy SHALL emit structured JSON access logs for all requests.
#### Scenario: Access log emitted
- **WHEN** any HTTP request passes through Caddy
- **THEN** a JSON log line with remote IP, method, path, status, and duration is written to stdout

View file

@ -0,0 +1,67 @@
## ADDED Requirements
### Requirement: Health endpoints
Both apps SHALL expose a `/health` endpoint returning service and database status.
#### Scenario: Healthy service
- **WHEN** the app is running and the database is reachable
- **THEN** `GET /health` returns `{ "status": "ok", "db": "connected" }` with HTTP 200
#### Scenario: Degraded service
- **WHEN** the app is running but the database is unreachable
- **THEN** `GET /health` returns `{ "status": "degraded", "db": "unreachable" }` with HTTP 503
### Requirement: Prometheus metrics
Both apps SHALL expose a `/metrics` endpoint with Prometheus-formatted metrics.
#### Scenario: Default Node.js metrics
- **WHEN** Prometheus scrapes `/metrics`
- **THEN** it receives event loop lag, heap usage, and GC metrics
#### Scenario: HTTP request metrics
- **WHEN** requests are served
- **THEN** `http_request_duration_seconds` histogram is updated with route, method, and status labels
#### Scenario: Planner-specific metrics
- **WHEN** the Planner is running
- **THEN** `planner_active_sessions` and `planner_connected_clients` gauges reflect current state
### Requirement: Structured logging
Both apps SHALL output structured JSON logs in production.
#### Scenario: Request logging
- **WHEN** an HTTP request is served in production
- **THEN** a JSON log line is emitted with method, path, status, duration, and timestamp
#### Scenario: Dev mode pretty printing
- **WHEN** running in development
- **THEN** logs are human-readable (pretty-printed)
### Requirement: Grafana dashboards
The flagship instance SHALL have pre-configured Grafana dashboards.
#### Scenario: Overview dashboard
- **WHEN** an operator opens Grafana
- **THEN** they see request rate, error rate, and latency percentiles
#### Scenario: Infrastructure dashboard
- **WHEN** an operator checks infrastructure health
- **THEN** they see CPU, memory, disk usage, and DB connection pool stats
### Requirement: Alerting
Grafana SHALL alert on critical conditions.
#### Scenario: Disk usage alert
- **WHEN** disk usage exceeds 80% for 15 minutes
- **THEN** an alert fires
#### Scenario: App down alert
- **WHEN** an app returns zero healthy responses for 2 minutes
- **THEN** an alert fires
### Requirement: Log aggregation
Loki SHALL collect and index logs from all Docker containers.
#### Scenario: Search logs
- **WHEN** an operator searches logs in Grafana
- **THEN** they can filter by container name, log level, and time range

View file

@ -0,0 +1,62 @@
## 1. Health Endpoints
- [ ] 1.1 Add `/health` API route to Journal — checks DB with a simple query, returns status JSON
- [ ] 1.2 Add `/health` endpoint to Planner server.ts — checks DB connectivity
- [ ] 1.3 Update Docker healthcheck in docker-compose.yml to use `/health` instead of `pg_isready`
- [ ] 1.4 Add health routes to route configs (journal routes.ts)
## 2. Structured Logging
- [ ] 2.1 Add `pino` dependency to both apps
- [ ] 2.2 Create `apps/journal/app/lib/logger.server.ts` — Pino instance, JSON in prod, pretty in dev
- [ ] 2.3 Create `apps/planner/app/lib/logger.server.ts` — same pattern
- [ ] 2.4 Replace key `console.log`/`console.error` calls with logger (auth, DB errors, session lifecycle)
- [ ] 2.5 Add request logging middleware to Planner server.ts (method, path, status, duration)
## 3. Prometheus Metrics
- [ ] 3.1 Add `prom-client` dependency to both apps
- [ ] 3.2 Create metrics utility for Journal — default metrics + http_request_duration_seconds histogram
- [ ] 3.3 Create metrics utility for Planner — default metrics + http histogram + planner_active_sessions + planner_connected_clients gauges + brouter_request_duration_seconds histogram
- [ ] 3.4 Add `/metrics` endpoint to Journal (API route)
- [ ] 3.5 Add `/metrics` endpoint to Planner server.ts
- [ ] 3.6 Wire request duration tracking into request handlers
## 4. Caddy Access Logs
- [ ] 4.1 Enable Caddy structured JSON access logging in Caddyfile (log directive)
## 5. Monitoring Stack (Docker Compose)
- [ ] 5.1 Add Prometheus container to docker-compose.yml with scrape config for journal:3000 and planner:3001
- [ ] 5.2 Add Loki container to docker-compose.yml with Docker logging driver config
- [ ] 5.3 Add Grafana container to docker-compose.yml with provisioned data sources (Prometheus + Loki)
- [ ] 5.4 Create `infrastructure/prometheus/prometheus.yml` — scrape config
- [ ] 5.5 Create `infrastructure/loki/loki-config.yml` — retention, storage
- [ ] 5.6 Create `infrastructure/grafana/provisioning/` — data sources + dashboard provider config
- [ ] 5.7 Expose Grafana via Caddy on grafana.trails.cool with basic auth
## 6. Dashboards
- [ ] 6.1 Create overview dashboard JSON — request rate, error rate, latency p50/p95/p99
- [ ] 6.2 Create planner dashboard JSON — active sessions, connected clients, BRouter latency
- [ ] 6.3 Create infrastructure dashboard JSON — node_exporter or cAdvisor metrics (CPU, memory, disk)
## 7. Alerting
- [ ] 7.1 Configure Grafana alert rule: disk usage > 80%
- [ ] 7.2 Configure Grafana alert rule: app health check failing for 2 min
- [ ] 7.3 Configure Grafana alert rule: error rate > 5% for 5 min
- [ ] 7.4 Set up alert notification channel (email or webhook)
## 8. DNS & Terraform
- [ ] 8.1 Add grafana.trails.cool DNS record
- [ ] 8.2 Add Grafana basic auth credentials to deploy secrets
## 9. Verify
- [ ] 9.1 Test /health endpoints locally — verify OK and degraded responses
- [ ] 9.2 Test /metrics endpoints — verify Prometheus format output
- [ ] 9.3 Test Grafana dashboards load with data after deploy
- [ ] 9.4 Test alert fires when simulating disk full condition

View file

@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-03-25

View file

@ -0,0 +1,104 @@
## Context
Security audit findings:
- ✅ Solid: cookies (httpOnly/secure/sameSite), auth (passkeys/JWT), SQL
injection safe (Drizzle), XSS safe (React), rate limiting on Planner API
- ❌ Missing: security headers, CSP, secret scanning, dependency auditing,
non-root Docker, scanner blocking, vulnerability disclosure policy
- ⚠️ Weak dev-secret fallbacks in code (`"dev-jwt-secret-change-in-production"`)
## Goals / Non-Goals
**Goals:**
- Security headers on all responses via Caddy
- Block known scanner paths at Caddy level (before hitting Node)
- Gitleaks in CI to catch committed secrets
- pnpm audit in CI to catch vulnerable dependencies
- Dependabot for automated dependency updates
- Non-root user in all Docker containers
- SECURITY.md for responsible disclosure
**Non-Goals:**
- WAF (web application firewall) — overkill for current scale
- CSRF tokens — current SameSite + JSON API pattern is sufficient
- Container image scanning (Trivy etc.) — add later
- mTLS between containers — Docker network is trusted
- Removing dev-secret fallbacks — they only activate when env vars are missing,
which is a deployment bug, not a security issue
## Decisions
### D1: Security headers in Caddyfile
Add a `header` block to both site configs:
```
header {
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
X-Content-Type-Options "nosniff"
X-Frame-Options "DENY"
Referrer-Policy "strict-origin-when-cross-origin"
Permissions-Policy "camera=(), microphone=(), geolocation=()"
}
```
CSP is tricky with React Router's inline scripts. Use a permissive initial
policy: `default-src 'self'; script-src 'self' 'unsafe-inline'; style-src
'self' 'unsafe-inline'; img-src 'self' data: https://*.tile.openstreetmap.org;
connect-src 'self' wss: https://*.sentry.io https://*.ingest.de.sentry.io;`
### D2: Scanner path blocking in Caddy
Add a `respond` directive before `reverse_proxy` that returns 403 for
common scanner paths:
```
@scanners path /.env* /.git* /wp-* /admin* /config.* /backup* /api/v1*
respond @scanners 403
```
This prevents the app from even seeing these requests — no Sentry noise,
no wasted compute.
### D3: Gitleaks via GitHub Action
Add `gitleaks/gitleaks-action@v2` as a CI step. It scans commits for secrets
(API keys, tokens, passwords). Runs on every PR. Add `.gitleaks.toml` for
any false-positive allowlists (e.g., Sentry DSNs are public).
### D4: pnpm audit in CI
Add `pnpm audit --audit-level=high` as a CI step. Fail only on high/critical
vulnerabilities to avoid noise from low-severity advisories. Run alongside
lint/typecheck.
### D5: Dependabot for pnpm
Create `.github/dependabot.yml` targeting pnpm ecosystem. Weekly schedule,
grouped by update type. Auto-creates PRs for outdated dependencies.
### D6: Non-root user in Dockerfiles
Add to the runtime stage of each Dockerfile:
```dockerfile
RUN addgroup --system app && adduser --system --ingroup app app
USER app
```
This limits the impact of container escape vulnerabilities. BRouter needs
read access to segments, which works since the volume is readable.
### D7: Fail2ban on the server
Install `fail2ban` on the Hetzner server to block IPs after repeated SSH
failures or scanner patterns. Configuration via Terraform user-data or
manual setup documented in deployment docs.
## Risks / Trade-offs
- **CSP with inline scripts** → React Router uses inline scripts for hydration.
Must use `'unsafe-inline'` for `script-src`. Can tighten with nonce-based
CSP later if React Router supports it.
- **Scanner blocking is path-based** → Won't catch all bots, but eliminates
the most common credential-harvesting patterns.
- **Non-root may break file permissions** → Test that the app can still read
build output and node_modules. Should work since COPY sets root ownership
by default and files are world-readable.

View file

@ -0,0 +1,42 @@
## Why
A security audit found several gaps: no security headers (HSTS, CSP, etc.),
no secret scanning in CI, no dependency vulnerability scanning, Docker
containers running as root, and bot scanners probing for `.env` files with
no filtering. The cookie and auth setup is solid, but the infrastructure and
CI layers need hardening.
## What Changes
- **Security headers**: HSTS, X-Content-Type-Options, X-Frame-Options,
Referrer-Policy, Permissions-Policy via Caddyfile
- **Content-Security-Policy**: Restrict script/style/font sources
- **Gitleaks**: Secret scanning in CI to prevent credential leaks
- **Dependency auditing**: `pnpm audit` in CI + Dependabot for automated updates
- **Docker hardening**: Non-root user in all Dockerfiles
- **Bot/scanner blocking**: Caddy matcher to reject known scanner paths
(`.env`, `.git`, `wp-config`, etc.) with 403 before hitting the app
- **Fail2ban or equivalent**: Rate-limit SSH brute force and scanner IPs
at the server level
- **SECURITY.md**: Vulnerability disclosure policy
## Capabilities
### New Capabilities
- `security-hardening`: Security headers, CI secret/dependency scanning,
Docker non-root, scanner blocking, server-level rate limiting
### Modified Capabilities
- `infrastructure`: Caddy security headers + scanner blocking, Docker non-root,
Terraform firewall adjustments, CI scanning steps
## Impact
- **Caddyfile**: Security headers + scanner path blocking
- **Dockerfiles**: Add non-root user (journal, planner, brouter)
- **CI**: Add gitleaks step, pnpm audit step
- **Repo**: Add `.gitleaks.toml`, `dependabot.yml`, `SECURITY.md`
- **Server**: Optional fail2ban or UFW configuration
- **Dependencies**: None for app code; gitleaks is a CI action

View file

@ -0,0 +1,23 @@
## MODIFIED Requirements
### Requirement: Service configuration
Each service SHALL be configured with security best practices including non-root execution and security headers.
#### Scenario: Caddy security headers
- **WHEN** Caddy proxies a request
- **THEN** it adds HSTS, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, and Permissions-Policy headers
#### Scenario: Caddy scanner blocking
- **WHEN** a request matches known scanner paths (.env, .git, wp-config, etc.)
- **THEN** Caddy returns 403 without forwarding to the application
### Requirement: CI/CD pipeline
GitHub Actions SHALL include security scanning steps alongside build and test.
#### Scenario: Gitleaks scan
- **WHEN** a PR is opened
- **THEN** gitleaks scans for committed secrets
#### Scenario: Dependency audit
- **WHEN** CI runs
- **THEN** pnpm audit checks for high/critical vulnerabilities

View file

@ -0,0 +1,59 @@
## ADDED Requirements
### Requirement: Security response headers
All HTTP responses SHALL include security headers to protect against common web attacks.
#### Scenario: HSTS header
- **WHEN** a browser receives a response from trails.cool
- **THEN** the response includes `Strict-Transport-Security: max-age=31536000; includeSubDomains; preload`
#### Scenario: Content sniffing prevention
- **WHEN** a browser receives a response
- **THEN** the response includes `X-Content-Type-Options: nosniff`
#### Scenario: Clickjacking prevention
- **WHEN** a browser receives a response
- **THEN** the response includes `X-Frame-Options: DENY`
### Requirement: Scanner path blocking
Known vulnerability scanner paths SHALL be blocked at the reverse proxy level before reaching the application.
#### Scenario: Env file scanner
- **WHEN** a request is made to `/.env`, `/.env.local`, `/.env.prod`, or similar paths
- **THEN** Caddy returns 403 without forwarding to the application
#### Scenario: Git config scanner
- **WHEN** a request is made to `/.git/config` or `/.git/HEAD`
- **THEN** Caddy returns 403 without forwarding to the application
### Requirement: Secret scanning in CI
The CI pipeline SHALL scan commits for accidentally committed secrets.
#### Scenario: Secret detected
- **WHEN** a PR contains a committed API key, token, or password
- **THEN** the CI pipeline fails with a clear message indicating the leaked secret
#### Scenario: Known public values allowed
- **WHEN** a known-public value (e.g., Sentry DSN) is committed
- **THEN** gitleaks allows it via the `.gitleaks.toml` allowlist
### Requirement: Dependency vulnerability scanning
The CI pipeline SHALL check for known vulnerabilities in dependencies.
#### Scenario: High severity vulnerability
- **WHEN** a dependency has a high or critical vulnerability advisory
- **THEN** the CI pipeline fails
### Requirement: Non-root Docker containers
Application containers SHALL run as a non-root user.
#### Scenario: Container user
- **WHEN** a container starts
- **THEN** the process runs as a non-root user (not UID 0)
### Requirement: Vulnerability disclosure policy
The repository SHALL include a SECURITY.md with responsible disclosure instructions.
#### Scenario: Security contact
- **WHEN** a security researcher finds a vulnerability
- **THEN** SECURITY.md provides clear instructions for reporting it

View file

@ -0,0 +1,48 @@
## 1. Caddy Security Headers
- [ ] 1.1 Add security headers block to Caddyfile: HSTS, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, Permissions-Policy
- [ ] 1.2 Add Content-Security-Policy header (permissive initial policy allowing React Router inline scripts, OSM tiles, Sentry, WebSocket)
- [ ] 1.3 Test headers with securityheaders.com or curl after deploy
## 2. Scanner Blocking
- [ ] 2.1 Add Caddy path matcher for common scanner targets (.env*, .git*, wp-*, admin*, config.*, backup*)
- [ ] 2.2 Return 403 for matched paths before reverse_proxy
- [ ] 2.3 Verify scanner paths return 403, normal paths still work
## 3. CI Secret Scanning
- [ ] 3.1 Add gitleaks/gitleaks-action@v2 step to CI workflow
- [ ] 3.2 Create .gitleaks.toml with allowlist for Sentry DSNs (public values)
- [ ] 3.3 Verify gitleaks passes on current codebase
## 4. Dependency Auditing
- [ ] 4.1 Add `pnpm audit --audit-level=high` step to CI workflow
- [ ] 4.2 Create .github/dependabot.yml for weekly pnpm dependency updates
- [ ] 4.3 Verify pnpm audit passes on current codebase
## 5. Docker Hardening
- [ ] 5.1 Add non-root user to Journal Dockerfile runtime stage
- [ ] 5.2 Add non-root user to Planner Dockerfile runtime stage
- [ ] 5.3 Add non-root user to BRouter Dockerfile
- [ ] 5.4 Test containers start and serve correctly as non-root
## 6. Server Hardening
- [ ] 6.1 Install and configure fail2ban on Hetzner server (SSH protection)
- [ ] 6.2 Configure UFW firewall (allow 22, 80, 443 only)
- [ ] 6.3 Document server hardening steps in deployment docs
## 7. Documentation
- [ ] 7.1 Create SECURITY.md with vulnerability disclosure policy and security contact
- [ ] 7.2 Update privacy manifest with security practices documented
## 8. Verify
- [ ] 8.1 Run securityheaders.com scan on trails.cool and planner.trails.cool
- [ ] 8.2 Verify scanner paths return 403 in production
- [ ] 8.3 Verify CI passes with all new scanning steps
- [ ] 8.4 Verify Docker containers run as non-root