8396041c26
poi-index shipped and is live in production (self-hosted planet POI index, 8.4M rows serving /api/pois; Overpass removed from the Planner). Archive the change and apply its spec deltas: - new capability: poi-index - osm-poi-overlays: POIs from the instance index, not Overpass - rate-limiting: /api/pois limit replaces the Overpass proxy limit - infrastructure: POI extract (BRouter host) + import (flagship) components Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
213 lines
11 KiB
Markdown
213 lines
11 KiB
Markdown
## Purpose
|
|
|
|
Server provisioning on Hetzner, Docker Compose deployment, CI/CD pipelines, database and BRouter management, TLS, Sentry, Grafana, and monitoring stack for the flagship instance.
|
|
## Requirements
|
|
### Requirement: Terraform Hetzner provisioning
|
|
Infrastructure SHALL be provisioned on Hetzner Cloud using Terraform with the Hetzner provider.
|
|
|
|
#### Scenario: Provision server
|
|
- **WHEN** `terraform apply` is run
|
|
- **THEN** a Hetzner cx23 server (2 vCPU, 4 GB RAM, 40 GB SSD) is created with Docker installed
|
|
|
|
### Requirement: Docker Compose deployment
|
|
All services SHALL be deployed via Docker Compose, including Grafana, Prometheus, and Loki for the flagship instance.
|
|
|
|
#### Scenario: Monitoring stack starts
|
|
- **WHEN** `docker compose up -d` is run
|
|
- **THEN** Grafana, Prometheus, Loki, Promtail, postgres-exporter, node-exporter, and cAdvisor containers start alongside the application containers
|
|
|
|
### Requirement: Service configuration
|
|
Each service SHALL be configured via environment variables defined in Docker Compose, with security best practices including non-root execution and security headers.
|
|
|
|
#### Scenario: Journal configuration
|
|
- **WHEN** the Journal container starts
|
|
- **THEN** it reads DOMAIN, DATABASE_URL, PLANNER_URL, JWT_SECRET, SESSION_SECRET, INTEGRATION_SECRET, WAHOO_* credentials, and SENTRY_DSN from environment variables
|
|
|
|
#### Scenario: Planner configuration
|
|
- **WHEN** the Planner container starts
|
|
- **THEN** it reads BROUTER_URL, DATABASE_URL, and INTEGRATION_SECRET from environment variables
|
|
|
|
#### Scenario: Caddy security headers
|
|
- **WHEN** Caddy proxies a request
|
|
- **THEN** it adds HSTS, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, and Permissions-Policy headers
|
|
|
|
#### Scenario: Caddy scanner blocking
|
|
- **WHEN** a request matches known scanner paths (.env, .git, wp-config, etc.)
|
|
- **THEN** Caddy returns 403 without forwarding to the application
|
|
|
|
### Requirement: PostgreSQL with PostGIS
|
|
The database SHALL be PostgreSQL with the PostGIS extension for spatial queries.
|
|
|
|
#### Scenario: PostGIS available
|
|
- **WHEN** the PostgreSQL container starts
|
|
- **THEN** the PostGIS extension is available and can be enabled with `CREATE EXTENSION postgis`
|
|
|
|
### Requirement: BRouter segment management
|
|
The infrastructure SHALL support downloading and updating planet-wide RD5 segments from brouter.de to the dedicated BRouter host. Segment files SHALL live under `~trails/brouter/segments/` on the dedicated host and SHALL be owned by the `trails` user.
|
|
|
|
#### Scenario: Download segments
|
|
- **WHEN** the segment download script runs on the dedicated host as the `trails` user
|
|
- **THEN** all planet-wide RD5 files referenced by the tile list are downloaded to `~trails/brouter/segments/`, skipping files that already exist
|
|
|
|
#### Scenario: Segment update
|
|
- **WHEN** an operator re-runs the segment download script
|
|
- **THEN** outdated or missing RD5 files are re-fetched from brouter.de and the BRouter container is restarted
|
|
|
|
### Requirement: CI/CD pipeline
|
|
GitHub Actions SHALL use separate workflows for app deployment, infrastructure deployment, and BRouter deployment, with secrets decrypted from a SOPS-encrypted file.
|
|
|
|
#### Scenario: App deployment
|
|
- **WHEN** code changes are pushed to main in apps/ or packages/
|
|
- **THEN** the cd-apps workflow builds Docker images, pushes to ghcr.io, and deploys app containers to the flagship host
|
|
|
|
#### Scenario: Infrastructure deployment
|
|
- **WHEN** changes are pushed to main in infrastructure/
|
|
- **THEN** the cd-infra workflow copies configs and restarts infrastructure services on the flagship host without rebuilding app images and without touching the BRouter host
|
|
|
|
#### Scenario: BRouter deployment
|
|
- **WHEN** changes are pushed to main in docker/brouter/ or the BRouter host compose config
|
|
- **THEN** the cd-brouter workflow SSHes as the `trails` user into the dedicated BRouter host using `BROUTER_DEPLOY_HOST` / `BROUTER_DEPLOY_SSH_KEY` and runs `docker compose up -d` in `~trails/brouter/`
|
|
|
|
#### Scenario: Secret decryption at deploy time
|
|
- **WHEN** `cd-apps.yml` runs
|
|
- **THEN** `secrets.app.env` is decrypted and injected into the Journal and Planner containers
|
|
- **WHEN** `cd-infra.yml` runs
|
|
- **THEN** both `secrets.app.env` and `secrets.infra.env` are decrypted and merged for infrastructure services
|
|
|
|
#### Scenario: Gitleaks scan
|
|
- **WHEN** a PR is opened
|
|
- **THEN** gitleaks scans for committed secrets
|
|
|
|
#### Scenario: Dependency audit
|
|
- **WHEN** CI runs
|
|
- **THEN** pnpm audit checks for high/critical vulnerabilities
|
|
|
|
### Requirement: Backup strategy
|
|
The infrastructure SHALL include daily backups of the PostgreSQL database.
|
|
|
|
#### Scenario: Daily backup
|
|
- **WHEN** the daily backup cron runs
|
|
- **THEN** a PostgreSQL dump is uploaded to the Hetzner Storage Box
|
|
|
|
### Requirement: Domain and TLS
|
|
The infrastructure SHALL configure DNS and TLS for trails.cool and planner.trails.cool.
|
|
|
|
#### Scenario: HTTPS access
|
|
- **WHEN** a user navigates to https://trails.cool
|
|
- **THEN** the connection is secured with a valid TLS certificate
|
|
|
|
### Requirement: Sentry error tracking
|
|
The system SHALL enrich Sentry events with user and session context, use route-aware tracing, and prevent source maps from being served to clients.
|
|
|
|
#### Scenario: Journal error includes user context
|
|
- **WHEN** an authenticated Journal user triggers an error
|
|
- **THEN** the Sentry event SHALL include the user's ID and username
|
|
|
|
#### Scenario: Journal error without user context
|
|
- **WHEN** an unauthenticated visitor triggers an error
|
|
- **THEN** the Sentry event SHALL have no user context (Sentry.setUser(null))
|
|
|
|
#### Scenario: Planner error includes session ID
|
|
- **WHEN** an error occurs during a Planner session
|
|
- **THEN** the Sentry event SHALL include a `session_id` tag with the active session ID
|
|
|
|
#### Scenario: Route-level performance traces
|
|
- **WHEN** a user navigates between routes in either app
|
|
- **THEN** Sentry SHALL create a transaction span named after the route pattern (e.g., `/routes/:id`)
|
|
|
|
#### Scenario: Source maps not served to clients
|
|
- **WHEN** a client requests a `.map` file from the production server
|
|
- **THEN** the server SHALL return 404 (source maps are uploaded to Sentry during build, not shipped in the bundle)
|
|
|
|
### Requirement: Grafana authentication
|
|
Grafana SHALL authenticate users via GitHub OAuth, restricted to the trails-cool GitHub organization.
|
|
|
|
#### Scenario: GitHub OAuth login
|
|
- **WHEN** a user navigates to grafana.internal.trails.cool
|
|
- **THEN** they are redirected to GitHub for authentication and granted access if they are a member of the trails-cool organization
|
|
|
|
#### Scenario: No password-based login
|
|
- **WHEN** Grafana is deployed
|
|
- **THEN** the login form is disabled and only GitHub OAuth is available
|
|
|
|
### Requirement: Grafana database access
|
|
The `grafana_reader` PostgreSQL role SHALL have SELECT access to the `pgboss` schema for job queue observability.
|
|
|
|
#### Scenario: Grant access on deploy
|
|
- **WHEN** the infrastructure deploy runs
|
|
- **THEN** `grafana_reader` is granted `USAGE` on the `pgboss` schema and `SELECT` on all tables in it
|
|
|
|
### Requirement: Monitoring stack
|
|
The Grafana Service Health dashboard SHALL include a job queue health panel.
|
|
|
|
#### Scenario: Job queue panel displays metrics
|
|
- **WHEN** a user views the Service Health dashboard
|
|
- **THEN** they see a panel showing job queue depth, completed jobs per hour, and failed jobs
|
|
- **AND** failed jobs are highlighted for investigation
|
|
|
|
### Requirement: Metrics collection
|
|
Prometheus SHALL scrape metrics from all application and infrastructure services.
|
|
|
|
#### Scenario: Exporter targets
|
|
- **WHEN** Prometheus is running
|
|
- **THEN** it scrapes metrics from journal (/api/metrics), planner (/metrics), postgres-exporter, node-exporter, cAdvisor, and Caddy (:2019)
|
|
|
|
#### Scenario: pg_stat_statements
|
|
- **WHEN** postgres-exporter scrapes PostgreSQL
|
|
- **THEN** slow query metrics from pg_stat_statements are exposed with query text via a custom queries config
|
|
|
|
### Requirement: Container log shipping
|
|
Promtail SHALL scrape all Docker container logs and push them to Loki for querying in Grafana.
|
|
|
|
#### Scenario: Logs visible in Grafana
|
|
- **WHEN** a container writes to stdout or stderr
|
|
- **THEN** the log line is available in Grafana Explore via Loki with container name labels
|
|
|
|
### Requirement: Caddy access logging
|
|
Caddy SHALL emit structured JSON access logs for all requests.
|
|
|
|
#### Scenario: Access log emitted
|
|
- **WHEN** any HTTP request passes through Caddy
|
|
- **THEN** a JSON log line with remote IP, method, path, status, and duration is written to stdout
|
|
|
|
### Requirement: Private network between flagship and BRouter hosts
|
|
The flagship host and the dedicated BRouter host SHALL be joined on a Hetzner vSwitch in the same datacenter. All traffic between Planner and BRouter SHALL traverse this private network.
|
|
|
|
#### Scenario: vSwitch reachability
|
|
- **WHEN** the flagship host issues a request to the BRouter host's vSwitch IP on the BRouter service port with a valid `X-BRouter-Auth` header
|
|
- **THEN** the request succeeds over the private network without traversing the public internet
|
|
|
|
#### Scenario: No public BRouter exposure
|
|
- **WHEN** Hetzner Cloud firewall rules or equivalent host firewall rules are inspected
|
|
- **THEN** no rule allows inbound traffic to the BRouter service port from any public IP
|
|
|
|
### Requirement: Non-root deploy user on the BRouter host
|
|
The BRouter host SHALL be administered by the trails.cool project through a non-root `trails` user that is a member of the `docker` group. The CD workflow SHALL NOT require sudo or root SSH access on this host.
|
|
|
|
#### Scenario: Deploy with trails user
|
|
- **WHEN** the cd-brouter workflow connects to the BRouter host
|
|
- **THEN** it authenticates as `trails` and successfully runs `docker compose` commands without invoking sudo
|
|
|
|
#### Scenario: Scoped ownership
|
|
- **WHEN** files are created by the deploy or segment-download scripts
|
|
- **THEN** they live under `~trails/brouter/` and are owned by `trails:trails`
|
|
|
|
### Requirement: POI extract pipeline on the BRouter host
|
|
The BRouter host SHALL run a scheduled job (monthly by default) that downloads an OSM data file, filters it to the planner's POI category selectors with osmium, and publishes the resulting artifact with a checksum-and-timestamp manifest at a vSwitch-only URL via the existing Caddy sidecar. Transient working data SHALL live under the `trails` user's directory and be cleaned up after each run.
|
|
|
|
#### Scenario: Monthly extract published
|
|
- **WHEN** the scheduled extract job completes
|
|
- **THEN** a fresh artifact and manifest are available to the flagship over the vSwitch
|
|
- **AND** the planet working files are removed from disk
|
|
|
|
#### Scenario: Artifact not publicly reachable
|
|
- **WHEN** a host outside the vSwitch requests the artifact URL
|
|
- **THEN** the request is refused
|
|
|
|
### Requirement: POI import job on the flagship
|
|
The flagship SHALL run a scheduled import job that fetches the artifact over the vSwitch, verifies its checksum, loads it into the staging table, and performs the guarded atomic swap. Job outcome and index age SHALL be visible in the monitoring stack.
|
|
|
|
#### Scenario: Import failure is visible
|
|
- **WHEN** an import run fails (fetch, checksum, load, or guard)
|
|
- **THEN** the failure is recorded in metrics/logs and surfaces on the Grafana planner dashboard
|
|
|