trails/infrastructure
Ullrich Schäfer 963902514b
Failover across multiple Overpass upstreams
Public Overpass instances go bad without warning — we just lived
through a day where overpass.private.coffee (our single upstream)
returned 504s after 60s while lz4.overpass-api.de served the same
query in 0.6s. Survive that by trying a list of upstreams in order
until one responds usefully.

Server changes:
- OVERPASS_URLS env — comma-separated list, tried in order. Falls
  back to OVERPASS_URL for backward compat, then to a built-in
  default pair (lz4.overpass-api.de, overpass-api.de).
- Per-upstream timeout of 10s via AbortSignal.timeout, so a saturated
  instance fails over in seconds, not minutes.
- Client abort propagation via AbortSignal.any — if the browser
  cancels (e.g. user panned the map), the in-flight upstream fetch is
  aborted too. Stops wasting upstream capacity on requests nobody
  wants.
- Rate-limited-body detection (Overpass returns 200 with a
  `rate_limited` marker when throttling) triggers failover.
- Per-upstream labels on overpass_upstream_requests_total and
  overpass_upstream_duration_seconds so the dashboard can break out
  health + latency by instance. Histogram buckets extended to 30s.

Compose: OVERPASS_URLS passthrough with the default pair hard-wired,
overridable via SOPS.

Tests: 8 cases covering the new fetchWithFailover helper — happy
path, 5xx failover, rate_limited failover, network error failover,
timeout tagging, all-upstreams-fail, client abort stops the loop,
per-attempt latency observation.

Follow-ups left out of scope:
- Client-side debounce (UI concern).
- Moving `.observe()` after body read for accuracy in measurement.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 07:36:10 +02:00
..
grafana Update pgboss column names for v12 (snake_case) 2026-04-20 16:49:43 +02:00
loki Pass Grafana env vars to Caddy container, fix Loki compactor config 2026-03-27 13:53:56 +01:00
postgres Add pg-boss background job queue with session expiry 2026-04-13 21:17:24 +02:00
prometheus Reduce Prometheus scrape interval from 15s to 5s 2026-04-03 10:15:08 +01:00
promtail Fix journal health check and improve log collection 2026-04-12 15:00:00 +02:00
scripts Add infrastructure: Terraform, Docker Compose, BRouter, CD pipeline (#2) 2026-03-22 13:32:01 +01:00
terraform Harden SSH: disable passwords, forwarding, add keepalive 2026-03-29 22:32:30 +02:00
.env.example Failover across multiple Overpass upstreams 2026-04-21 07:36:10 +02:00
Caddyfile Allow Planner to connect to Journal in CSP 2026-03-27 21:34:41 +01:00
docker-compose.yml Failover across multiple Overpass upstreams 2026-04-21 07:36:10 +02:00
secrets.app.env Enable demo-activity-bot on prod 2026-04-19 10:34:19 +02:00
secrets.infra.env Add GRAFANA_SERVICE_TOKEN to encrypted infra secrets 2026-03-29 19:49:29 +02:00