Failover across multiple Overpass upstreams

Public Overpass instances go bad without warning — we just lived
through a day where overpass.private.coffee (our single upstream)
returned 504s after 60s while lz4.overpass-api.de served the same
query in 0.6s. Survive that by trying a list of upstreams in order
until one responds usefully.

Server changes:
- OVERPASS_URLS env — comma-separated list, tried in order. Falls
  back to OVERPASS_URL for backward compat, then to a built-in
  default pair (lz4.overpass-api.de, overpass-api.de).
- Per-upstream timeout of 10s via AbortSignal.timeout, so a saturated
  instance fails over in seconds, not minutes.
- Client abort propagation via AbortSignal.any — if the browser
  cancels (e.g. user panned the map), the in-flight upstream fetch is
  aborted too. Stops wasting upstream capacity on requests nobody
  wants.
- Rate-limited-body detection (Overpass returns 200 with a
  `rate_limited` marker when throttling) triggers failover.
- Per-upstream labels on overpass_upstream_requests_total and
  overpass_upstream_duration_seconds so the dashboard can break out
  health + latency by instance. Histogram buckets extended to 30s.

Compose: OVERPASS_URLS passthrough with the default pair hard-wired,
overridable via SOPS.

Tests: 8 cases covering the new fetchWithFailover helper — happy
path, 5xx failover, rate_limited failover, network error failover,
timeout tagging, all-upstreams-fail, client abort stops the loop,
per-attempt latency observation.

Follow-ups left out of scope:
- Client-side debounce (UI concern).
- Moving `.observe()` after body read for accuracy in measurement.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-04-21 07:36:10 +02:00
parent d5309b480c
commit 963902514b
No known key found for this signature in database
GPG key ID: A32FF691A0F752D9
6 changed files with 358 additions and 25 deletions

View file

@ -7,6 +7,11 @@ JWT_SECRET=change-me
S3_ACCESS_KEY=
S3_SECRET_KEY=
# Overpass upstream failover. Comma-separated list tried in order;
# first 2xx response wins. Falls back to the single-URL OVERPASS_URL
# if this is unset, and to a built-in default list if neither is set.
# OVERPASS_URLS=https://lz4.overpass-api.de/api/interpreter,https://overpass-api.de/api/interpreter
# Demo-activity-bot. Only enable in prod. When true, the journal worker
# bootstraps a demo user and generates public demo routes + activities
# every 90 min. Retention is in days (default 14).

View file

@ -55,7 +55,12 @@ services:
restart: unless-stopped
environment:
BROUTER_URL: http://brouter:17777
OVERPASS_URL: https://overpass.private.coffee/api/interpreter
# Ordered failover list for the Overpass proxy. The code defaults
# to the same pair if unset; declaring it here makes the prod
# upstream explicit and easy to reshuffle via env when a
# community instance misbehaves. Override per-env with
# OVERPASS_URLS in the SOPS file.
OVERPASS_URLS: ${OVERPASS_URLS:-https://lz4.overpass-api.de/api/interpreter,https://overpass-api.de/api/interpreter}
DATABASE_URL: postgres://trails:${POSTGRES_PASSWORD:-trails}@postgres:5432/trails
NODE_ENV: production
PORT: 3001