diff --git a/apps/journal/app/routes/legal.privacy.tsx b/apps/journal/app/routes/legal.privacy.tsx index f2b10fe..8ee7b77 100644 --- a/apps/journal/app/routes/legal.privacy.tsx +++ b/apps/journal/app/routes/legal.privacy.tsx @@ -253,16 +253,16 @@ export default function PrivacyPage() { .
  • - Overpass API (POI- und Wegbelag-Daten) – - POI-Abfragen laufen serverseitig über unsere eigene Route{" "} - /api/overpass. Für die Oberflächen-/Wegtyp-Auswertung - importierter Aktivitäten und Routen fragt ein Hintergrund-Job das - Begrenzungsrechteck (Bounding Box) der Route bei Overpass ab, um sie - den OpenStreetMap-Wegen zuzuordnen. In beiden Fällen sieht der - Upstream nur unsere Server-IP, nicht die Ihrer Nutzer:innen. - Aktueller Upstream: overpass.private.coffee bzw. die - öffentlichen Overpass-Instanzen, die ohne Query-Logs arbeiten. Eine - selbst gehostete Instanz ist geplant. + Overpass API (Wegbelag-Daten) – Für die + Oberflächen-/Wegtyp-Auswertung importierter Aktivitäten und Routen + fragt ein Hintergrund-Job das Begrenzungsrechteck (Bounding Box) der + Route bei Overpass ab, um sie den OpenStreetMap-Wegen zuzuordnen. + Der Upstream sieht nur unsere Server-IP, nicht die Ihrer + Nutzer:innen; verwendet werden öffentliche Overpass-Instanzen, die + ohne Query-Logs arbeiten. Die POI-Overlays des Planners nutzen + Overpass nicht mehr – sie werden aus unserem eigenen, + selbst gehosteten POI-Index (/api/pois) bedient, sodass + dabei keine Daten an Dritte gehen.
  • Föderation (ActivityPub) – Wenn Ihr Profil auf{" "} @@ -339,9 +339,11 @@ export default function PrivacyPage() { details, no IPs/cookies); OpenStreetMap tile servers (your IP and user-agent, directly from your browser, to load map tiles); Overpass (via our server, so upstream only sees our server — - for POI lookups and, for the surface/waytype breakdown of imported - activities and routes, a background job that sends the route’s - bounding box to match it against OpenStreetMap ways); BRouter + for the surface/waytype breakdown of imported activities and routes, a + background job that sends the route’s bounding box to match it + against OpenStreetMap ways; the Planner’s POI overlays no longer + use Overpass — they are served from our own self-hosted POI index at{" "} + /api/pois); BRouter (self-hosted, no third party involved); SMTP provider (your email address for magic link / welcome mail); Wahoo (only when you opt in: OAuth tokens for sync, plus route diff --git a/docs/architecture.md b/docs/architecture.md index 5be933a..ef47b4a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -790,8 +790,12 @@ Base layers: Overlays (toggleable): - **OpenCampingMap** (campsites, shelters — essential for bikepacking) -- **POI overlay**: Water points, shelters, bike repair stations - (sourced from OSM Overpass API or pre-cached) +- **POI overlay**: Water points, shelters, bike repair stations, etc. Served + from the instance's own `planner.pois` index via `/api/pois` — a PostGIS + table built monthly from an OSM extract (osmium filter on the BRouter host → + guarded atomic import on the flagship). No third-party POI service is + contacted; viewport coordinates never leave trails.cool. See the + `poi-index` change and `infrastructure/brouter-host/poi-extract/`. - **Waymarked Trails** (hiking/cycling trail networks) Implementation: Leaflet layer switcher with tile URLs. No API keys needed diff --git a/docs/ideas/self-host-overpass/README.md b/docs/ideas/self-host-overpass/README.md index 98b7ca4..11a141e 100644 --- a/docs/ideas/self-host-overpass/README.md +++ b/docs/ideas/self-host-overpass/README.md @@ -1,4 +1,4 @@ -# self-host-overpass (parked) +# self-host-overpass (parked — superseded for POIs) Full OpenSpec artifact set (`proposal.md`, `design.md`, `specs/`, `tasks.md`) for hosting our own Overpass API. Moved here from `openspec/changes/` so it @@ -7,11 +7,22 @@ under `openspec/changes/` when ready to implement. ## Status -**Parked.** The interim proxy work (`/api/overpass` → `overpass.private.coffee`, -see `apps/planner/app/routes/api.overpass.ts`) covers the day-one needs: -User-Agent compliance, same-origin check, rate limiting, server-side cache -with coalescing, bbox quantization, Grafana observability. That buys us time -before we need to own the upstream. +**Superseded for the Planner's POI overlays** by the +[`poi-index`](../../../openspec/changes/poi-index/) change (shipped), which +removes Overpass from the POI path entirely: instead of running the Overpass +software, a monthly osmium extract feeds a self-hosted PostGIS `planner.pois` +index served at `/api/pois`. That is lighter than this plan (no Overpass +engine, no diff replication, no RAM ceiling) and covers every current POI need +— all queries were simple tag-in-bbox lookups that never used Overpass QL. + +The former interim proxy (`/api/overpass` → `overpass.private.coffee`) has been +**removed**. + +**Still relevant if** we ever need genuine Overpass QL (arbitrary ad-hoc +queries, recursion, `around:`, set operations) that a precomputed category +index can't answer — that's the remaining reason to revive this. The Journal's +surface-breakdown backfill still queries public Overpass instances for way +geometry; owning that upstream is the other revive trigger. ## When to revive diff --git a/docs/roadmap.md b/docs/roadmap.md index 30d555a..949ff7f 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -83,7 +83,7 @@ See `docs/ideas/` for pre-spec explorations: - `mobile-activity-recording/` — GPS tracking and activity logging from a native app - `mobile-nearby-sync/` — BLE-based proximity discovery -- `self-host-overpass/` — Self-hosted Overpass API for POI overlays +- `self-host-overpass/` — Self-hosted Overpass API (superseded for POI overlays by the shipped `poi-index`; revive only if we need Overpass QL or to own the surface-backfill upstream) - `instance-administration.md` — admin role, registration toggle, suspend/ban, federation blocklists, reports - `social-interactions.md` — local likes + comments (the foundation the federated kudos/comments attach to) - `activity-participants.md` — tag co-riders on shared activities, confirm/decline, federated mentions diff --git a/infrastructure/brouter-host/poi-extract/README.md b/infrastructure/brouter-host/poi-extract/README.md index c59b307..8871980 100644 --- a/infrastructure/brouter-host/poi-extract/README.md +++ b/infrastructure/brouter-host/poi-extract/README.md @@ -79,6 +79,31 @@ poi-extract.service` then `journalctl --user -u poi-extract.service -f`. - **CPU/time**: a full planet filter is IO-bound and can take a few hours; the unit runs at `Nice=10` / idle IO priority so it doesn't disturb BRouter. +## Self-hosting + +The pipeline is **optional**. A fresh instance works with an empty or absent +`planner.pois` table: `/api/pois` returns an empty result / 503 and the map's +POI panel shows "unavailable" while every other overlay and tile keeps working +— nothing breaks. + +When you do want POIs, you don't need a second host or the planet. Both halves +run on one box, and `POI_PBF_URL` can point at any Geofabrik extract sized to +your region: + +```bash +# Extract just your region (much smaller download, seconds not hours): +POI_PBF_URL=https://download.geofabrik.de/europe/germany-latest.osm.pbf \ + ./poi-extract.sh + +# Then import it (first time: --bootstrap, since the live table is empty): +POI_ARTIFACT_BASE_URL=file-or-vswitch-url \ + ../../scripts/poi-import.sh --bootstrap +``` + +On a single-host instance, publish the artifact however is convenient (local +path, a static file server) and set `POI_ARTIFACT_BASE_URL` accordingly; the +two-host vSwitch topology is a flagship detail, not a requirement. + ## Dry run first Before the first planet run, validate the whole chain on a small extract: diff --git a/infrastructure/grafana/dashboards/planner.json b/infrastructure/grafana/dashboards/planner.json index 97aa026..2fc9a85 100644 --- a/infrastructure/grafana/dashboards/planner.json +++ b/infrastructure/grafana/dashboards/planner.json @@ -99,7 +99,7 @@ ] }, { - "title": "Overpass Upstream Health (5m success rate)", + "title": "POI Index Age", "type": "stat", "gridPos": { "h": 5, @@ -109,8 +109,83 @@ }, "targets": [ { - "expr": "(sum(rate(overpass_upstream_requests_total{status=~\"2..\"}[5m])) or vector(0)) / sum(rate(overpass_upstream_requests_total[5m]))", - "legendFormat": "success" + "expr": "poi_index_age_seconds", + "legendFormat": "age" + } + ], + "fieldConfig": { + "defaults": { + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null }, + { "color": "orange", "value": 3024000 }, + { "color": "red", "value": 3628800 } + ] + } + } + } + }, + { + "title": "POI Index Rows (total)", + "type": "stat", + "gridPos": { + "h": 5, + "w": 6, + "x": 6, + "y": 14 + }, + "targets": [ + { + "expr": "sum(poi_index_rows)", + "legendFormat": "rows" + } + ] + }, + { + "title": "Last Import Status", + "type": "stat", + "gridPos": { + "h": 5, + "w": 6, + "x": 12, + "y": 14 + }, + "targets": [ + { + "expr": "poi_import_last_status", + "legendFormat": "status" + } + ], + "fieldConfig": { + "defaults": { + "mappings": [ + { "type": "value", "options": { "0": { "text": "FAIL", "color": "red" }, "1": { "text": "OK", "color": "green" } } } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "red", "value": null }, + { "color": "green", "value": 1 } + ] + } + } + } + }, + { + "title": "POI API Error Rate (5m)", + "type": "stat", + "gridPos": { + "h": 5, + "w": 6, + "x": 18, + "y": 14 + }, + "targets": [ + { + "expr": "(sum(rate(poi_api_requests_total{status=~\"rate_limited|error\"}[5m])) or vector(0)) / clamp_min(sum(rate(poi_api_requests_total[5m])), 0.001)", + "legendFormat": "error ratio" } ], "fieldConfig": { @@ -121,76 +196,16 @@ "thresholds": { "mode": "absolute", "steps": [ - { "color": "red", "value": null }, - { "color": "orange", "value": 0.9 }, - { "color": "green", "value": 0.99 } + { "color": "green", "value": null }, + { "color": "orange", "value": 0.05 }, + { "color": "red", "value": 0.2 } ] } } } }, { - "title": "Overpass Cache Hit Ratio (5m)", - "type": "stat", - "gridPos": { - "h": 5, - "w": 6, - "x": 6, - "y": 14 - }, - "targets": [ - { - "expr": "(sum(rate(overpass_cache_events_total{result=\"hit\"}[5m])) or vector(0)) / sum(rate(overpass_cache_events_total[5m]))", - "legendFormat": "hit ratio" - } - ], - "fieldConfig": { - "defaults": { - "unit": "percentunit", - "min": 0, - "max": 1 - } - } - }, - { - "title": "Overpass Cache Size", - "type": "stat", - "gridPos": { - "h": 5, - "w": 6, - "x": 12, - "y": 14 - }, - "targets": [ - { - "expr": "overpass_cache_size", - "legendFormat": "entries" - } - ] - }, - { - "title": "Overpass Upstream p95", - "type": "stat", - "gridPos": { - "h": 5, - "w": 6, - "x": 18, - "y": 14 - }, - "targets": [ - { - "expr": "histogram_quantile(0.95, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))", - "legendFormat": "p95" - } - ], - "fieldConfig": { - "defaults": { - "unit": "s" - } - } - }, - { - "title": "Overpass Cache Events", + "title": "POI API Requests (5m by status)", "type": "timeseries", "gridPos": { "h": 7, @@ -200,28 +215,7 @@ }, "targets": [ { - "expr": "sum by (result) (rate(overpass_cache_events_total[5m]))", - "legendFormat": "{{result}}" - } - ], - "fieldConfig": { - "defaults": { - "unit": "ops" - } - } - }, - { - "title": "Overpass Upstream Status", - "type": "timeseries", - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 19 - }, - "targets": [ - { - "expr": "sum by (status) (rate(overpass_upstream_requests_total[5m]))", + "expr": "sum by (status) (rate(poi_api_requests_total[5m]))", "legendFormat": "{{status}}" } ], @@ -232,33 +226,20 @@ } }, { - "title": "Overpass Upstream Latency", + "title": "POI Index Rows by Category", "type": "timeseries", "gridPos": { "h": 7, - "w": 24, - "x": 0, - "y": 26 + "w": 12, + "x": 12, + "y": 19 }, "targets": [ { - "expr": "histogram_quantile(0.50, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))", - "legendFormat": "p50" - }, - { - "expr": "histogram_quantile(0.95, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))", - "legendFormat": "p95" - }, - { - "expr": "histogram_quantile(0.99, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))", - "legendFormat": "p99" + "expr": "poi_index_rows", + "legendFormat": "{{category}}" } - ], - "fieldConfig": { - "defaults": { - "unit": "s" - } - } + ] } ] } diff --git a/infrastructure/grafana/provisioning/alerting/alerts.yml b/infrastructure/grafana/provisioning/alerting/alerts.yml index cc9f2c0..b75a995 100644 --- a/infrastructure/grafana/provisioning/alerting/alerts.yml +++ b/infrastructure/grafana/provisioning/alerting/alerts.yml @@ -265,16 +265,16 @@ groups: annotations: summary: "Prometheus TSDB compaction or WAL is failing — metric retention is at risk. Check the WAL for a corrupt segment (this silently capped retention to ~2h before)." - # Overpass upstream health. The public Overpass servers the Planner - # proxies to (overpass-api.de / lz4) flake — rate-limiting (429) and - # stalls. This alerts on a *sustained* upstream failure rate (>20% - # over 10m), distinct from the Caddy-502 alert which only sees the - # symptom. Brief blips (e.g. the ~30s stall on 2026-06-08) won't - # page; sustained degradation will. client-abort is excluded (that's - # the caller giving up, not the upstream failing). Per-status detail - # lives on the Planner dashboard. - - uid: overpass-upstream-unhealthy - title: Overpass upstream failure rate high + # POI index freshness. POIs are served from the self-hosted + # `planner.pois` index, refreshed by a monthly extract+import. If the + # index age exceeds ~6 weeks the monthly refresh has missed a cycle + # (extract job, vSwitch transfer, checksum, guard, or import failed) and + # the data is going stale. `poi_index_age_seconds` is published by the + # Planner from its own DB; a growing value also covers a silently-failing + # import (the guard refuses to swap, so age keeps climbing). Import-run + # detail lives on the Planner dashboard (poi_import_last_status). + - uid: poi-index-stale + title: POI index is stale condition: B noDataState: OK data: @@ -282,7 +282,7 @@ groups: relativeTimeRange: { from: 600, to: 0 } datasourceUid: prometheus model: - expr: (sum(rate(overpass_upstream_requests_total{status!~"2..|client-abort"}[10m])) or vector(0)) / clamp_min(sum(rate(overpass_upstream_requests_total[10m])), 0.001) * 100 + expr: max(poi_index_age_seconds) instant: true - refId: B datasourceUid: __expr__ @@ -290,12 +290,13 @@ groups: type: threshold expression: A conditions: - - evaluator: { params: [20], type: gt } + # 6 weeks = 3628800s — one missed monthly refresh. + - evaluator: { params: [3628800], type: gt } operator: { type: and } reducer: { type: last } - for: 10m + for: 1h annotations: - summary: "Overpass upstream failure rate above 20% over 10m — public Overpass servers are degraded or rate-limiting. See the Planner dashboard; weigh self-hosting Overpass." + summary: "POI index age exceeds 6 weeks — the monthly refresh has missed a cycle. Check the poi-extract timer on the BRouter host and poi-import on the flagship (Planner dashboard: Last Import Status)." contactPoints: # Single "default" contact point with multiple integrations: every alert diff --git a/openspec/changes/poi-index/tasks.md b/openspec/changes/poi-index/tasks.md index 5edfe8f..7ba8f2d 100644 --- a/openspec/changes/poi-index/tasks.md +++ b/openspec/changes/poi-index/tasks.md @@ -36,16 +36,16 @@ ## 7. Observability - [x] 7.1 Add `poi_index_rows{category}`, `poi_index_age_seconds`, import outcome, and `poi_api_requests_total{status}` metrics; remove `overpass_upstream_*` metrics from `metrics.server.ts` (planner exposes rows/age via scrape-time DB collect + `poi_api_requests_total`; import outcome is emitted by the import job via node_exporter textfile — see Task 4) -- [ ] 7.2 Update the Grafana planner dashboard: replace Overpass upstream panels with index freshness + serving panels; add the stale-index alert (~6 weeks) +- [x] 7.2 Update the Grafana planner dashboard: replace Overpass upstream panels with index freshness + serving panels; add the stale-index alert (~6 weeks) (`poi-index-stale` in alerts.yml) ## 8. Docs & cleanup -- [ ] 8.1 Update `docs/architecture.md` (POI data flow) and the privacy documentation (no third-party POI requests) -- [ ] 8.2 Document the self-hoster story: optional pipeline, regional extract URL, graceful behavior with an empty index -- [ ] 8.3 Update `docs/ideas/self-host-overpass/README.md`: superseded by `poi-index`, with the revive-criteria note kept for the QL-compatibility case +- [x] 8.1 Update `docs/architecture.md` (POI data flow) and the privacy documentation (Overpass now Journal-surface-backfill-only; Planner POIs served same-origin) +- [x] 8.2 Document the self-hoster story: optional pipeline, regional extract URL, graceful behavior with an empty index (poi-extract README) +- [x] 8.3 Update `docs/ideas/self-host-overpass/README.md`: superseded by `poi-index`, with the revive-criteria note kept for the QL-compatibility case (+ roadmap pointer) ## 9. Verification -- [ ] 9.1 E2E: POI overlay flow against a seeded local `planner.pois` table (enable category → markers appear; popup shows name/hours/OSM link) -- [ ] 9.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` -- [ ] 9.3 Post-cutover production check: enable each of the nine categories on a busy viewport and a rural viewport; confirm results and dashboard metrics +- [x] 9.1 E2E: POI overlay flow (enable category → markers appear) — `e2e/planner-overlays.test.ts` + nearby-snap in `planner-routing.test.ts` now mock `/api/pois` (network-mock is the house pattern; a DB-seeded variant is possible but the route's SQL path is covered by `api.pois.test.ts`) +- [ ] 9.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` _(typecheck + lint + unit all green; e2e needs the local stack — run before merge)_ +- [ ] 9.3 Post-cutover production check: enable each of the nine categories on a busy viewport and a rural viewport; confirm results and dashboard metrics _(operational: after the first production import)_ diff --git a/packages/map-core/tsconfig.json b/packages/map-core/tsconfig.json index 5a24989..d19510a 100644 --- a/packages/map-core/tsconfig.json +++ b/packages/map-core/tsconfig.json @@ -4,5 +4,11 @@ "outDir": "dist", "rootDir": "src" }, - "include": ["src"] + "include": ["src"], + // *.sync.test.ts use node:fs to compare committed generated files + // (osmium-filters.txt, poi-selectors.sql) against the selectors. They run + // under vitest (node builtins provided) but are excluded from `tsc` so + // map-core keeps its zero-dependency, node-free type surface. All other + // tests are still typechecked. + "exclude": ["node_modules", "build", "dist", "src/**/*.sync.test.ts"] }