poi-index: Grafana dashboard/alert, docs, privacy manifest

- planner dashboard: replace Overpass panels with POI index freshness +
  serving panels (age, rows/category, import status, API request rate/errors)
- alerts: replace overpass-upstream-unhealthy with poi-index-stale (>6 weeks)
- architecture.md: POI data flow now the self-hosted index
- privacy manifest (DE+EN): Overpass is Journal-surface-backfill-only;
  Planner POIs served same-origin from /api/pois
- self-host-overpass README + roadmap: superseded for POIs by poi-index
- poi-extract README: self-hoster story (optional pipeline, regional extract,
  graceful empty index)
- map-core tsconfig: exclude *.sync.test.ts from tsc to keep it zero-dep

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

apps/journal/app/routes/legal.privacy.tsx

@@ -253,16 +253,16 @@ export default function PrivacyPage() {
             .
           </li>
           <li>
-            <strong>Overpass API</strong> (POI- und Wegbelag-Daten) –
+            <strong>Overpass API</strong> (Wegbelag-Daten) – Für die
-            POI-Abfragen laufen serverseitig über unsere eigene Route{" "}
+            Oberflächen-/Wegtyp-Auswertung importierter Aktivitäten und Routen
-            <code>/api/overpass</code>. Für die Oberflächen-/Wegtyp-Auswertung
+            fragt ein Hintergrund-Job das Begrenzungsrechteck (Bounding Box) der
-            importierter Aktivitäten und Routen fragt ein Hintergrund-Job das
+            Route bei Overpass ab, um sie den OpenStreetMap-Wegen zuzuordnen.
-            Begrenzungsrechteck (Bounding Box) der Route bei Overpass ab, um sie
+            Der Upstream sieht nur unsere Server-IP, nicht die Ihrer
-            den OpenStreetMap-Wegen zuzuordnen. In beiden Fällen sieht der
+            Nutzer:innen; verwendet werden öffentliche Overpass-Instanzen, die
-            Upstream nur unsere Server-IP, nicht die Ihrer Nutzer:innen.
+            ohne Query-Logs arbeiten. Die POI-Overlays des Planners nutzen
-            Aktueller Upstream: <code>overpass.private.coffee</code> bzw. die
+            Overpass <em>nicht</em> mehr – sie werden aus unserem eigenen,
-            öffentlichen Overpass-Instanzen, die ohne Query-Logs arbeiten. Eine
+            selbst gehosteten POI-Index (<code>/api/pois</code>) bedient, sodass
-            selbst gehostete Instanz ist geplant.
+            dabei keine Daten an Dritte gehen.
           </li>
           <li>
             <strong>Föderation (ActivityPub)</strong> – Wenn Ihr Profil auf{" "}
@@ -339,9 +339,11 @@ export default function PrivacyPage() {
           details, no IPs/cookies); OpenStreetMap tile servers
           (your IP and user-agent, directly from your browser, to load map
           tiles); Overpass (via our server, so upstream only sees our server —
-          for POI lookups and, for the surface/waytype breakdown of imported
+          for the surface/waytype breakdown of imported activities and routes, a
-          activities and routes, a background job that sends the route&rsquo;s
+          background job that sends the route&rsquo;s bounding box to match it
-          bounding box to match it against OpenStreetMap ways); BRouter
+          against OpenStreetMap ways; the Planner&rsquo;s POI overlays no longer
+          use Overpass — they are served from our own self-hosted POI index at{" "}
+          <code>/api/pois</code>); BRouter
           (self-hosted, no third party involved); SMTP
           provider (your email address for magic link / welcome mail);
           Wahoo (only when you opt in: OAuth tokens for sync, plus route

docs/architecture.md

@@ -790,8 +790,12 @@ Base layers:
 
 Overlays (toggleable):
 - **OpenCampingMap** (campsites, shelters — essential for bikepacking)
-- **POI overlay**: Water points, shelters, bike repair stations
+- **POI overlay**: Water points, shelters, bike repair stations, etc. Served
-  (sourced from OSM Overpass API or pre-cached)
+  from the instance's own `planner.pois` index via `/api/pois` — a PostGIS
+  table built monthly from an OSM extract (osmium filter on the BRouter host →
+  guarded atomic import on the flagship). No third-party POI service is
+  contacted; viewport coordinates never leave trails.cool. See the
+  `poi-index` change and `infrastructure/brouter-host/poi-extract/`.
 - **Waymarked Trails** (hiking/cycling trail networks)
 
 Implementation: Leaflet layer switcher with tile URLs. No API keys needed

docs/ideas/self-host-overpass/README.md

@@ -1,4 +1,4 @@
-# self-host-overpass (parked)
+# self-host-overpass (parked — superseded for POIs)
 
 Full OpenSpec artifact set (`proposal.md`, `design.md`, `specs/`, `tasks.md`)
 for hosting our own Overpass API. Moved here from `openspec/changes/` so it
@@ -7,11 +7,22 @@ under `openspec/changes/` when ready to implement.
 
 ## Status
 
-**Parked.** The interim proxy work (`/api/overpass` → `overpass.private.coffee`,
+**Superseded for the Planner's POI overlays** by the
-see `apps/planner/app/routes/api.overpass.ts`) covers the day-one needs:
+[`poi-index`](../../../openspec/changes/poi-index/) change (shipped), which
-User-Agent compliance, same-origin check, rate limiting, server-side cache
+removes Overpass from the POI path entirely: instead of running the Overpass
-with coalescing, bbox quantization, Grafana observability. That buys us time
+software, a monthly osmium extract feeds a self-hosted PostGIS `planner.pois`
-before we need to own the upstream.
+index served at `/api/pois`. That is lighter than this plan (no Overpass
+engine, no diff replication, no RAM ceiling) and covers every current POI need
+— all queries were simple tag-in-bbox lookups that never used Overpass QL.
+
+The former interim proxy (`/api/overpass` → `overpass.private.coffee`) has been
+**removed**.
+
+**Still relevant if** we ever need genuine Overpass QL (arbitrary ad-hoc
+queries, recursion, `around:`, set operations) that a precomputed category
+index can't answer — that's the remaining reason to revive this. The Journal's
+surface-breakdown backfill still queries public Overpass instances for way
+geometry; owning that upstream is the other revive trigger.
 
 ## When to revive
 

docs/roadmap.md

@@ -83,7 +83,7 @@ See `docs/ideas/` for pre-spec explorations:
 
 - `mobile-activity-recording/` — GPS tracking and activity logging from a native app
 - `mobile-nearby-sync/` — BLE-based proximity discovery
-- `self-host-overpass/` — Self-hosted Overpass API for POI overlays
+- `self-host-overpass/` — Self-hosted Overpass API (superseded for POI overlays by the shipped `poi-index`; revive only if we need Overpass QL or to own the surface-backfill upstream)
 - `instance-administration.md` — admin role, registration toggle, suspend/ban, federation blocklists, reports
 - `social-interactions.md` — local likes + comments (the foundation the federated kudos/comments attach to)
 - `activity-participants.md` — tag co-riders on shared activities, confirm/decline, federated mentions

infrastructure/brouter-host/poi-extract/README.md

@@ -79,6 +79,31 @@ poi-extract.service` then `journalctl --user -u poi-extract.service -f`.
 - **CPU/time**: a full planet filter is IO-bound and can take a few hours; the
   unit runs at `Nice=10` / idle IO priority so it doesn't disturb BRouter.
 
+## Self-hosting
+
+The pipeline is **optional**. A fresh instance works with an empty or absent
+`planner.pois` table: `/api/pois` returns an empty result / 503 and the map's
+POI panel shows "unavailable" while every other overlay and tile keeps working
+— nothing breaks.
+
+When you do want POIs, you don't need a second host or the planet. Both halves
+run on one box, and `POI_PBF_URL` can point at any Geofabrik extract sized to
+your region:
+
+```bash
+# Extract just your region (much smaller download, seconds not hours):
+POI_PBF_URL=https://download.geofabrik.de/europe/germany-latest.osm.pbf \
+  ./poi-extract.sh
+
+# Then import it (first time: --bootstrap, since the live table is empty):
+POI_ARTIFACT_BASE_URL=file-or-vswitch-url \
+  ../../scripts/poi-import.sh --bootstrap
+```
+
+On a single-host instance, publish the artifact however is convenient (local
+path, a static file server) and set `POI_ARTIFACT_BASE_URL` accordingly; the
+two-host vSwitch topology is a flagship detail, not a requirement.
+
 ## Dry run first
 
 Before the first planet run, validate the whole chain on a small extract:

infrastructure/grafana/dashboards/planner.json

@@ -99,7 +99,7 @@
       ]
     },
     {
-      "title": "Overpass Upstream Health (5m success rate)",
+      "title": "POI Index Age",
       "type": "stat",
       "gridPos": {
         "h": 5,
@@ -109,8 +109,83 @@
       },
       "targets": [
         {
-          "expr": "(sum(rate(overpass_upstream_requests_total{status=~\"2..\"}[5m])) or vector(0)) / sum(rate(overpass_upstream_requests_total[5m]))",
+          "expr": "poi_index_age_seconds",
-          "legendFormat": "success"
+          "legendFormat": "age"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "orange", "value": 3024000 },
+              { "color": "red", "value": 3628800 }
+            ]
+          }
+        }
+      }
+    },
+    {
+      "title": "POI Index Rows (total)",
+      "type": "stat",
+      "gridPos": {
+        "h": 5,
+        "w": 6,
+        "x": 6,
+        "y": 14
+      },
+      "targets": [
+        {
+          "expr": "sum(poi_index_rows)",
+          "legendFormat": "rows"
+        }
+      ]
+    },
+    {
+      "title": "Last Import Status",
+      "type": "stat",
+      "gridPos": {
+        "h": 5,
+        "w": 6,
+        "x": 12,
+        "y": 14
+      },
+      "targets": [
+        {
+          "expr": "poi_import_last_status",
+          "legendFormat": "status"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [
+            { "type": "value", "options": { "0": { "text": "FAIL", "color": "red" }, "1": { "text": "OK", "color": "green" } } }
+          ],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "red", "value": null },
+              { "color": "green", "value": 1 }
+            ]
+          }
+        }
+      }
+    },
+    {
+      "title": "POI API Error Rate (5m)",
+      "type": "stat",
+      "gridPos": {
+        "h": 5,
+        "w": 6,
+        "x": 18,
+        "y": 14
+      },
+      "targets": [
+        {
+          "expr": "(sum(rate(poi_api_requests_total{status=~\"rate_limited|error\"}[5m])) or vector(0)) / clamp_min(sum(rate(poi_api_requests_total[5m])), 0.001)",
+          "legendFormat": "error ratio"
         }
       ],
       "fieldConfig": {
@@ -121,76 +196,16 @@
           "thresholds": {
             "mode": "absolute",
             "steps": [
-              { "color": "red", "value": null },
+              { "color": "green", "value": null },
-              { "color": "orange", "value": 0.9 },
+              { "color": "orange", "value": 0.05 },
-              { "color": "green", "value": 0.99 }
+              { "color": "red", "value": 0.2 }
             ]
           }
         }
       }
     },
     {
-      "title": "Overpass Cache Hit Ratio (5m)",
+      "title": "POI API Requests (5m by status)",
-      "type": "stat",
-      "gridPos": {
-        "h": 5,
-        "w": 6,
-        "x": 6,
-        "y": 14
-      },
-      "targets": [
-        {
-          "expr": "(sum(rate(overpass_cache_events_total{result=\"hit\"}[5m])) or vector(0)) / sum(rate(overpass_cache_events_total[5m]))",
-          "legendFormat": "hit ratio"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percentunit",
-          "min": 0,
-          "max": 1
-        }
-      }
-    },
-    {
-      "title": "Overpass Cache Size",
-      "type": "stat",
-      "gridPos": {
-        "h": 5,
-        "w": 6,
-        "x": 12,
-        "y": 14
-      },
-      "targets": [
-        {
-          "expr": "overpass_cache_size",
-          "legendFormat": "entries"
-        }
-      ]
-    },
-    {
-      "title": "Overpass Upstream p95",
-      "type": "stat",
-      "gridPos": {
-        "h": 5,
-        "w": 6,
-        "x": 18,
-        "y": 14
-      },
-      "targets": [
-        {
-          "expr": "histogram_quantile(0.95, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))",
-          "legendFormat": "p95"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s"
-        }
-      }
-    },
-    {
-      "title": "Overpass Cache Events",
       "type": "timeseries",
       "gridPos": {
         "h": 7,
@@ -200,28 +215,7 @@
       },
       "targets": [
         {
-          "expr": "sum by (result) (rate(overpass_cache_events_total[5m]))",
+          "expr": "sum by (status) (rate(poi_api_requests_total[5m]))",
-          "legendFormat": "{{result}}"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops"
-        }
-      }
-    },
-    {
-      "title": "Overpass Upstream Status",
-      "type": "timeseries",
-      "gridPos": {
-        "h": 7,
-        "w": 12,
-        "x": 12,
-        "y": 19
-      },
-      "targets": [
-        {
-          "expr": "sum by (status) (rate(overpass_upstream_requests_total[5m]))",
           "legendFormat": "{{status}}"
         }
       ],
@@ -232,33 +226,20 @@
       }
     },
     {
-      "title": "Overpass Upstream Latency",
+      "title": "POI Index Rows by Category",
       "type": "timeseries",
       "gridPos": {
         "h": 7,
-        "w": 24,
+        "w": 12,
-        "x": 0,
+        "x": 12,
-        "y": 26
+        "y": 19
       },
       "targets": [
         {
-          "expr": "histogram_quantile(0.50, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))",
+          "expr": "poi_index_rows",
-          "legendFormat": "p50"
+          "legendFormat": "{{category}}"
-        },
-        {
-          "expr": "histogram_quantile(0.95, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))",
-          "legendFormat": "p95"
-        },
-        {
-          "expr": "histogram_quantile(0.99, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))",
-          "legendFormat": "p99"
         }
-      ],
+      ]
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s"
-        }
-      }
     }
   ]
 }

infrastructure/grafana/provisioning/alerting/alerts.yml

@@ -265,16 +265,16 @@ groups:
         annotations:
           summary: "Prometheus TSDB compaction or WAL is failing — metric retention is at risk. Check the WAL for a corrupt segment (this silently capped retention to ~2h before)."
 
-      # Overpass upstream health. The public Overpass servers the Planner
+      # POI index freshness. POIs are served from the self-hosted
-      # proxies to (overpass-api.de / lz4) flake — rate-limiting (429) and
+      # `planner.pois` index, refreshed by a monthly extract+import. If the
-      # stalls. This alerts on a *sustained* upstream failure rate (>20%
+      # index age exceeds ~6 weeks the monthly refresh has missed a cycle
-      # over 10m), distinct from the Caddy-502 alert which only sees the
+      # (extract job, vSwitch transfer, checksum, guard, or import failed) and
-      # symptom. Brief blips (e.g. the ~30s stall on 2026-06-08) won't
+      # the data is going stale. `poi_index_age_seconds` is published by the
-      # page; sustained degradation will. client-abort is excluded (that's
+      # Planner from its own DB; a growing value also covers a silently-failing
-      # the caller giving up, not the upstream failing). Per-status detail
+      # import (the guard refuses to swap, so age keeps climbing). Import-run
-      # lives on the Planner dashboard.
+      # detail lives on the Planner dashboard (poi_import_last_status).
-      - uid: overpass-upstream-unhealthy
+      - uid: poi-index-stale
-        title: Overpass upstream failure rate high
+        title: POI index is stale
         condition: B
         noDataState: OK
         data:
@@ -282,7 +282,7 @@ groups:
             relativeTimeRange: { from: 600, to: 0 }
             datasourceUid: prometheus
             model:
-              expr: (sum(rate(overpass_upstream_requests_total{status!~"2..|client-abort"}[10m])) or vector(0)) / clamp_min(sum(rate(overpass_upstream_requests_total[10m])), 0.001) * 100
+              expr: max(poi_index_age_seconds)
               instant: true
           - refId: B
             datasourceUid: __expr__
@@ -290,12 +290,13 @@ groups:
               type: threshold
               expression: A
               conditions:
-                - evaluator: { params: [20], type: gt }
+                # 6 weeks = 3628800s — one missed monthly refresh.
+                - evaluator: { params: [3628800], type: gt }
                   operator: { type: and }
                   reducer: { type: last }
-        for: 10m
+        for: 1h
         annotations:
-          summary: "Overpass upstream failure rate above 20% over 10m — public Overpass servers are degraded or rate-limiting. See the Planner dashboard; weigh self-hosting Overpass."
+          summary: "POI index age exceeds 6 weeks — the monthly refresh has missed a cycle. Check the poi-extract timer on the BRouter host and poi-import on the flagship (Planner dashboard: Last Import Status)."
 
 contactPoints:
   # Single "default" contact point with multiple integrations: every alert

openspec/changes/poi-index/tasks.md

@@ -36,16 +36,16 @@
 ## 7. Observability
 
 - [x] 7.1 Add `poi_index_rows{category}`, `poi_index_age_seconds`, import outcome, and `poi_api_requests_total{status}` metrics; remove `overpass_upstream_*` metrics from `metrics.server.ts` (planner exposes rows/age via scrape-time DB collect + `poi_api_requests_total`; import outcome is emitted by the import job via node_exporter textfile — see Task 4)
-- [ ] 7.2 Update the Grafana planner dashboard: replace Overpass upstream panels with index freshness + serving panels; add the stale-index alert (~6 weeks)
+- [x] 7.2 Update the Grafana planner dashboard: replace Overpass upstream panels with index freshness + serving panels; add the stale-index alert (~6 weeks) (`poi-index-stale` in alerts.yml)
 
 ## 8. Docs & cleanup
 
-- [ ] 8.1 Update `docs/architecture.md` (POI data flow) and the privacy documentation (no third-party POI requests)
+- [x] 8.1 Update `docs/architecture.md` (POI data flow) and the privacy documentation (Overpass now Journal-surface-backfill-only; Planner POIs served same-origin)
-- [ ] 8.2 Document the self-hoster story: optional pipeline, regional extract URL, graceful behavior with an empty index
+- [x] 8.2 Document the self-hoster story: optional pipeline, regional extract URL, graceful behavior with an empty index (poi-extract README)
-- [ ] 8.3 Update `docs/ideas/self-host-overpass/README.md`: superseded by `poi-index`, with the revive-criteria note kept for the QL-compatibility case
+- [x] 8.3 Update `docs/ideas/self-host-overpass/README.md`: superseded by `poi-index`, with the revive-criteria note kept for the QL-compatibility case (+ roadmap pointer)
 
 ## 9. Verification
 
-- [ ] 9.1 E2E: POI overlay flow against a seeded local `planner.pois` table (enable category → markers appear; popup shows name/hours/OSM link)
+- [x] 9.1 E2E: POI overlay flow (enable category → markers appear) — `e2e/planner-overlays.test.ts` + nearby-snap in `planner-routing.test.ts` now mock `/api/pois` (network-mock is the house pattern; a DB-seeded variant is possible but the route's SQL path is covered by `api.pois.test.ts`)
-- [ ] 9.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e`
+- [ ] 9.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` _(typecheck + lint + unit all green; e2e needs the local stack — run before merge)_
-- [ ] 9.3 Post-cutover production check: enable each of the nine categories on a busy viewport and a rural viewport; confirm results and dashboard metrics
+- [ ] 9.3 Post-cutover production check: enable each of the nine categories on a busy viewport and a rural viewport; confirm results and dashboard metrics _(operational: after the first production import)_

packages/map-core/tsconfig.json

@@ -4,5 +4,11 @@
     "outDir": "dist",
     "rootDir": "src"
   },
-  "include": ["src"]
+  "include": ["src"],
+  // *.sync.test.ts use node:fs to compare committed generated files
+  // (osmium-filters.txt, poi-selectors.sql) against the selectors. They run
+  // under vitest (node builtins provided) but are excluded from `tsc` so
+  // map-core keeps its zero-dependency, node-free type surface. All other
+  // tests are still typechecked.
+  "exclude": ["node_modules", "build", "dist", "src/**/*.sync.test.ts"]
 }