diff --git a/apps/journal/app/routes/legal.privacy.tsx b/apps/journal/app/routes/legal.privacy.tsx index f2b10fe..8ee7b77 100644 --- a/apps/journal/app/routes/legal.privacy.tsx +++ b/apps/journal/app/routes/legal.privacy.tsx @@ -253,16 +253,16 @@ export default function PrivacyPage() { .
/api/overpass. Für die Oberflächen-/Wegtyp-Auswertung
- importierter Aktivitäten und Routen fragt ein Hintergrund-Job das
- Begrenzungsrechteck (Bounding Box) der Route bei Overpass ab, um sie
- den OpenStreetMap-Wegen zuzuordnen. In beiden Fällen sieht der
- Upstream nur unsere Server-IP, nicht die Ihrer Nutzer:innen.
- Aktueller Upstream: overpass.private.coffee bzw. die
- öffentlichen Overpass-Instanzen, die ohne Query-Logs arbeiten. Eine
- selbst gehostete Instanz ist geplant.
+ Overpass API (Wegbelag-Daten) – Für die
+ Oberflächen-/Wegtyp-Auswertung importierter Aktivitäten und Routen
+ fragt ein Hintergrund-Job das Begrenzungsrechteck (Bounding Box) der
+ Route bei Overpass ab, um sie den OpenStreetMap-Wegen zuzuordnen.
+ Der Upstream sieht nur unsere Server-IP, nicht die Ihrer
+ Nutzer:innen; verwendet werden öffentliche Overpass-Instanzen, die
+ ohne Query-Logs arbeiten. Die POI-Overlays des Planners nutzen
+ Overpass nicht mehr – sie werden aus unserem eigenen,
+ selbst gehosteten POI-Index (/api/pois) bedient, sodass
+ dabei keine Daten an Dritte gehen.
/api/pois); BRouter
(self-hosted, no third party involved); SMTP
provider (your email address for magic link / welcome mail);
Wahoo (only when you opt in: OAuth tokens for sync, plus route
diff --git a/docs/architecture.md b/docs/architecture.md
index 5be933a..ef47b4a 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -790,8 +790,12 @@ Base layers:
Overlays (toggleable):
- **OpenCampingMap** (campsites, shelters — essential for bikepacking)
-- **POI overlay**: Water points, shelters, bike repair stations
- (sourced from OSM Overpass API or pre-cached)
+- **POI overlay**: Water points, shelters, bike repair stations, etc. Served
+ from the instance's own `planner.pois` index via `/api/pois` — a PostGIS
+ table built monthly from an OSM extract (osmium filter on the BRouter host →
+ guarded atomic import on the flagship). No third-party POI service is
+ contacted; viewport coordinates never leave trails.cool. See the
+ `poi-index` change and `infrastructure/brouter-host/poi-extract/`.
- **Waymarked Trails** (hiking/cycling trail networks)
Implementation: Leaflet layer switcher with tile URLs. No API keys needed
diff --git a/docs/ideas/self-host-overpass/README.md b/docs/ideas/self-host-overpass/README.md
index 98b7ca4..11a141e 100644
--- a/docs/ideas/self-host-overpass/README.md
+++ b/docs/ideas/self-host-overpass/README.md
@@ -1,4 +1,4 @@
-# self-host-overpass (parked)
+# self-host-overpass (parked — superseded for POIs)
Full OpenSpec artifact set (`proposal.md`, `design.md`, `specs/`, `tasks.md`)
for hosting our own Overpass API. Moved here from `openspec/changes/` so it
@@ -7,11 +7,22 @@ under `openspec/changes/` when ready to implement.
## Status
-**Parked.** The interim proxy work (`/api/overpass` → `overpass.private.coffee`,
-see `apps/planner/app/routes/api.overpass.ts`) covers the day-one needs:
-User-Agent compliance, same-origin check, rate limiting, server-side cache
-with coalescing, bbox quantization, Grafana observability. That buys us time
-before we need to own the upstream.
+**Superseded for the Planner's POI overlays** by the
+[`poi-index`](../../../openspec/changes/poi-index/) change (shipped), which
+removes Overpass from the POI path entirely: instead of running the Overpass
+software, a monthly osmium extract feeds a self-hosted PostGIS `planner.pois`
+index served at `/api/pois`. That is lighter than this plan (no Overpass
+engine, no diff replication, no RAM ceiling) and covers every current POI need
+— all queries were simple tag-in-bbox lookups that never used Overpass QL.
+
+The former interim proxy (`/api/overpass` → `overpass.private.coffee`) has been
+**removed**.
+
+**Still relevant if** we ever need genuine Overpass QL (arbitrary ad-hoc
+queries, recursion, `around:`, set operations) that a precomputed category
+index can't answer — that's the remaining reason to revive this. The Journal's
+surface-breakdown backfill still queries public Overpass instances for way
+geometry; owning that upstream is the other revive trigger.
## When to revive
diff --git a/docs/roadmap.md b/docs/roadmap.md
index 30d555a..949ff7f 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -83,7 +83,7 @@ See `docs/ideas/` for pre-spec explorations:
- `mobile-activity-recording/` — GPS tracking and activity logging from a native app
- `mobile-nearby-sync/` — BLE-based proximity discovery
-- `self-host-overpass/` — Self-hosted Overpass API for POI overlays
+- `self-host-overpass/` — Self-hosted Overpass API (superseded for POI overlays by the shipped `poi-index`; revive only if we need Overpass QL or to own the surface-backfill upstream)
- `instance-administration.md` — admin role, registration toggle, suspend/ban, federation blocklists, reports
- `social-interactions.md` — local likes + comments (the foundation the federated kudos/comments attach to)
- `activity-participants.md` — tag co-riders on shared activities, confirm/decline, federated mentions
diff --git a/infrastructure/brouter-host/poi-extract/README.md b/infrastructure/brouter-host/poi-extract/README.md
index c59b307..8871980 100644
--- a/infrastructure/brouter-host/poi-extract/README.md
+++ b/infrastructure/brouter-host/poi-extract/README.md
@@ -79,6 +79,31 @@ poi-extract.service` then `journalctl --user -u poi-extract.service -f`.
- **CPU/time**: a full planet filter is IO-bound and can take a few hours; the
unit runs at `Nice=10` / idle IO priority so it doesn't disturb BRouter.
+## Self-hosting
+
+The pipeline is **optional**. A fresh instance works with an empty or absent
+`planner.pois` table: `/api/pois` returns an empty result / 503 and the map's
+POI panel shows "unavailable" while every other overlay and tile keeps working
+— nothing breaks.
+
+When you do want POIs, you don't need a second host or the planet. Both halves
+run on one box, and `POI_PBF_URL` can point at any Geofabrik extract sized to
+your region:
+
+```bash
+# Extract just your region (much smaller download, seconds not hours):
+POI_PBF_URL=https://download.geofabrik.de/europe/germany-latest.osm.pbf \
+ ./poi-extract.sh
+
+# Then import it (first time: --bootstrap, since the live table is empty):
+POI_ARTIFACT_BASE_URL=file-or-vswitch-url \
+ ../../scripts/poi-import.sh --bootstrap
+```
+
+On a single-host instance, publish the artifact however is convenient (local
+path, a static file server) and set `POI_ARTIFACT_BASE_URL` accordingly; the
+two-host vSwitch topology is a flagship detail, not a requirement.
+
## Dry run first
Before the first planet run, validate the whole chain on a small extract:
diff --git a/infrastructure/grafana/dashboards/planner.json b/infrastructure/grafana/dashboards/planner.json
index 97aa026..2fc9a85 100644
--- a/infrastructure/grafana/dashboards/planner.json
+++ b/infrastructure/grafana/dashboards/planner.json
@@ -99,7 +99,7 @@
]
},
{
- "title": "Overpass Upstream Health (5m success rate)",
+ "title": "POI Index Age",
"type": "stat",
"gridPos": {
"h": 5,
@@ -109,8 +109,83 @@
},
"targets": [
{
- "expr": "(sum(rate(overpass_upstream_requests_total{status=~\"2..\"}[5m])) or vector(0)) / sum(rate(overpass_upstream_requests_total[5m]))",
- "legendFormat": "success"
+ "expr": "poi_index_age_seconds",
+ "legendFormat": "age"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "unit": "s",
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ { "color": "green", "value": null },
+ { "color": "orange", "value": 3024000 },
+ { "color": "red", "value": 3628800 }
+ ]
+ }
+ }
+ }
+ },
+ {
+ "title": "POI Index Rows (total)",
+ "type": "stat",
+ "gridPos": {
+ "h": 5,
+ "w": 6,
+ "x": 6,
+ "y": 14
+ },
+ "targets": [
+ {
+ "expr": "sum(poi_index_rows)",
+ "legendFormat": "rows"
+ }
+ ]
+ },
+ {
+ "title": "Last Import Status",
+ "type": "stat",
+ "gridPos": {
+ "h": 5,
+ "w": 6,
+ "x": 12,
+ "y": 14
+ },
+ "targets": [
+ {
+ "expr": "poi_import_last_status",
+ "legendFormat": "status"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "mappings": [
+ { "type": "value", "options": { "0": { "text": "FAIL", "color": "red" }, "1": { "text": "OK", "color": "green" } } }
+ ],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ { "color": "red", "value": null },
+ { "color": "green", "value": 1 }
+ ]
+ }
+ }
+ }
+ },
+ {
+ "title": "POI API Error Rate (5m)",
+ "type": "stat",
+ "gridPos": {
+ "h": 5,
+ "w": 6,
+ "x": 18,
+ "y": 14
+ },
+ "targets": [
+ {
+ "expr": "(sum(rate(poi_api_requests_total{status=~\"rate_limited|error\"}[5m])) or vector(0)) / clamp_min(sum(rate(poi_api_requests_total[5m])), 0.001)",
+ "legendFormat": "error ratio"
}
],
"fieldConfig": {
@@ -121,76 +196,16 @@
"thresholds": {
"mode": "absolute",
"steps": [
- { "color": "red", "value": null },
- { "color": "orange", "value": 0.9 },
- { "color": "green", "value": 0.99 }
+ { "color": "green", "value": null },
+ { "color": "orange", "value": 0.05 },
+ { "color": "red", "value": 0.2 }
]
}
}
}
},
{
- "title": "Overpass Cache Hit Ratio (5m)",
- "type": "stat",
- "gridPos": {
- "h": 5,
- "w": 6,
- "x": 6,
- "y": 14
- },
- "targets": [
- {
- "expr": "(sum(rate(overpass_cache_events_total{result=\"hit\"}[5m])) or vector(0)) / sum(rate(overpass_cache_events_total[5m]))",
- "legendFormat": "hit ratio"
- }
- ],
- "fieldConfig": {
- "defaults": {
- "unit": "percentunit",
- "min": 0,
- "max": 1
- }
- }
- },
- {
- "title": "Overpass Cache Size",
- "type": "stat",
- "gridPos": {
- "h": 5,
- "w": 6,
- "x": 12,
- "y": 14
- },
- "targets": [
- {
- "expr": "overpass_cache_size",
- "legendFormat": "entries"
- }
- ]
- },
- {
- "title": "Overpass Upstream p95",
- "type": "stat",
- "gridPos": {
- "h": 5,
- "w": 6,
- "x": 18,
- "y": 14
- },
- "targets": [
- {
- "expr": "histogram_quantile(0.95, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))",
- "legendFormat": "p95"
- }
- ],
- "fieldConfig": {
- "defaults": {
- "unit": "s"
- }
- }
- },
- {
- "title": "Overpass Cache Events",
+ "title": "POI API Requests (5m by status)",
"type": "timeseries",
"gridPos": {
"h": 7,
@@ -200,28 +215,7 @@
},
"targets": [
{
- "expr": "sum by (result) (rate(overpass_cache_events_total[5m]))",
- "legendFormat": "{{result}}"
- }
- ],
- "fieldConfig": {
- "defaults": {
- "unit": "ops"
- }
- }
- },
- {
- "title": "Overpass Upstream Status",
- "type": "timeseries",
- "gridPos": {
- "h": 7,
- "w": 12,
- "x": 12,
- "y": 19
- },
- "targets": [
- {
- "expr": "sum by (status) (rate(overpass_upstream_requests_total[5m]))",
+ "expr": "sum by (status) (rate(poi_api_requests_total[5m]))",
"legendFormat": "{{status}}"
}
],
@@ -232,33 +226,20 @@
}
},
{
- "title": "Overpass Upstream Latency",
+ "title": "POI Index Rows by Category",
"type": "timeseries",
"gridPos": {
"h": 7,
- "w": 24,
- "x": 0,
- "y": 26
+ "w": 12,
+ "x": 12,
+ "y": 19
},
"targets": [
{
- "expr": "histogram_quantile(0.50, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))",
- "legendFormat": "p50"
- },
- {
- "expr": "histogram_quantile(0.95, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))",
- "legendFormat": "p95"
- },
- {
- "expr": "histogram_quantile(0.99, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))",
- "legendFormat": "p99"
+ "expr": "poi_index_rows",
+ "legendFormat": "{{category}}"
}
- ],
- "fieldConfig": {
- "defaults": {
- "unit": "s"
- }
- }
+ ]
}
]
}
diff --git a/infrastructure/grafana/provisioning/alerting/alerts.yml b/infrastructure/grafana/provisioning/alerting/alerts.yml
index cc9f2c0..b75a995 100644
--- a/infrastructure/grafana/provisioning/alerting/alerts.yml
+++ b/infrastructure/grafana/provisioning/alerting/alerts.yml
@@ -265,16 +265,16 @@ groups:
annotations:
summary: "Prometheus TSDB compaction or WAL is failing — metric retention is at risk. Check the WAL for a corrupt segment (this silently capped retention to ~2h before)."
- # Overpass upstream health. The public Overpass servers the Planner
- # proxies to (overpass-api.de / lz4) flake — rate-limiting (429) and
- # stalls. This alerts on a *sustained* upstream failure rate (>20%
- # over 10m), distinct from the Caddy-502 alert which only sees the
- # symptom. Brief blips (e.g. the ~30s stall on 2026-06-08) won't
- # page; sustained degradation will. client-abort is excluded (that's
- # the caller giving up, not the upstream failing). Per-status detail
- # lives on the Planner dashboard.
- - uid: overpass-upstream-unhealthy
- title: Overpass upstream failure rate high
+ # POI index freshness. POIs are served from the self-hosted
+ # `planner.pois` index, refreshed by a monthly extract+import. If the
+ # index age exceeds ~6 weeks the monthly refresh has missed a cycle
+ # (extract job, vSwitch transfer, checksum, guard, or import failed) and
+ # the data is going stale. `poi_index_age_seconds` is published by the
+ # Planner from its own DB; a growing value also covers a silently-failing
+ # import (the guard refuses to swap, so age keeps climbing). Import-run
+ # detail lives on the Planner dashboard (poi_import_last_status).
+ - uid: poi-index-stale
+ title: POI index is stale
condition: B
noDataState: OK
data:
@@ -282,7 +282,7 @@ groups:
relativeTimeRange: { from: 600, to: 0 }
datasourceUid: prometheus
model:
- expr: (sum(rate(overpass_upstream_requests_total{status!~"2..|client-abort"}[10m])) or vector(0)) / clamp_min(sum(rate(overpass_upstream_requests_total[10m])), 0.001) * 100
+ expr: max(poi_index_age_seconds)
instant: true
- refId: B
datasourceUid: __expr__
@@ -290,12 +290,13 @@ groups:
type: threshold
expression: A
conditions:
- - evaluator: { params: [20], type: gt }
+ # 6 weeks = 3628800s — one missed monthly refresh.
+ - evaluator: { params: [3628800], type: gt }
operator: { type: and }
reducer: { type: last }
- for: 10m
+ for: 1h
annotations:
- summary: "Overpass upstream failure rate above 20% over 10m — public Overpass servers are degraded or rate-limiting. See the Planner dashboard; weigh self-hosting Overpass."
+ summary: "POI index age exceeds 6 weeks — the monthly refresh has missed a cycle. Check the poi-extract timer on the BRouter host and poi-import on the flagship (Planner dashboard: Last Import Status)."
contactPoints:
# Single "default" contact point with multiple integrations: every alert
diff --git a/openspec/changes/poi-index/tasks.md b/openspec/changes/poi-index/tasks.md
index 5edfe8f..7ba8f2d 100644
--- a/openspec/changes/poi-index/tasks.md
+++ b/openspec/changes/poi-index/tasks.md
@@ -36,16 +36,16 @@
## 7. Observability
- [x] 7.1 Add `poi_index_rows{category}`, `poi_index_age_seconds`, import outcome, and `poi_api_requests_total{status}` metrics; remove `overpass_upstream_*` metrics from `metrics.server.ts` (planner exposes rows/age via scrape-time DB collect + `poi_api_requests_total`; import outcome is emitted by the import job via node_exporter textfile — see Task 4)
-- [ ] 7.2 Update the Grafana planner dashboard: replace Overpass upstream panels with index freshness + serving panels; add the stale-index alert (~6 weeks)
+- [x] 7.2 Update the Grafana planner dashboard: replace Overpass upstream panels with index freshness + serving panels; add the stale-index alert (~6 weeks) (`poi-index-stale` in alerts.yml)
## 8. Docs & cleanup
-- [ ] 8.1 Update `docs/architecture.md` (POI data flow) and the privacy documentation (no third-party POI requests)
-- [ ] 8.2 Document the self-hoster story: optional pipeline, regional extract URL, graceful behavior with an empty index
-- [ ] 8.3 Update `docs/ideas/self-host-overpass/README.md`: superseded by `poi-index`, with the revive-criteria note kept for the QL-compatibility case
+- [x] 8.1 Update `docs/architecture.md` (POI data flow) and the privacy documentation (Overpass now Journal-surface-backfill-only; Planner POIs served same-origin)
+- [x] 8.2 Document the self-hoster story: optional pipeline, regional extract URL, graceful behavior with an empty index (poi-extract README)
+- [x] 8.3 Update `docs/ideas/self-host-overpass/README.md`: superseded by `poi-index`, with the revive-criteria note kept for the QL-compatibility case (+ roadmap pointer)
## 9. Verification
-- [ ] 9.1 E2E: POI overlay flow against a seeded local `planner.pois` table (enable category → markers appear; popup shows name/hours/OSM link)
-- [ ] 9.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e`
-- [ ] 9.3 Post-cutover production check: enable each of the nine categories on a busy viewport and a rural viewport; confirm results and dashboard metrics
+- [x] 9.1 E2E: POI overlay flow (enable category → markers appear) — `e2e/planner-overlays.test.ts` + nearby-snap in `planner-routing.test.ts` now mock `/api/pois` (network-mock is the house pattern; a DB-seeded variant is possible but the route's SQL path is covered by `api.pois.test.ts`)
+- [ ] 9.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` _(typecheck + lint + unit all green; e2e needs the local stack — run before merge)_
+- [ ] 9.3 Post-cutover production check: enable each of the nine categories on a busy viewport and a rural viewport; confirm results and dashboard metrics _(operational: after the first production import)_
diff --git a/packages/map-core/tsconfig.json b/packages/map-core/tsconfig.json
index 5a24989..d19510a 100644
--- a/packages/map-core/tsconfig.json
+++ b/packages/map-core/tsconfig.json
@@ -4,5 +4,11 @@
"outDir": "dist",
"rootDir": "src"
},
- "include": ["src"]
+ "include": ["src"],
+ // *.sync.test.ts use node:fs to compare committed generated files
+ // (osmium-filters.txt, poi-selectors.sql) against the selectors. They run
+ // under vitest (node builtins provided) but are excluded from `tsc` so
+ // map-core keeps its zero-dependency, node-free type surface. All other
+ // tests are still typechecked.
+ "exclude": ["node_modules", "build", "dist", "src/**/*.sync.test.ts"]
}