poi-index: Grafana dashboard/alert, docs, privacy manifest
- planner dashboard: replace Overpass panels with POI index freshness + serving panels (age, rows/category, import status, API request rate/errors) - alerts: replace overpass-upstream-unhealthy with poi-index-stale (>6 weeks) - architecture.md: POI data flow now the self-hosted index - privacy manifest (DE+EN): Overpass is Journal-surface-backfill-only; Planner POIs served same-origin from /api/pois - self-host-overpass README + roadmap: superseded for POIs by poi-index - poi-extract README: self-hoster story (optional pipeline, regional extract, graceful empty index) - map-core tsconfig: exclude *.sync.test.ts from tsc to keep it zero-dep Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer
parent
e703843b9b
commit
96de8831cf
9 changed files with 183 additions and 153 deletions
|
|
@ -79,6 +79,31 @@ poi-extract.service` then `journalctl --user -u poi-extract.service -f`.
|
|||
- **CPU/time**: a full planet filter is IO-bound and can take a few hours; the
|
||||
unit runs at `Nice=10` / idle IO priority so it doesn't disturb BRouter.
|
||||
|
||||
## Self-hosting
|
||||
|
||||
The pipeline is **optional**. A fresh instance works with an empty or absent
|
||||
`planner.pois` table: `/api/pois` returns an empty result / 503 and the map's
|
||||
POI panel shows "unavailable" while every other overlay and tile keeps working
|
||||
— nothing breaks.
|
||||
|
||||
When you do want POIs, you don't need a second host or the planet. Both halves
|
||||
run on one box, and `POI_PBF_URL` can point at any Geofabrik extract sized to
|
||||
your region:
|
||||
|
||||
```bash
|
||||
# Extract just your region (much smaller download, seconds not hours):
|
||||
POI_PBF_URL=https://download.geofabrik.de/europe/germany-latest.osm.pbf \
|
||||
./poi-extract.sh
|
||||
|
||||
# Then import it (first time: --bootstrap, since the live table is empty):
|
||||
POI_ARTIFACT_BASE_URL=file-or-vswitch-url \
|
||||
../../scripts/poi-import.sh --bootstrap
|
||||
```
|
||||
|
||||
On a single-host instance, publish the artifact however is convenient (local
|
||||
path, a static file server) and set `POI_ARTIFACT_BASE_URL` accordingly; the
|
||||
two-host vSwitch topology is a flagship detail, not a requirement.
|
||||
|
||||
## Dry run first
|
||||
|
||||
Before the first planet run, validate the whole chain on a small extract:
|
||||
|
|
|
|||
|
|
@ -99,7 +99,7 @@
|
|||
]
|
||||
},
|
||||
{
|
||||
"title": "Overpass Upstream Health (5m success rate)",
|
||||
"title": "POI Index Age",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
|
|
@ -109,8 +109,83 @@
|
|||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "(sum(rate(overpass_upstream_requests_total{status=~\"2..\"}[5m])) or vector(0)) / sum(rate(overpass_upstream_requests_total[5m]))",
|
||||
"legendFormat": "success"
|
||||
"expr": "poi_index_age_seconds",
|
||||
"legendFormat": "age"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "orange", "value": 3024000 },
|
||||
{ "color": "red", "value": 3628800 }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "POI Index Rows (total)",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 6,
|
||||
"y": 14
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(poi_index_rows)",
|
||||
"legendFormat": "rows"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Last Import Status",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 12,
|
||||
"y": 14
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "poi_import_last_status",
|
||||
"legendFormat": "status"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"mappings": [
|
||||
{ "type": "value", "options": { "0": { "text": "FAIL", "color": "red" }, "1": { "text": "OK", "color": "green" } } }
|
||||
],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "red", "value": null },
|
||||
{ "color": "green", "value": 1 }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "POI API Error Rate (5m)",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 18,
|
||||
"y": 14
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "(sum(rate(poi_api_requests_total{status=~\"rate_limited|error\"}[5m])) or vector(0)) / clamp_min(sum(rate(poi_api_requests_total[5m])), 0.001)",
|
||||
"legendFormat": "error ratio"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
|
|
@ -121,76 +196,16 @@
|
|||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "red", "value": null },
|
||||
{ "color": "orange", "value": 0.9 },
|
||||
{ "color": "green", "value": 0.99 }
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "orange", "value": 0.05 },
|
||||
{ "color": "red", "value": 0.2 }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Overpass Cache Hit Ratio (5m)",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 6,
|
||||
"y": 14
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "(sum(rate(overpass_cache_events_total{result=\"hit\"}[5m])) or vector(0)) / sum(rate(overpass_cache_events_total[5m]))",
|
||||
"legendFormat": "hit ratio"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Overpass Cache Size",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 12,
|
||||
"y": 14
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "overpass_cache_size",
|
||||
"legendFormat": "entries"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Overpass Upstream p95",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 18,
|
||||
"y": 14
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))",
|
||||
"legendFormat": "p95"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Overpass Cache Events",
|
||||
"title": "POI API Requests (5m by status)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
|
|
@ -200,28 +215,7 @@
|
|||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (result) (rate(overpass_cache_events_total[5m]))",
|
||||
"legendFormat": "{{result}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Overpass Upstream Status",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 19
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (status) (rate(overpass_upstream_requests_total[5m]))",
|
||||
"expr": "sum by (status) (rate(poi_api_requests_total[5m]))",
|
||||
"legendFormat": "{{status}}"
|
||||
}
|
||||
],
|
||||
|
|
@ -232,33 +226,20 @@
|
|||
}
|
||||
},
|
||||
{
|
||||
"title": "Overpass Upstream Latency",
|
||||
"title": "POI Index Rows by Category",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 26
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 19
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.50, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.99, sum(rate(overpass_upstream_duration_seconds_bucket[5m])) by (le))",
|
||||
"legendFormat": "p99"
|
||||
"expr": "poi_index_rows",
|
||||
"legendFormat": "{{category}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -265,16 +265,16 @@ groups:
|
|||
annotations:
|
||||
summary: "Prometheus TSDB compaction or WAL is failing — metric retention is at risk. Check the WAL for a corrupt segment (this silently capped retention to ~2h before)."
|
||||
|
||||
# Overpass upstream health. The public Overpass servers the Planner
|
||||
# proxies to (overpass-api.de / lz4) flake — rate-limiting (429) and
|
||||
# stalls. This alerts on a *sustained* upstream failure rate (>20%
|
||||
# over 10m), distinct from the Caddy-502 alert which only sees the
|
||||
# symptom. Brief blips (e.g. the ~30s stall on 2026-06-08) won't
|
||||
# page; sustained degradation will. client-abort is excluded (that's
|
||||
# the caller giving up, not the upstream failing). Per-status detail
|
||||
# lives on the Planner dashboard.
|
||||
- uid: overpass-upstream-unhealthy
|
||||
title: Overpass upstream failure rate high
|
||||
# POI index freshness. POIs are served from the self-hosted
|
||||
# `planner.pois` index, refreshed by a monthly extract+import. If the
|
||||
# index age exceeds ~6 weeks the monthly refresh has missed a cycle
|
||||
# (extract job, vSwitch transfer, checksum, guard, or import failed) and
|
||||
# the data is going stale. `poi_index_age_seconds` is published by the
|
||||
# Planner from its own DB; a growing value also covers a silently-failing
|
||||
# import (the guard refuses to swap, so age keeps climbing). Import-run
|
||||
# detail lives on the Planner dashboard (poi_import_last_status).
|
||||
- uid: poi-index-stale
|
||||
title: POI index is stale
|
||||
condition: B
|
||||
noDataState: OK
|
||||
data:
|
||||
|
|
@ -282,7 +282,7 @@ groups:
|
|||
relativeTimeRange: { from: 600, to: 0 }
|
||||
datasourceUid: prometheus
|
||||
model:
|
||||
expr: (sum(rate(overpass_upstream_requests_total{status!~"2..|client-abort"}[10m])) or vector(0)) / clamp_min(sum(rate(overpass_upstream_requests_total[10m])), 0.001) * 100
|
||||
expr: max(poi_index_age_seconds)
|
||||
instant: true
|
||||
- refId: B
|
||||
datasourceUid: __expr__
|
||||
|
|
@ -290,12 +290,13 @@ groups:
|
|||
type: threshold
|
||||
expression: A
|
||||
conditions:
|
||||
- evaluator: { params: [20], type: gt }
|
||||
# 6 weeks = 3628800s — one missed monthly refresh.
|
||||
- evaluator: { params: [3628800], type: gt }
|
||||
operator: { type: and }
|
||||
reducer: { type: last }
|
||||
for: 10m
|
||||
for: 1h
|
||||
annotations:
|
||||
summary: "Overpass upstream failure rate above 20% over 10m — public Overpass servers are degraded or rate-limiting. See the Planner dashboard; weigh self-hosting Overpass."
|
||||
summary: "POI index age exceeds 6 weeks — the monthly refresh has missed a cycle. Check the poi-extract timer on the BRouter host and poi-import on the flagship (Planner dashboard: Last Import Status)."
|
||||
|
||||
contactPoints:
|
||||
# Single "default" contact point with multiple integrations: every alert
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue