poi-index: extract pipeline (BRouter host) + import job (flagship) + e2e

- poi-extract.sh: PBF download -> osmium tags-filter -> centroid NDJSON +
  manifest, published via the Caddy sidecar at vSwitch-only /poi/*
- osmium-filters.txt + poi-selectors.sql generated from map-core selectors,
  guarded by sync tests so poiCategories stays the single source of truth
- poi-import.sh: checksum verify -> classify into category rows ->
  70% guard (--bootstrap override) -> atomic rename swap; node_exporter
  textfile metric for import outcome
- systemd service+timer units for both halves (monthly, offset)
- cd-infra.yml: ship infrastructure/scripts to the flagship
- e2e: mock /api/pois instead of /api/overpass

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

infrastructure/brouter-host/Caddyfile

@@ -24,11 +24,23 @@
 	@authed header X-BRouter-Auth {$BROUTER_AUTH_TOKEN}
 
 	handle @authed {
-		reverse_proxy brouter:17777 {
-			# Don't forward the auth header upstream — BRouter doesn't
-			# use it, and it's cleaner to contain the credential at
-			# the proxy boundary.
-			header_up -X-BRouter-Auth
+		# POI index artifact + manifest published by poi-extract.sh. Served
+		# read-only from the mounted publish dir; the flagship import job
+		# pulls /poi/manifest.json and /poi/pois.ndjson.gz over the vSwitch.
+		# Same auth + vSwitch-only binding as BRouter, so it's unreachable
+		# from the public internet.
+		handle_path /poi/* {
+			root * /srv/poi
+			file_server
+		}
+
+		handle {
+			reverse_proxy brouter:17777 {
+				# Don't forward the auth header upstream — BRouter doesn't
+				# use it, and it's cleaner to contain the credential at
+				# the proxy boundary.
+				header_up -X-BRouter-Auth
+			}
 		}
 	}
 

infrastructure/brouter-host/docker-compose.yml

@@ -58,6 +58,9 @@ services:
       BROUTER_AUTH_TOKEN: ${BROUTER_AUTH_TOKEN:?BROUTER_AUTH_TOKEN must be set}
     volumes:
       - ./Caddyfile:/etc/caddy/Caddyfile:ro
+      # POI index artifact published by poi-extract.sh, served read-only under
+      # /poi/* on the same vSwitch-only listener as BRouter.
+      - ./poi-extract/publish:/srv/poi:ro
       - caddy-data:/data
       - caddy-config:/config
     networks:

infrastructure/brouter-host/poi-extract/README.md

@@ -0,0 +1,92 @@
+# POI extract pipeline (BRouter host)
+
+Monthly job that turns an OSM PBF into the compact POI artifact the flagship's
+`planner.pois` index is built from. This is the "filter where the disk is"
+half of the [`poi-index`](../../../openspec/changes/poi-index/) change — it
+replaces the third-party Overpass dependency for the Planner's POI overlays.
+
+Runs on the dedicated BRouter host (`ullrich.is`, the box with 1.8 TB free)
+under the non-root `trails` user. The flagship import job
+(`infrastructure/scripts/poi-import.sh`) pulls the result over the vSwitch.
+
+## Files
+
+| File | Role |
+|------|------|
+| `poi-extract.sh` | Download PBF → `osmium tags-filter` → `osmium export` → `to-ndjson.py` → gzip + manifest |
+| `to-ndjson.py` | GeoJSONSeq → NDJSON, reducing ways/relations to their bbox centre (Overpass `out center` equivalent) |
+| `osmium-filters.txt` | osmium tag filters, **generated** from `@trails-cool/map-core` POI selectors |
+| `gen-osmium-filters.ts` | Regenerates `osmium-filters.txt` (run after changing categories) |
+| `poi-extract.service` / `.timer` | systemd units (monthly) |
+
+## Single source of truth
+
+`osmium-filters.txt` is derived from `poiCategories` in
+`packages/map-core/src/poi.ts`. After changing categories, regenerate it:
+
+```bash
+npx tsx infrastructure/brouter-host/poi-extract/gen-osmium-filters.ts
+```
+
+`packages/map-core/src/osmium-filters.sync.test.ts` fails CI if the committed
+file drifts from the selectors, so the host never needs a Node runtime.
+
+## Artifact contract
+
+`poi-extract.sh` writes to the publish dir (default `./publish`, mounted into
+the Caddy sidecar at `/srv/poi`):
+
+- `pois.ndjson.gz` — one POI per line:
+  `{osm_type, osm_id, name, lat, lon, tags}`. Category is **not** stored here;
+  the flagship importer classifies each element from its `tags` using the same
+  map-core selectors (so the selector logic is never duplicated on this host).
+- `manifest.json` — `{artifact, sha256, row_count, generated_at, source}`.
+
+Caddy serves both under `/poi/*` on the existing vSwitch-only `:17777`
+listener, gated by the same `X-BRouter-Auth` shared secret as BRouter — so the
+artifact is unreachable from the public internet.
+
+## Prerequisites
+
+Install on the host: `osmium-tool`, `python3`, `curl`, `gzip`, `coreutils`
+(`sha256sum`). On Debian/Ubuntu: `apt install osmium-tool python3`.
+
+## Install the timer
+
+Deployed under `~trails/brouter/poi-extract/`. As a user unit with lingering:
+
+```bash
+loginctl enable-linger trails
+mkdir -p ~/.config/systemd/user
+cp poi-extract.service poi-extract.timer ~/.config/systemd/user/
+systemctl --user daemon-reload
+systemctl --user enable --now poi-extract.timer
+systemctl --user list-timers poi-extract.timer
+```
+
+Trigger a run by hand (e.g. the first bootstrap): `systemctl --user start
+poi-extract.service` then `journalctl --user -u poi-extract.service -f`.
+
+## Footprint
+
+- **Bandwidth**: the planet PBF is ~80 GB downloaded per run (monthly). Point
+  `POI_PBF_URL` at a Geofabrik mirror or a regional/continental extract to cut
+  this — e.g. `europe-latest.osm.pbf` (~30 GB) or `germany-latest.osm.pbf`
+  (~4 GB) for a smaller instance.
+- **Disk**: the planet PBF + filtered PBF live transiently in `./work` and are
+  removed on exit (even on failure) via a trap. Peak transient usage ≈ size of
+  the PBF. The published artifact is a few hundred MB gzipped.
+- **CPU/time**: a full planet filter is IO-bound and can take a few hours; the
+  unit runs at `Nice=10` / idle IO priority so it doesn't disturb BRouter.
+
+## Dry run first
+
+Before the first planet run, validate the whole chain on a small extract:
+
+```bash
+POI_PBF_URL=https://download.geofabrik.de/europe/germany-latest.osm.pbf \
+  ./poi-extract.sh
+cat publish/manifest.json          # row_count sane? sha256 present?
+zcat publish/pois.ndjson.gz | head # records well-formed?
+zcat publish/pois.ndjson.gz | wc -l
+```

infrastructure/brouter-host/poi-extract/gen-osmium-filters.ts

@@ -0,0 +1,18 @@
+/**
+ * Regenerate `osmium-filters.txt` from the POI category selectors in
+ * `@trails-cool/map-core` — the single source of truth for "what is a
+ * shelter". Run this after changing `poiCategories`:
+ *
+ *   pnpm --filter @trails-cool/map-core exec tsx \
+ *     ../../infrastructure/brouter-host/poi-extract/gen-osmium-filters.ts
+ *
+ * The committed file is what the (Node-free) BRouter host reads at extract
+ * time. `osmium-filters.sync.test.ts` in map-core fails CI if the two drift.
+ */
+import { writeFileSync } from "node:fs";
+import { fileURLToPath } from "node:url";
+import { osmiumTagFilters } from "../../../packages/map-core/src/poi.ts";
+
+const out = fileURLToPath(new URL("./osmium-filters.txt", import.meta.url));
+writeFileSync(out, osmiumTagFilters().join("\n") + "\n");
+console.log(`Wrote ${out}`);

infrastructure/brouter-host/poi-extract/osmium-filters.txt

@@ -0,0 +1,3 @@
+nwr/amenity=drinking_water,water_point,shelter,restaurant,cafe,fast_food,pub,biergarten,bicycle_parking,bicycle_repair_station,bicycle_rental,toilets
+nwr/tourism=wilderness_hut,camp_site,caravan_site,hotel,hostel,guest_house,viewpoint
+nwr/shop=supermarket,convenience,bakery

infrastructure/brouter-host/poi-extract/poi-extract.service

@@ -0,0 +1,23 @@
+# POI extract — one-shot unit driven by poi-extract.timer. Runs as the
+# non-root `trails` user (docker-group, scoped to ~trails/brouter/). Install as
+# a user unit: `systemctl --user enable --now poi-extract.timer` with lingering
+# enabled (`loginctl enable-linger trails`), or as a system unit with
+# User=trails. See README.md.
+[Unit]
+Description=Filter an OSM PBF to trails.cool POI categories and publish the artifact
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=oneshot
+# Adjust WorkingDirectory to the deployed path (e.g. /home/trails/brouter/poi-extract).
+WorkingDirectory=%h/brouter/poi-extract
+ExecStart=%h/brouter/poi-extract/poi-extract.sh
+# The planet download + filter is long-running and IO-heavy; give it room and
+# keep it off the box's foreground priority.
+Nice=10
+IOSchedulingClass=idle
+TimeoutStartSec=6h
+
+[Install]
+WantedBy=default.target

infrastructure/brouter-host/poi-extract/poi-extract.sh

@@ -0,0 +1,103 @@
+#!/bin/bash
+# POI extract pipeline — runs on the dedicated BRouter host under the `trails`
+# user (the box with 1.8 TB of disk). Downloads an OSM PBF, filters it to the
+# planner's POI category selectors with osmium, reduces every element to a
+# centroid, and publishes a compact NDJSON artifact + manifest that the
+# flagship import job pulls over the vSwitch.
+#
+# Nothing here is planner/journal code — it's plain osmium + python so the host
+# needs no Node runtime. The tag filter comes from `osmium-filters.txt`, which
+# is generated from `@trails-cool/map-core`'s POI selectors (single source of
+# truth; kept in sync by osmium-filters.sync.test.ts).
+#
+# Prerequisites on the host: osmium-tool, python3, curl, gzip, sha256sum.
+#
+# Usage:
+#   ./poi-extract.sh
+#
+# Env:
+#   POI_PBF_URL   OSM PBF to filter. Default: the full planet. A self-hoster or
+#                 a dry run can point this at a Geofabrik regional extract, e.g.
+#                 https://download.geofabrik.de/europe/germany-latest.osm.pbf
+#   POI_WORK_DIR  Scratch dir for the (large, transient) planet download.
+#                 Default: <script dir>/work
+#   POI_PUBLISH_DIR  Where the artifact + manifest are written for Caddy to
+#                 serve. Default: <script dir>/publish (mounted into the Caddy
+#                 sidecar at /srv/poi — see docker-compose.yml + Caddyfile).
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+FILTERS_FILE="$SCRIPT_DIR/osmium-filters.txt"
+TRANSFORMER="$SCRIPT_DIR/to-ndjson.py"
+
+POI_PBF_URL="${POI_PBF_URL:-https://planet.osm.org/pbf/planet-latest.osm.pbf}"
+POI_WORK_DIR="${POI_WORK_DIR:-$SCRIPT_DIR/work}"
+POI_PUBLISH_DIR="${POI_PUBLISH_DIR:-$SCRIPT_DIR/publish}"
+
+PLANET_PBF="$POI_WORK_DIR/planet.osm.pbf"
+FILTERED_PBF="$POI_WORK_DIR/filtered.osm.pbf"
+ARTIFACT="$POI_PUBLISH_DIR/pois.ndjson.gz"
+MANIFEST="$POI_PUBLISH_DIR/manifest.json"
+
+mkdir -p "$POI_WORK_DIR" "$POI_PUBLISH_DIR"
+
+# Working files can be huge (planet PBF ~80 GB); always clean them up, even on
+# failure, so a broken run can't fill the shared host's disk.
+cleanup() { rm -f "$PLANET_PBF" "$FILTERED_PBF"; }
+trap cleanup EXIT
+
+log() { echo "[poi-extract] $*"; }
+
+# --- 1. Read the tag filters (single source of truth: map-core) -------------
+if [ ! -s "$FILTERS_FILE" ]; then
+  echo "ERROR: $FILTERS_FILE missing/empty — regenerate with gen-osmium-filters.ts" >&2
+  exit 1
+fi
+mapfile -t FILTERS < "$FILTERS_FILE"
+log "tag filters: ${FILTERS[*]}"
+
+# --- 2. Download the PBF -----------------------------------------------------
+log "downloading $POI_PBF_URL"
+curl --fail --location --show-error --silent --output "$PLANET_PBF" "$POI_PBF_URL"
+log "downloaded $(du -h "$PLANET_PBF" | cut -f1)"
+
+# --- 3. Filter to our categories --------------------------------------------
+log "filtering with osmium tags-filter"
+osmium tags-filter --overwrite --output "$FILTERED_PBF" "$PLANET_PBF" "${FILTERS[@]}"
+log "filtered to $(du -h "$FILTERED_PBF" | cut -f1)"
+
+# --- 4. Export to centroid NDJSON -------------------------------------------
+# osmium export emits one GeoJSON feature per line; to-ndjson.py reduces each to
+# a bbox centre + compact record and reports the row count on stderr.
+log "exporting + transforming to NDJSON"
+rowcount_file="$(mktemp)"
+osmium export "$FILTERED_PBF" \
+  --output-format=geojsonseq \
+  --add-unique-id=type_id \
+  --geometry-types=point,linestring,polygon \
+  --overwrite --output - \
+  | python3 "$TRANSFORMER" 2>"$rowcount_file" \
+  | gzip -c > "$ARTIFACT"
+ROW_COUNT="$(cat "$rowcount_file")"
+rm -f "$rowcount_file"
+log "wrote $ROW_COUNT rows -> $ARTIFACT ($(du -h "$ARTIFACT" | cut -f1))"
+
+if [ "$ROW_COUNT" -eq 0 ]; then
+  echo "ERROR: 0 rows produced — refusing to publish an empty artifact" >&2
+  exit 1
+fi
+
+# --- 5. Manifest (checksum + timestamp + row count) -------------------------
+SHA256="$(sha256sum "$ARTIFACT" | cut -d' ' -f1)"
+GENERATED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+cat > "$MANIFEST" <<EOF
+{
+  "artifact": "pois.ndjson.gz",
+  "sha256": "$SHA256",
+  "row_count": $ROW_COUNT,
+  "generated_at": "$GENERATED_AT",
+  "source": "$POI_PBF_URL"
+}
+EOF
+log "manifest written: $MANIFEST"
+log "done"

infrastructure/brouter-host/poi-extract/poi-extract.timer

@@ -0,0 +1,14 @@
+# Monthly POI extract. POIs of these categories churn slowly, so monthly data
+# is plenty (Organic Maps ships roughly monthly too). Runs early on the 1st;
+# the flagship import timer is offset a day later to pull the fresh artifact.
+[Unit]
+Description=Monthly trails.cool POI extract
+
+[Timer]
+OnCalendar=*-*-01 02:00:00
+# If the box was down at the scheduled time, run once it's back.
+Persistent=true
+RandomizedDelaySec=1h
+
+[Install]
+WantedBy=timers.target

infrastructure/brouter-host/poi-extract/to-ndjson.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+"""Transform osmium's GeoJSONSeq output into the compact POI artifact.
+
+Reads a GeoJSON Feature per line on stdin (as produced by
+`osmium export -f geojsonseq --add-unique-id=type_id`) and writes one NDJSON
+object per line on stdout:
+
+    {"osm_type": "n", "osm_id": "123", "name": "Brunnen",
+     "lat": 52.52, "lon": 13.40, "tags": {"amenity": "drinking_water", ...}}
+
+Ways and relations are reduced to the centre of their bounding box — the same
+representative point Overpass returned for `out center`, so markers land where
+they used to. Category classification is intentionally NOT done here: the
+flagship importer derives categories from the tags using the map-core
+selectors (the single source of truth), so this host stays Node-free and the
+selector logic is never duplicated.
+"""
+import json
+import sys
+
+
+def coords(geometry):
+    """Yield every (lon, lat) pair in a GeoJSON geometry, any type."""
+    def walk(node):
+        if (
+            isinstance(node, list)
+            and len(node) == 2
+            and all(isinstance(n, (int, float)) for n in node)
+        ):
+            yield node
+        elif isinstance(node, list):
+            for child in node:
+                yield from walk(child)
+
+    yield from walk(geometry.get("coordinates", []))
+
+
+def bbox_center(geometry):
+    """Bounding-box centre of a geometry, matching Overpass `out center`."""
+    pts = list(coords(geometry))
+    if not pts:
+        return None
+    lons = [p[0] for p in pts]
+    lats = [p[1] for p in pts]
+    return (min(lons) + max(lons)) / 2, (min(lats) + max(lats)) / 2
+
+
+def main():
+    written = 0
+    for line in sys.stdin:
+        line = line.strip()
+        if not line:
+            continue
+        feature = json.loads(line)
+        geometry = feature.get("geometry") or {}
+        center = bbox_center(geometry)
+        if center is None:
+            continue
+        lon, lat = center
+
+        props = feature.get("properties") or {}
+        # osmium `--add-unique-id=type_id` emits ids like "n123", "w456",
+        # "r789" under the `id` / `@id` key depending on version.
+        raw_id = str(feature.get("id") or props.get("@id") or props.get("id") or "")
+        if not raw_id or raw_id[0] not in "nwr":
+            continue
+        osm_type, osm_id = raw_id[0], raw_id[1:]
+
+        # Tags are the feature properties minus osmium's synthetic id keys.
+        tags = {k: v for k, v in props.items() if k not in ("@id", "id")}
+
+        json.dump(
+            {
+                "osm_type": osm_type,
+                "osm_id": osm_id,
+                "name": tags.get("name"),
+                "lat": round(lat, 7),
+                "lon": round(lon, 7),
+                "tags": tags,
+            },
+            sys.stdout,
+            ensure_ascii=False,
+            separators=(",", ":"),
+        )
+        sys.stdout.write("\n")
+        written += 1
+
+    # Row count to stderr so the caller can record it in the manifest.
+    print(written, file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()

infrastructure/scripts/README.md

@@ -0,0 +1,45 @@
+# Flagship operational scripts
+
+Deployed to `/opt/trails-cool/scripts/` by `cd-infra.yml`.
+
+## `backup-postgres.sh`
+Daily `pg_dump` + retention sweep. Run via cron (see the header).
+
+## POI index import (`poi-import.sh`)
+
+The "load where the database is" half of the [`poi-index`](../../openspec/changes/poi-index/)
+change. Pulls the artifact the BRouter host published, classifies each element
+into `planner.pois` category rows, and swaps it in atomically.
+
+- `poi-import.sh` — fetch (vSwitch) → checksum → load → classify → **guarded
+  atomic rename swap**. `--bootstrap` skips the 70% row-count guard for the
+  first import (when the live table is legitimately empty).
+- `poi-selectors.sql` — **generated** from `@trails-cool/map-core` selectors;
+  the classifier JOINs on it. Regenerate with
+  `npx tsx infrastructure/scripts/gen-poi-selectors-sql.ts` (CI sync test:
+  `packages/map-core/src/poi-selectors-sql.sync.test.ts`).
+- `poi-import.service` / `.timer` — monthly, offset a day after the extract.
+
+### First-time setup on the flagship
+
+```bash
+# 1. Create the empty live table from the Drizzle schema (idempotent).
+#    (Run from a checkout with DATABASE_URL pointed at production, or via the
+#    normal db:push deploy path.)
+pnpm db:push
+
+# 2. Install the timer.
+cp /opt/trails-cool/scripts/poi-import.{service,timer} /etc/systemd/system/
+systemctl daemon-reload
+systemctl enable --now poi-import.timer
+
+# 3. Bootstrap the first import (guard disabled — live table is empty).
+BROUTER_AUTH_TOKEN=$(grep BROUTER_AUTH_TOKEN /opt/trails-cool/.env | cut -d= -f2-) \
+  /opt/trails-cool/scripts/poi-import.sh --bootstrap
+```
+
+Prometheus picks up import outcome from node_exporter's textfile collector
+(`poi_import_last_status`, `…_last_success_timestamp_seconds`, `…_last_rows`)
+when `NODE_EXPORTER_TEXTFILE_DIR` is set (the unit sets it). Index size and age
+are additionally exposed by the planner at `/metrics`
+(`poi_index_rows{category}`, `poi_index_age_seconds`).

infrastructure/scripts/gen-poi-selectors-sql.ts

@@ -0,0 +1,36 @@
+/**
+ * Regenerate `poi-selectors.sql` from the POI category selectors in
+ * `@trails-cool/map-core` — the single source of truth. The flagship import
+ * script (`poi-import.sh`, bash + psql, no Node) loads this to classify each
+ * extracted OSM element into one row per matching category, mirroring
+ * `matchingCategoryIds`.
+ *
+ * Run after changing `poiCategories`:
+ *   npx tsx infrastructure/scripts/gen-poi-selectors-sql.ts
+ *
+ * `packages/map-core/src/poi-selectors-sql.sync.test.ts` fails CI on drift.
+ */
+import { writeFileSync } from "node:fs";
+import { fileURLToPath } from "node:url";
+import { poiCategories } from "../../packages/map-core/src/poi.ts";
+
+function sqlLiteral(s: string): string {
+  return `'${s.replace(/'/g, "''")}'`;
+}
+
+const rows = poiCategories.flatMap((cat) =>
+  cat.selectors.map((s) => `  (${sqlLiteral(s.key)}, ${sqlLiteral(s.value)}, ${sqlLiteral(cat.id)})`),
+);
+
+const sql = `-- GENERATED from @trails-cool/map-core poiCategories. Do not edit by hand;
+-- regenerate with: npx tsx infrastructure/scripts/gen-poi-selectors-sql.ts
+-- Loaded by poi-import.sh inside the import transaction to classify each
+-- extracted OSM element into one row per matching category.
+CREATE TEMP TABLE poi_selector (key text, value text, category text) ON COMMIT DROP;
+INSERT INTO poi_selector (key, value, category) VALUES
+${rows.join(",\n")};
+`;
+
+const out = fileURLToPath(new URL("./poi-selectors.sql", import.meta.url));
+writeFileSync(out, sql);
+console.log(`Wrote ${out}`);

infrastructure/scripts/poi-import.service

@@ -0,0 +1,19 @@
+# POI import — one-shot unit driven by poi-import.timer. Runs on the flagship
+# as root (the deploy user), pulling the artifact the BRouter host published and
+# swapping it into planner.pois. Install: copy to /etc/systemd/system/, then
+# `systemctl enable --now poi-import.timer`. See infrastructure/scripts/README
+# / the poi-index change docs. Requires BROUTER_AUTH_TOKEN in the environment
+# file below (reuse the deployed /opt/trails-cool/.env).
+[Unit]
+Description=Import the trails.cool POI index artifact into planner.pois
+Wants=network-online.target
+After=network-online.target docker.service
+
+[Service]
+Type=oneshot
+WorkingDirectory=/opt/trails-cool
+EnvironmentFile=/opt/trails-cool/.env
+# Optional: expose import-outcome metrics to node_exporter's textfile collector.
+Environment=NODE_EXPORTER_TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector
+ExecStart=/opt/trails-cool/scripts/poi-import.sh
+TimeoutStartSec=2h

infrastructure/scripts/poi-import.sh

@@ -0,0 +1,162 @@
+#!/bin/bash
+# POI import — runs on the flagship. Pulls the POI artifact the BRouter host
+# published, classifies each element into category rows, and swaps it into the
+# live `planner.pois` table atomically. This is the "load where the database
+# is" half of the poi-index change.
+#
+# Serving is never interrupted: the new data is built in a side table and the
+# live table is replaced with a rename inside one transaction. A truncated
+# download can never blank the map — the swap is refused if the fresh dataset
+# has fewer than 70% of the live table's rows (override with --bootstrap on the
+# very first import, when the live table is legitimately empty).
+#
+# Prerequisites: the live table must already exist (created empty by
+# `pnpm db:push` from the Drizzle schema); curl, gunzip, sha256sum on the host.
+#
+# Usage:
+#   ./poi-import.sh [--bootstrap]
+#
+# Env:
+#   POI_ARTIFACT_BASE_URL   vSwitch URL of the published artifact dir.
+#                           Default: http://10.0.1.10:17777/poi
+#   BROUTER_AUTH_TOKEN      Shared secret Caddy requires (X-BRouter-Auth).
+#   COMPOSE_FILE            docker compose file. Default: /opt/trails-cool/docker-compose.yml
+#   NODE_EXPORTER_TEXTFILE_DIR  If set, an import-outcome .prom metric is written here.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SELECTORS_SQL="$SCRIPT_DIR/poi-selectors.sql"
+
+POI_ARTIFACT_BASE_URL="${POI_ARTIFACT_BASE_URL:-http://10.0.1.10:17777/poi}"
+COMPOSE_FILE="${COMPOSE_FILE:-/opt/trails-cool/docker-compose.yml}"
+BOOTSTRAP=0
+[ "${1:-}" = "--bootstrap" ] && BOOTSTRAP=1
+
+WORK_DIR="$(mktemp -d)"
+MANIFEST="$WORK_DIR/manifest.json"
+ARTIFACT="$WORK_DIR/pois.ndjson.gz"
+cleanup() { rm -rf "$WORK_DIR"; }
+trap cleanup EXIT
+
+log() { echo "[poi-import] $*"; }
+psql() { docker compose -f "$COMPOSE_FILE" exec -T postgres psql -U trails -d trails -v ON_ERROR_STOP=1 "$@"; }
+
+emit_metric() {
+  # Best-effort node_exporter textfile metric. status=1 success, 0 failure.
+  local status="$1" rows="${2:-0}"
+  [ -n "${NODE_EXPORTER_TEXTFILE_DIR:-}" ] || return 0
+  local now; now="$(date +%s)"
+  local tmp; tmp="$(mktemp)"
+  {
+    echo "# HELP poi_import_last_run_timestamp_seconds Unix time of the last POI import run."
+    echo "# TYPE poi_import_last_run_timestamp_seconds gauge"
+    echo "poi_import_last_run_timestamp_seconds $now"
+    echo "# HELP poi_import_last_status 1 if the last POI import swapped successfully, else 0."
+    echo "# TYPE poi_import_last_status gauge"
+    echo "poi_import_last_status $status"
+    echo "# HELP poi_import_last_rows Row count of the last successful POI import."
+    echo "# TYPE poi_import_last_rows gauge"
+    echo "poi_import_last_rows $rows"
+    if [ "$status" = "1" ]; then
+      echo "# HELP poi_import_last_success_timestamp_seconds Unix time of the last successful POI import."
+      echo "# TYPE poi_import_last_success_timestamp_seconds gauge"
+      echo "poi_import_last_success_timestamp_seconds $now"
+    fi
+  } > "$tmp"
+  mv "$tmp" "$NODE_EXPORTER_TEXTFILE_DIR/poi_import.prom"
+}
+
+fail() { log "ERROR: $*"; emit_metric 0; exit 1; }
+
+[ -s "$SELECTORS_SQL" ] || fail "$SELECTORS_SQL missing — regenerate with gen-poi-selectors-sql.ts"
+
+AUTH_HEADER=()
+[ -n "${BROUTER_AUTH_TOKEN:-}" ] && AUTH_HEADER=(-H "X-BRouter-Auth: ${BROUTER_AUTH_TOKEN}")
+
+# --- 1. Fetch manifest + artifact over the vSwitch ---------------------------
+log "fetching manifest from $POI_ARTIFACT_BASE_URL"
+curl --fail --silent --show-error "${AUTH_HEADER[@]}" -o "$MANIFEST" "$POI_ARTIFACT_BASE_URL/manifest.json" \
+  || fail "manifest fetch failed"
+EXPECTED_SHA="$(grep -o '"sha256"[[:space:]]*:[[:space:]]*"[0-9a-f]*"' "$MANIFEST" | grep -o '[0-9a-f]\{64\}')"
+MANIFEST_ROWS="$(grep -o '"row_count"[[:space:]]*:[[:space:]]*[0-9]*' "$MANIFEST" | grep -o '[0-9]*')"
+[ -n "$EXPECTED_SHA" ] || fail "manifest has no sha256"
+log "manifest: row_count=$MANIFEST_ROWS sha256=${EXPECTED_SHA:0:12}…"
+
+log "fetching artifact"
+curl --fail --silent --show-error "${AUTH_HEADER[@]}" -o "$ARTIFACT" "$POI_ARTIFACT_BASE_URL/pois.ndjson.gz" \
+  || fail "artifact fetch failed"
+
+# --- 2. Verify checksum ------------------------------------------------------
+ACTUAL_SHA="$(sha256sum "$ARTIFACT" | cut -d' ' -f1)"
+[ "$ACTUAL_SHA" = "$EXPECTED_SHA" ] || fail "checksum mismatch (got ${ACTUAL_SHA:0:12}…, want ${EXPECTED_SHA:0:12}…)"
+log "checksum verified"
+
+# --- 3. Load raw NDJSON into a staging import table --------------------------
+log "loading raw NDJSON"
+psql -c "CREATE TABLE IF NOT EXISTS planner.pois_raw_import (doc jsonb); TRUNCATE planner.pois_raw_import;"
+# Each gzip line is one JSON object. Load it as a single column value using
+# control-char delimiter/quote that never appear in JSON so the whole line
+# lands intact, then it's cast to jsonb.
+gunzip -c "$ARTIFACT" | psql -c \
+  "\copy planner.pois_raw_import (doc) FROM STDIN WITH (FORMAT csv, DELIMITER E'\x1f', QUOTE E'\x1e')" \
+  || fail "COPY failed"
+
+# --- 4. Build the staging table, classified into category rows ---------------
+# One transaction: temp selector table (from map-core) → pois_staging built
+# LIKE the live table so structure/defaults match the Drizzle schema exactly →
+# classified insert → indexes. The 70% guard + swap run in step 5.
+log "building classified staging table"
+psql <<SQL || fail "staging build failed"
+BEGIN;
+\\i $SELECTORS_SQL
+DROP TABLE IF EXISTS planner.pois_staging;
+CREATE TABLE planner.pois_staging (LIKE planner.pois INCLUDING DEFAULTS);
+INSERT INTO planner.pois_staging (osm_type, osm_id, category, name, geom, tags, imported_at)
+SELECT r.doc->>'osm_type',
+       r.doc->>'osm_id',
+       s.category,
+       r.doc->>'name',
+       ST_SetSRID(ST_MakePoint((r.doc->>'lon')::float8, (r.doc->>'lat')::float8), 4326),
+       COALESCE(r.doc->'tags', '{}'::jsonb),
+       now()
+FROM planner.pois_raw_import r
+JOIN poi_selector s ON r.doc->'tags'->>s.key = s.value;
+ALTER TABLE planner.pois_staging
+  ADD CONSTRAINT pois_staging_pkey PRIMARY KEY (osm_type, osm_id, category);
+CREATE INDEX pois_staging_geom_idx ON planner.pois_staging USING gist (geom);
+CREATE INDEX pois_staging_category_idx ON planner.pois_staging (category);
+COMMIT;
+SQL
+
+STAGING_ROWS="$(psql -tA -c "SELECT count(*) FROM planner.pois_staging;")"
+LIVE_ROWS="$(psql -tA -c "SELECT count(*) FROM planner.pois;")"
+log "staging=$STAGING_ROWS live=$LIVE_ROWS bootstrap=$BOOTSTRAP"
+[ "$STAGING_ROWS" -gt 0 ] || fail "staging is empty — refusing to swap"
+
+# --- 5. 70% guard + atomic swap ---------------------------------------------
+if [ "$BOOTSTRAP" -ne 1 ] && [ "$LIVE_ROWS" -gt 0 ]; then
+  # Integer 70% threshold: staging * 100 >= live * 70
+  if [ $(( STAGING_ROWS * 100 )) -lt $(( LIVE_ROWS * 70 )) ]; then
+    fail "guard tripped: staging ($STAGING_ROWS) < 70% of live ($LIVE_ROWS). No swap. Use --bootstrap to override."
+  fi
+fi
+
+log "swapping staging into live (atomic)"
+psql <<'SQL' || fail "swap failed"
+BEGIN;
+DROP TABLE IF EXISTS planner.pois_old;
+ALTER TABLE planner.pois RENAME TO pois_old;
+ALTER TABLE planner.pois_staging RENAME TO pois;
+-- Rename constraints/indexes to the canonical Drizzle names so a later
+-- `db:push` sees no drift. pois_old still holds the old canonical names until
+-- it's dropped, so free them first.
+DROP TABLE planner.pois_old;
+ALTER TABLE planner.pois RENAME CONSTRAINT pois_staging_pkey TO pois_osm_type_osm_id_category_pk;
+ALTER INDEX planner.pois_staging_geom_idx RENAME TO pois_geom_idx;
+ALTER INDEX planner.pois_staging_category_idx RENAME TO pois_category_idx;
+COMMIT;
+SQL
+
+psql -c "DROP TABLE IF EXISTS planner.pois_raw_import;"
+log "done: live table now has $STAGING_ROWS rows"
+emit_metric 1 "$STAGING_ROWS"

infrastructure/scripts/poi-import.timer

@@ -0,0 +1,12 @@
+# Monthly POI import, offset a day after the BRouter host's extract (which runs
+# on the 1st) so the fresh artifact is already published when this pulls it.
+[Unit]
+Description=Monthly trails.cool POI index import
+
+[Timer]
+OnCalendar=*-*-02 04:00:00
+Persistent=true
+RandomizedDelaySec=30m
+
+[Install]
+WantedBy=timers.target

infrastructure/scripts/poi-selectors.sql

@@ -0,0 +1,28 @@
+-- GENERATED from @trails-cool/map-core poiCategories. Do not edit by hand;
+-- regenerate with: npx tsx infrastructure/scripts/gen-poi-selectors-sql.ts
+-- Loaded by poi-import.sh inside the import transaction to classify each
+-- extracted OSM element into one row per matching category.
+CREATE TEMP TABLE poi_selector (key text, value text, category text) ON COMMIT DROP;
+INSERT INTO poi_selector (key, value, category) VALUES
+  ('amenity', 'drinking_water', 'drinking_water'),
+  ('amenity', 'water_point', 'drinking_water'),
+  ('amenity', 'shelter', 'shelter'),
+  ('tourism', 'wilderness_hut', 'shelter'),
+  ('tourism', 'camp_site', 'camping'),
+  ('tourism', 'caravan_site', 'camping'),
+  ('amenity', 'restaurant', 'food'),
+  ('amenity', 'cafe', 'food'),
+  ('amenity', 'fast_food', 'food'),
+  ('amenity', 'pub', 'food'),
+  ('amenity', 'biergarten', 'food'),
+  ('shop', 'supermarket', 'groceries'),
+  ('shop', 'convenience', 'groceries'),
+  ('shop', 'bakery', 'groceries'),
+  ('amenity', 'bicycle_parking', 'bike_infra'),
+  ('amenity', 'bicycle_repair_station', 'bike_infra'),
+  ('amenity', 'bicycle_rental', 'bike_infra'),
+  ('tourism', 'hotel', 'accommodation'),
+  ('tourism', 'hostel', 'accommodation'),
+  ('tourism', 'guest_house', 'accommodation'),
+  ('tourism', 'viewpoint', 'viewpoints'),
+  ('amenity', 'toilets', 'toilets');