Merge pull request #565 from trails-cool/archive-poi-index

Archive poi-index + sync specs

openspec/changes/archive/2026-07-13-poi-index/tasks.md

@@ -13,14 +13,14 @@
 - [x] 3.1 Create `infrastructure/brouter-host/poi-extract/`: script that downloads the configured PBF (`POI_PBF_URL`, planet by default), runs `osmium tags-filter` + `osmium export` (centroids) into gzipped NDJSON, writes a manifest (sha256 + timestamp + row count), and cleans up working files _(NDJSON is `{osm_type, osm_id, name, lat, lon, tags}`; the importer derives `categories` from tags via map-core, so selector logic isn't duplicated on the Node-free host)_
 - [x] 3.2 Publish artifact + manifest via the existing Caddy sidecar at a vSwitch-only address (`/poi/*` on `:17777`, same `X-BRouter-Auth` gate) _(public-unreachability follows from the existing vSwitch-only bind + UFW; verify on the host)_
 - [x] 3.3 Add a monthly systemd timer under the `trails` user; document disk/bandwidth footprint in `infrastructure/brouter-host/poi-extract/README.md`
-- [ ] 3.4 Dry-run on a small Geofabrik extract first; then a full planet run — record row counts per category and artifact size _(operational: run on the BRouter host)_
+- [x] 3.4 Dry-run on a small Geofabrik extract first; then a full planet run — record row counts per category and artifact size _(Malta smoke 2947 rows; Germany 733,986 → 41 MB; **planet 8,623,453 rows → 468 MB gz**, filtered PBF 619 MB, source planet.osm.org)_
 
 ## 4. Import job (flagship)
 
 - [x] 4.1 Create the import script: fetch manifest + artifact over the vSwitch, verify checksum, COPY into `planner.pois_staging`, build indexes, atomic rename swap in one transaction
 - [x] 4.2 Implement the 70% row-count guard with a `--bootstrap` override; abort loudly (log + metric) when tripped
 - [x] 4.3 Schedule via systemd timer (`poi-import.timer`, offset a day after the extract); wire into `cd-infra.yml` deploy sources (`infrastructure/scripts` added to the SCP list)
-- [ ] 4.4 First production import with `--bootstrap`; spot-check several viewports against live Overpass results for category parity _(operational: run on the flagship)_
+- [x] 4.4 First production import with `--bootstrap` (germany), then planet re-import (guard germany→planet passed): live `planner.pois` = **8,434,518 rows** across all nine categories; spot-checked Berlin/Fehmarn/Tokyo/NYC viewports return expected POIs. _(Two import-script bugs found + fixed in prod: `\i` selectors path inside the postgres container → stream via stdin (#563); multi-selector same-category PK collision → `DISTINCT ON` (#564).)_
 
 ## 5. Serving route
 
@@ -48,4 +48,4 @@
 
 - [x] 9.1 E2E: POI overlay flow (enable category → markers appear) — `e2e/planner-overlays.test.ts` + nearby-snap in `planner-routing.test.ts` now mock `/api/pois` (network-mock is the house pattern; a DB-seeded variant is possible but the route's SQL path is covered by `api.pois.test.ts`)
 - [x] 9.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` _(typecheck + lint + unit green; full planner e2e project green incl. all three POI tests. Journal e2e specs need `E2E=true` + the `trails_e2e` scratch DB — unrelated to this planner-only change; CI runs them with that harness.)_
-- [ ] 9.3 Post-cutover production check: enable each of the nine categories on a busy viewport and a rural viewport; confirm results and dashboard metrics _(operational: after the first production import)_
+- [x] 9.3 Post-cutover production check: verified via the cmux browser on a live session (Brandenburg) — POI markers + clusters render incl. camping tents; `/api/pois` returns data for busy (Berlin/Tokyo/NYC) and rural (Fehmarn) viewports; all nine categories populated. _(Remaining manual op step: enable the two monthly systemd timers — `poi-extract.timer` on the BRouter host, `poi-import.timer` on the flagship — a persistence change to apply by hand; see the poi-extract + scripts READMEs.)_

openspec/specs/infrastructure/spec.md

@@ -192,3 +192,22 @@ The BRouter host SHALL be administered by the trails.cool project through a non-
 - **WHEN** files are created by the deploy or segment-download scripts
 - **THEN** they live under `~trails/brouter/` and are owned by `trails:trails`
 
+### Requirement: POI extract pipeline on the BRouter host
+The BRouter host SHALL run a scheduled job (monthly by default) that downloads an OSM data file, filters it to the planner's POI category selectors with osmium, and publishes the resulting artifact with a checksum-and-timestamp manifest at a vSwitch-only URL via the existing Caddy sidecar. Transient working data SHALL live under the `trails` user's directory and be cleaned up after each run.
+
+#### Scenario: Monthly extract published
+- **WHEN** the scheduled extract job completes
+- **THEN** a fresh artifact and manifest are available to the flagship over the vSwitch
+- **AND** the planet working files are removed from disk
+
+#### Scenario: Artifact not publicly reachable
+- **WHEN** a host outside the vSwitch requests the artifact URL
+- **THEN** the request is refused
+
+### Requirement: POI import job on the flagship
+The flagship SHALL run a scheduled import job that fetches the artifact over the vSwitch, verifies its checksum, loads it into the staging table, and performs the guarded atomic swap. Job outcome and index age SHALL be visible in the monitoring stack.
+
+#### Scenario: Import failure is visible
+- **WHEN** an import run fails (fetch, checksum, load, or guard)
+- **THEN** the failure is recorded in metrics/logs and surfaces on the Grafana planner dashboard
+

openspec/specs/osm-poi-overlays/spec.md

@@ -16,11 +16,11 @@ The Planner SHALL provide a collapsible panel for toggling POI categories on the
 - **THEN** the panel collapses and POI markers remain visible on the map
 
 ### Requirement: POI categories
-The Planner SHALL support the following POI categories queried from OpenStreetMap via Overpass API: drinking water, shelter, camping, food & drink, groceries, bike infrastructure, accommodation, viewpoints, and toilets.
+The Planner SHALL support the following POI categories served from the instance's own POI index (`/api/pois`): drinking water, shelter, camping, food & drink, groceries, bike infrastructure, accommodation, viewpoints, and toilets.
 
 #### Scenario: Enable a POI category
 - **WHEN** a user enables the "Drinking water" category in the POI panel
-- **THEN** drinking water POIs within the current map viewport are fetched from Overpass and rendered as markers
+- **THEN** drinking water POIs within the current map viewport are fetched from the instance's POI index and rendered as markers
 
 #### Scenario: Disable a POI category
 - **WHEN** a user disables a previously enabled POI category
@@ -58,19 +58,17 @@ The Planner SHALL load POIs only within the current map viewport, refreshing whe
 
 #### Scenario: Cached results
 - **WHEN** the user pans back to a previously viewed area within 10 minutes
-- **THEN** cached POI results are displayed without a new Overpass query
+- **THEN** cached POI results are displayed without a new POI request
 
-### Requirement: Overpass rate limit handling
-The Planner SHALL handle Overpass API rate limits gracefully.
+### Requirement: POI service degradation handling
+The Planner SHALL handle POI endpoint failures gracefully.
 
 #### Scenario: Rate limited response
-- **WHEN** the Overpass API returns a 429 status
+- **WHEN** `/api/pois` returns a 429 status
 - **THEN** the Planner shows a temporary "POI data unavailable — try again shortly" message and sets a backoff delay before the next request
 
-Note: automatic retry is not implemented. The next request fires only on the next user viewport change or category toggle after the backoff delay has elapsed.
-
-#### Scenario: Overpass unavailable
-- **WHEN** the Overpass API is unreachable
+#### Scenario: POI service unavailable
+- **WHEN** `/api/pois` is unreachable or returns a server error
 - **THEN** the Planner shows a message and tile overlays continue to function normally
 
 ### Requirement: Profile-aware POI defaults

openspec/specs/poi-index/spec.md

@@ -0,0 +1,49 @@
+## Purpose
+
+A self-hosted POI index that serves OpenStreetMap points of interest for the Planner's POI categories from the instance's own database, replacing the third-party Overpass API. The index is refreshed on a schedule via an OSM extract pipeline and served through a same-origin, rate-limited endpoint with no third-party dependency at request time.
+
+## Requirements
+
+### Requirement: POI index table
+The instance SHALL maintain a `planner.pois` table of OSM points of interest for the planner's POI categories, storing OSM type and id, category, optional name, centroid point geometry, the element's tags, and import timestamp, indexed for bbox-and-category lookup. Category membership SHALL be derived from the structured tag selectors defined in `@trails-cool/map-core`, which are the single source of truth for the pipeline filter and import-time classification.
+
+#### Scenario: Bbox query by category
+- **WHEN** the serving layer queries the index for "drinking water" within a viewport bbox
+- **THEN** all indexed elements whose tags match that category's selectors and whose centroid lies in the bbox are returned
+
+#### Scenario: Way POIs appear as points
+- **WHEN** an OSM way (e.g. a campground area) matches a category selector
+- **THEN** it is indexed with its centroid as the point geometry
+
+### Requirement: Periodic import with atomic swap
+The instance SHALL refresh the POI index on a schedule (monthly by default) by filtering an OSM data file to the category selectors, loading the result into a staging table, and swapping it in atomically so serving is never interrupted. The import SHALL abort without swapping when the staging row count falls below 70% of the live table's row count, unless bootstrap mode is explicitly enabled.
+
+#### Scenario: Refresh replaces data without downtime
+- **WHEN** a scheduled import completes successfully
+- **THEN** the live table reflects the new dataset and no serving request observed an empty or partial table
+
+#### Scenario: Shrunken dataset refused
+- **WHEN** an import produces fewer than 70% of the live table's rows (e.g. truncated download)
+- **THEN** no swap occurs, the live data stays untouched, and the failure is logged and visible in metrics
+
+#### Scenario: Regional extract for self-hosters
+- **WHEN** an operator configures a regional extract URL instead of the planet file
+- **THEN** the same pipeline imports that region only
+
+### Requirement: POI serving endpoint
+The Planner SHALL serve POIs from the index via `/api/pois`, accepting a bbox and category list, enforcing the same-origin and session checks and the per-IP rate limit that the Overpass proxy enforced, and capping results at 100 per request. The response SHALL provide the fields the client's `Poi` shape requires (id, coordinates, name, category, tags). POI requests SHALL NOT contact any third-party service.
+
+#### Scenario: Happy-path query
+- **WHEN** the browser requests two enabled categories for the current viewport
+- **THEN** the endpoint returns matching POIs from the local index with their tags
+
+#### Scenario: Missing index degrades gracefully
+- **WHEN** the POI table is empty or absent (fresh self-hosted instance)
+- **THEN** the endpoint returns an empty result or a service-unavailable status without errors that break the map
+
+### Requirement: Index freshness observability
+The instance SHALL expose metrics for index size per category, index age, import job outcome, and serving request counts, and the monitoring stack SHALL alert when the index age indicates a missed refresh.
+
+#### Scenario: Stale index alerts
+- **WHEN** the index age exceeds the alert threshold (~6 weeks)
+- **THEN** an alert fires identifying the POI import as unhealthy

openspec/specs/rate-limiting/spec.md

@@ -18,9 +18,10 @@ The Planner SHALL limit route computations to 300 per session per hour (the `DEF
 - **WHEN** a session exceeds 300 BRouter calls in one hour
 - **THEN** the server responds with 429 and the client shows a "slow down" message
 
-### Requirement: Overpass API rate limit
-The Planner SHALL limit Overpass API calls to 120 per IP per minute to protect the upstream service.
+### Requirement: POI API rate limit
+The Planner SHALL limit `/api/pois` requests to 120 per IP per minute to protect the instance's database from abusive clients.
 
-#### Scenario: Overpass rate limit exceeded
-- **WHEN** a single IP exceeds 120 Overpass requests in one minute
-- **THEN** the `/api/overpass` proxy responds with 429 Too Many Requests
+#### Scenario: POI rate limit exceeded
+- **WHEN** a single IP exceeds 120 POI requests in one minute
+- **THEN** `/api/pois` responds with 429 Too Many Requests
+- **AND** no database query is executed for rejected requests