trails/openspec/changes/poi-index/design.md

Context

POI queries flow: browser (lib/overpass.ts, bbox quantized to a 0.01° grid) → /api/overpass proxy (same-origin + session checks, per-IP rate limit, response cache with coalescing, Grafana metrics) → overpass.private.coffee. All nine categories (packages/map-core/src/poi.ts) are unions of nwr["key"="value"] selectors with out center qt 100 — tag equality in a bbox, capped at 100 results, ways/relations reduced to center points. Nearby-POI lookup and snap-to-POI reuse the same path with a small bbox.

Two prior artifacts inform this change: the parked docs/ideas/self-host-overpass/ plan (full Overpass instance on the maintainer's dedicated box; heavy: regional DB, diff replication, ~23 GB RAM ceiling limits it to a DACH extract) and our Organic Maps review (2026-07-05), whose generator pipeline demonstrates the lighter pattern adopted here: batch-filter OSM data offline into a compact owned index, serve reads from that.

Relevant infra already in place: the BRouter host has 1.8 TB free disk, a Caddy sidecar, and a vSwitch link to the flagship (10.0.0.2); the flagship runs PostGIS and the planner already has a DB connection (planner.* schema, sessions).

Goals / Non-Goals

Goals:

• Zero third-party dependency for POI data; viewport coordinates never leave trails.cool infrastructure.
• Planet-wide coverage (the current upstream is planet-wide; a regional index would be a regression).
• Identical user-facing behavior: same categories, marker data, popups, caps, and degradation UI.
• Boring operations: a monthly batch job with an atomic swap and a freshness metric — no replication daemon.

Non-Goals:

• Overpass QL compatibility (nothing uses it; the parked self-host-Overpass plan covers that path if ever needed).
• POI name search, additional categories, journal consumption, or OSM editing — all possible later on top of the same table.
• Minute-fresh data. POIs of these kinds churn slowly; monthly is enough (Organic Maps ships roughly monthly map data too).

Decisions

1. Data model: one flat planner.pois table, centroid-only

planner.pois: osm_type (n/w/r), osm_id, category, name (nullable), geom (geometry(Point, 4326) — centroid for ways/relations, matching today's out center), tags (jsonb, the element's full tag set for popups: opening hours, website, OSM link), imported_at. Primary key (osm_type, osm_id, category) — an element matching two categories appears once per category, which is how the client already treats results. Indexes: GiST on geom, btree on category. Serving query: WHERE category = ANY($cats) AND geom && ST_MakeEnvelope($bbox) with the existing 100-result cap.

Alternative: normalized category join table — rejected; the categories are a small fixed set and the flat table keeps the import a single COPY.

2. Category selectors move to structured data in map-core

poiCategories entries change from Overpass QL strings to selectors: Array<{key, value}>. Three consumers derive from this one definition: the pipeline's osmium filter expression, the importer's category classification, and (no longer) any client-side query building. This keeps "what is a shelter" defined in exactly one place — the same property the QL strings had, preserved across the migration.

3. Pipeline topology: filter where the disk is, load where the database is

• BRouter host (monthly systemd timer): download planet PBF (~80 GB, kept only transiently), osmium tag-filter to our selectors (output: a few hundred MB), osmium export with centroid geometry to a gzipped NDJSON artifact {osm_type, osm_id, category[], name, lat, lon, tags}, publish it at a vSwitch-only path via the existing Caddy sidecar alongside a checksum + timestamp manifest.
• Flagship (systemd timer, offset a day): fetch manifest + artifact over the vSwitch, COPY into planner.pois_staging, build indexes, then swap inside one transaction (ALTER TABLE ... RENAME).
• Sanity guard: the importer aborts (no swap, loud log + metric) if the staging row count is < 70% of the live table's — a truncated download or broken filter can never blank the map. First import bootstraps with the guard disabled via flag.

Alternatives: streaming the planet through CI — GitHub runners lack the disk; importing directly from the BRouter host into flagship Postgres — rejected to avoid exposing Postgres on the vSwitch; a pg-boss job on the flagship — rejected because the heavy half (planet download/filter) can't run there anyway, and two small systemd timers are simpler than splitting one job across pg-boss and cron.

4. Serving route /api/pois, client keeps its shape

New apps/planner/app/routes/api.pois.ts (GET, bbox + categories params): same-origin + session checks, 120/IP/min rate limit (unchanged number — it now protects our DB), result cap 100, short Cache-Control so repeated identical viewport requests can be browser-cached. lib/overpass.ts is renamed/rewritten as a thin client keeping quantizeBbox (still valuable for client-cache hits), the Poi type, and the error mapping — so use-pois, use-nearby-pois, poi-cache, snap-to-POI, markers, and popups need no changes. The server-side response cache + coalescing from the proxy are dropped: a GiST-indexed point lookup is single-digit milliseconds and no upstream needs protecting.

5. Hard cutover, no dual-source period

/api/overpass and its private.coffee upstream are deleted in the same PR that lands /api/pois. Rollback for code problems is a git revert; the plausible non-code failure (import breaks later) degrades to stale POIs — objectively better than today's failure mode (no POIs during upstream flaps) — and is caught by the freshness metric. Keeping both paths alive would double the test surface for no benefit given the guard + metric.

6. Observability

New metrics replacing overpass_upstream_*: poi_index_rows{category}, poi_index_age_seconds (from the manifest timestamp), poi_api_requests_total{status}, import job success/failure. Grafana planner dashboard: upstream panels replaced by index freshness + serving panels; alert when index age exceeds ~6 weeks (one missed monthly run).

Risks / Trade-offs

• [Planet download monthly on the BRouter host uses bandwidth/disk on a shared box] → ~80 GB transient monthly, scoped under ~trails/; the box has 1.8 TB free and the download can use a Geofabrik mirror at off-peak hours. If it becomes unwelcome, per-continent Geofabrik extracts halve the footprint.
• [Data staleness up to a month (+ alert window)] → Acceptable for these categories; cadence is a timer setting if users report pain.
• [Row estimate off / table larger than expected on the 40 GB flagship SSD] → Measured before cutover in task order (filter first, count, then size the table); 9 categories ≈ 10M rows ≈ low single-digit GB with indexes. If planet-wide jsonb tags blow the budget, store a tag allowlist (popup fields only).
• [Self-hosters inherit a pipeline] → Documented as optional: the compose stack works with an empty/absent POI table (overlays show "unavailable"), and the import script accepts any Geofabrik extract URL so a small instance can index just its region cheaply.
• [Federated/self-host story diverges from flagship topology (no second host)] → The pipeline is two scripts; on a single-host instance both run on that host with a regional extract. Called out in the docs task.

Migration Plan

1. Land schema + pipeline + route behind no user-visible change (client still on proxy).
2. Run the first import end-to-end on production data; verify counts, spot-check categories against the live Overpass results for sample viewports.
3. Switch the client to /api/pois, delete the proxy, swap dashboards — one PR.
4. Update docs/architecture.md, the privacy notes, and the parked idea README (superseded pointer).

Rollback: revert step-3 PR (proxy code returns, upstream still exists); the index table is additive and harmless to leave in place.

Open Questions

• None blocking. Whether small self-hosted instances should get a prebuilt regional artifact from trails.cool (so they skip osmium entirely) is a nice-to-have to decide when someone asks.