openspec: archive poi-index; sync deltas into canonical specs
poi-index shipped and is live in production (self-hosted planet POI index, 8.4M rows serving /api/pois; Overpass removed from the Planner). Archive the change and apply its spec deltas: - new capability: poi-index - osm-poi-overlays: POIs from the instance index, not Overpass - rate-limiting: /api/pois limit replaces the Overpass proxy limit - infrastructure: POI extract (BRouter host) + import (flagship) components Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer
parent
5539b34613
commit
8396041c26
12 changed files with 82 additions and 15 deletions
|
|
@ -0,0 +1,2 @@
|
|||
schema: spec-driven
|
||||
created: 2026-07-05
|
||||
73
openspec/changes/archive/2026-07-13-poi-index/design.md
Normal file
73
openspec/changes/archive/2026-07-13-poi-index/design.md
Normal file
|
|
@ -0,0 +1,73 @@
|
|||
## Context
|
||||
|
||||
POI queries flow: browser (`lib/overpass.ts`, bbox quantized to a 0.01° grid) → `/api/overpass` proxy (same-origin + session checks, per-IP rate limit, response cache with coalescing, Grafana metrics) → `overpass.private.coffee`. All nine categories (`packages/map-core/src/poi.ts`) are unions of `nwr["key"="value"]` selectors with `out center qt 100` — tag equality in a bbox, capped at 100 results, ways/relations reduced to center points. Nearby-POI lookup and snap-to-POI reuse the same path with a small bbox.
|
||||
|
||||
Two prior artifacts inform this change: the parked `docs/ideas/self-host-overpass/` plan (full Overpass instance on the maintainer's dedicated box; heavy: regional DB, diff replication, ~23 GB RAM ceiling limits it to a DACH extract) and our Organic Maps review (2026-07-05), whose generator pipeline demonstrates the lighter pattern adopted here: batch-filter OSM data offline into a compact owned index, serve reads from that.
|
||||
|
||||
Relevant infra already in place: the BRouter host has 1.8 TB free disk, a Caddy sidecar, and a vSwitch link to the flagship (`10.0.0.2`); the flagship runs PostGIS and the planner already has a DB connection (`planner.*` schema, sessions).
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Zero third-party dependency for POI data; viewport coordinates never leave trails.cool infrastructure.
|
||||
- Planet-wide coverage (the current upstream is planet-wide; a regional index would be a regression).
|
||||
- Identical user-facing behavior: same categories, marker data, popups, caps, and degradation UI.
|
||||
- Boring operations: a monthly batch job with an atomic swap and a freshness metric — no replication daemon.
|
||||
|
||||
**Non-Goals:**
|
||||
- Overpass QL compatibility (nothing uses it; the parked self-host-Overpass plan covers that path if ever needed).
|
||||
- POI name search, additional categories, journal consumption, or OSM editing — all possible later on top of the same table.
|
||||
- Minute-fresh data. POIs of these kinds churn slowly; monthly is enough (Organic Maps ships roughly monthly map data too).
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Data model: one flat `planner.pois` table, centroid-only
|
||||
|
||||
`planner.pois`: `osm_type` (`n`/`w`/`r`), `osm_id`, `category`, `name` (nullable), `geom` (`geometry(Point, 4326)` — centroid for ways/relations, matching today's `out center`), `tags` (`jsonb`, the element's full tag set for popups: opening hours, website, OSM link), `imported_at`. Primary key `(osm_type, osm_id, category)` — an element matching two categories appears once per category, which is how the client already treats results. Indexes: GiST on `geom`, btree on `category`. Serving query: `WHERE category = ANY($cats) AND geom && ST_MakeEnvelope($bbox)` with the existing 100-result cap.
|
||||
|
||||
*Alternative:* normalized category join table — rejected; the categories are a small fixed set and the flat table keeps the import a single `COPY`.
|
||||
|
||||
### 2. Category selectors move to structured data in `map-core`
|
||||
|
||||
`poiCategories` entries change from Overpass QL strings to `selectors: Array<{key, value}>`. Three consumers derive from this one definition: the pipeline's osmium filter expression, the importer's category classification, and (no longer) any client-side query building. This keeps "what is a shelter" defined in exactly one place — the same property the QL strings had, preserved across the migration.
|
||||
|
||||
### 3. Pipeline topology: filter where the disk is, load where the database is
|
||||
|
||||
- **BRouter host (monthly systemd timer)**: download planet PBF (~80 GB, kept only transiently), `osmium tag-filter` to our selectors (output: a few hundred MB), `osmium export` with centroid geometry to a gzipped NDJSON artifact `{osm_type, osm_id, category[], name, lat, lon, tags}`, publish it at a vSwitch-only path via the existing Caddy sidecar alongside a checksum + timestamp manifest.
|
||||
- **Flagship (systemd timer, offset a day)**: fetch manifest + artifact over the vSwitch, `COPY` into `planner.pois_staging`, build indexes, then swap inside one transaction (`ALTER TABLE ... RENAME`).
|
||||
- **Sanity guard**: the importer aborts (no swap, loud log + metric) if the staging row count is < 70% of the live table's — a truncated download or broken filter can never blank the map. First import bootstraps with the guard disabled via flag.
|
||||
|
||||
*Alternatives:* streaming the planet through CI — GitHub runners lack the disk; importing directly from the BRouter host into flagship Postgres — rejected to avoid exposing Postgres on the vSwitch; a pg-boss job on the flagship — rejected because the heavy half (planet download/filter) can't run there anyway, and two small systemd timers are simpler than splitting one job across pg-boss and cron.
|
||||
|
||||
### 4. Serving route `/api/pois`, client keeps its shape
|
||||
|
||||
New `apps/planner/app/routes/api.pois.ts` (GET, `bbox` + `categories` params): same-origin + session checks, 120/IP/min rate limit (unchanged number — it now protects our DB), result cap 100, short `Cache-Control` so repeated identical viewport requests can be browser-cached. `lib/overpass.ts` is renamed/rewritten as a thin client keeping `quantizeBbox` (still valuable for client-cache hits), the `Poi` type, and the error mapping — so `use-pois`, `use-nearby-pois`, `poi-cache`, snap-to-POI, markers, and popups need no changes. The server-side response cache + coalescing from the proxy are dropped: a GiST-indexed point lookup is single-digit milliseconds and no upstream needs protecting.
|
||||
|
||||
### 5. Hard cutover, no dual-source period
|
||||
|
||||
`/api/overpass` and its private.coffee upstream are deleted in the same PR that lands `/api/pois`. Rollback for code problems is a git revert; the plausible non-code failure (import breaks later) degrades to *stale* POIs — objectively better than today's failure mode (no POIs during upstream flaps) — and is caught by the freshness metric. Keeping both paths alive would double the test surface for no benefit given the guard + metric.
|
||||
|
||||
### 6. Observability
|
||||
|
||||
New metrics replacing `overpass_upstream_*`: `poi_index_rows{category}`, `poi_index_age_seconds` (from the manifest timestamp), `poi_api_requests_total{status}`, import job success/failure. Grafana planner dashboard: upstream panels replaced by index freshness + serving panels; alert when index age exceeds ~6 weeks (one missed monthly run).
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Planet download monthly on the BRouter host uses bandwidth/disk on a shared box] → ~80 GB transient monthly, scoped under `~trails/`; the box has 1.8 TB free and the download can use a Geofabrik mirror at off-peak hours. If it becomes unwelcome, per-continent Geofabrik extracts halve the footprint.
|
||||
- [Data staleness up to a month (+ alert window)] → Acceptable for these categories; cadence is a timer setting if users report pain.
|
||||
- [Row estimate off / table larger than expected on the 40 GB flagship SSD] → Measured before cutover in task order (filter first, count, then size the table); 9 categories ≈ 10M rows ≈ low single-digit GB with indexes. If planet-wide jsonb tags blow the budget, store a tag allowlist (popup fields only).
|
||||
- [Self-hosters inherit a pipeline] → Documented as optional: the compose stack works with an empty/absent POI table (overlays show "unavailable"), and the import script accepts any Geofabrik extract URL so a small instance can index just its region cheaply.
|
||||
- [Federated/self-host story diverges from flagship topology (no second host)] → The pipeline is two scripts; on a single-host instance both run on that host with a regional extract. Called out in the docs task.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Land schema + pipeline + route behind no user-visible change (client still on proxy).
|
||||
2. Run the first import end-to-end on production data; verify counts, spot-check categories against the live Overpass results for sample viewports.
|
||||
3. Switch the client to `/api/pois`, delete the proxy, swap dashboards — one PR.
|
||||
4. Update `docs/architecture.md`, the privacy notes, and the parked idea README (superseded pointer).
|
||||
|
||||
Rollback: revert step-3 PR (proxy code returns, upstream still exists); the index table is additive and harmless to leave in place.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- None blocking. Whether small self-hosted instances should get a prebuilt regional artifact from trails.cool (so they skip osmium entirely) is a nice-to-have to decide when someone asks.
|
||||
32
openspec/changes/archive/2026-07-13-poi-index/proposal.md
Normal file
32
openspec/changes/archive/2026-07-13-poi-index/proposal.md
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
## Why
|
||||
|
||||
The planner's POI overlays depend on a third-party Overpass instance (`overpass.private.coffee` behind our `/api/overpass` proxy). Public Overpass servers flake (429s and stalls surface as 502s on our dashboard), coverage and policy are outside our control, and viewport coordinates still leave our infrastructure. Every current query is a simple tag-equality lookup within a bbox — none of Overpass QL's power is used — so we can replace the dependency entirely with our own PostGIS-backed POI index fed by periodic OSM extracts (the Organic Maps pattern: precompute at build time instead of querying a live third-party engine). This supersedes the parked `docs/ideas/self-host-overpass/` plan with something lighter: no Overpass software at all.
|
||||
|
||||
## What Changes
|
||||
|
||||
- **New POI index**: a `planner.pois` table (OSM id/type, category, name, centroid geometry, tags) holding planet-wide POIs for the planner's categories, with GiST/category indexes.
|
||||
- **Refresh pipeline**: a scheduled job on the BRouter host (1.8 TB disk) downloads the OSM planet file, filters it to our category selectors with osmium, and publishes a compact artifact over the vSwitch; a flagship-side import script loads it into a staging table and atomically swaps it in, refusing to swap if the new dataset shrinks implausibly.
|
||||
- **Single source of truth for categories**: `@trails-cool/map-core` POI categories change from Overpass QL strings to structured tag selectors (`{key, value}` lists) that drive both the pipeline filter and request-time classification.
|
||||
- **New serving route**: planner `/api/pois` (bbox + categories) queries the index with the same result cap, same-origin/session checks, and per-IP rate limit the proxy has today; the browser client keeps its bbox quantization, client cache, debounce, and error UI.
|
||||
- **BREAKING**: the `/api/overpass` proxy route and the `overpass.private.coffee` dependency are removed; no POI traffic leaves trails.cool infrastructure. Grafana's Overpass upstream panels are replaced by index freshness/serving metrics.
|
||||
- The parked `docs/ideas/self-host-overpass/` README is updated to point here as superseded.
|
||||
- Not in scope: POI search by name, categories beyond the current nine, journal use of the index, and editing/contributing back to OSM.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `poi-index`: The self-hosted POI dataset — import pipeline (selectors, cadence, atomic swap, sanity guard, freshness observability) and the serving contract (`/api/pois` request/response, caps, access checks).
|
||||
|
||||
### Modified Capabilities
|
||||
- `osm-poi-overlays`: POI data comes from the instance's own index instead of the Overpass API; the "Overpass rate limit handling" requirement is replaced by source-neutral degradation handling for the planner's own POI endpoint.
|
||||
- `rate-limiting`: the "Overpass API rate limit" requirement (120/IP/min on `/api/overpass`) is replaced by an equivalent limit on `/api/pois`, now protecting our own database rather than an upstream.
|
||||
- `infrastructure`: adds the scheduled extract job on the BRouter host, artifact transfer over the existing vSwitch, and the flagship import job + table as operated components with freshness monitoring.
|
||||
|
||||
## Impact
|
||||
|
||||
- `packages/map-core/src/poi.ts` — QL strings → structured selectors (planner client + pipeline both consume).
|
||||
- `apps/planner`: new `routes/api.pois.ts`; `lib/overpass.ts` becomes a thin `/api/pois` client (Poi shape unchanged, so `use-pois`, `use-nearby-pois`, snap-to-poi, markers, and popups are untouched); `routes/api.overpass.ts` deleted; metrics swapped.
|
||||
- `packages/db` — `planner.pois` schema + migration.
|
||||
- `infrastructure/brouter-host/` — extract job (osmium) + artifact publishing via the existing Caddy sidecar; flagship import script + schedule; Grafana dashboard update.
|
||||
- Privacy: planner viewport coordinates no longer sent to any third party — strengthens the zero-collection posture.
|
||||
- Data: planet filtered to nine categories ≈ 10M rows, single-digit GB in PostGIS — fits the flagship; the planet PBF (~80 GB) only ever lives on the BRouter host.
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
## ADDED Requirements
|
||||
|
||||
### Requirement: POI extract pipeline on the BRouter host
|
||||
The BRouter host SHALL run a scheduled job (monthly by default) that downloads an OSM data file, filters it to the planner's POI category selectors with osmium, and publishes the resulting artifact with a checksum-and-timestamp manifest at a vSwitch-only URL via the existing Caddy sidecar. Transient working data SHALL live under the `trails` user's directory and be cleaned up after each run.
|
||||
|
||||
#### Scenario: Monthly extract published
|
||||
- **WHEN** the scheduled extract job completes
|
||||
- **THEN** a fresh artifact and manifest are available to the flagship over the vSwitch
|
||||
- **AND** the planet working files are removed from disk
|
||||
|
||||
#### Scenario: Artifact not publicly reachable
|
||||
- **WHEN** a host outside the vSwitch requests the artifact URL
|
||||
- **THEN** the request is refused
|
||||
|
||||
### Requirement: POI import job on the flagship
|
||||
The flagship SHALL run a scheduled import job that fetches the artifact over the vSwitch, verifies its checksum, loads it into the staging table, and performs the guarded atomic swap. Job outcome and index age SHALL be visible in the monitoring stack.
|
||||
|
||||
#### Scenario: Import failure is visible
|
||||
- **WHEN** an import run fails (fetch, checksum, load, or guard)
|
||||
- **THEN** the failure is recorded in metrics/logs and surfaces on the Grafana planner dashboard
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: POI categories
|
||||
The Planner SHALL support the following POI categories served from the instance's own POI index (`/api/pois`): drinking water, shelter, camping, food & drink, groceries, bike infrastructure, accommodation, viewpoints, and toilets.
|
||||
|
||||
#### Scenario: Enable a POI category
|
||||
- **WHEN** a user enables the "Drinking water" category in the POI panel
|
||||
- **THEN** drinking water POIs within the current map viewport are fetched from the instance's POI index and rendered as markers
|
||||
|
||||
#### Scenario: Disable a POI category
|
||||
- **WHEN** a user disables a previously enabled POI category
|
||||
- **THEN** markers for that category are removed from the map
|
||||
|
||||
#### Scenario: Multiple categories enabled
|
||||
- **WHEN** a user enables "Camping" and "Drinking water" simultaneously
|
||||
- **THEN** both categories of markers are visible, each with distinct icons
|
||||
|
||||
### Requirement: Viewport-scoped POI loading
|
||||
The Planner SHALL load POIs only within the current map viewport, refreshing when the viewport changes.
|
||||
|
||||
#### Scenario: Load POIs on viewport
|
||||
- **WHEN** POI categories are enabled and the user pans or zooms the map
|
||||
- **THEN** POIs are fetched for the new viewport after an 800ms debounce (`DEBOUNCE_MS` in `use-pois.ts`); a minimum request interval of 2000ms additionally throttles rapid viewport changes
|
||||
|
||||
#### Scenario: Zoom threshold
|
||||
- **WHEN** the map zoom level is below 10 (`MIN_ZOOM` constant in `use-pois.ts`)
|
||||
- **THEN** POI queries are not sent and a message indicates the user should zoom in to see POIs
|
||||
|
||||
#### Scenario: Cached results
|
||||
- **WHEN** the user pans back to a previously viewed area within 10 minutes
|
||||
- **THEN** cached POI results are displayed without a new POI request
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
### Requirement: Overpass rate limit handling
|
||||
**Reason**: The Overpass API dependency is removed; POIs are served from the instance's own index. Degradation handling is re-specified source-neutrally below.
|
||||
**Migration**: The client error UI is unchanged; it now reacts to 429/5xx from `/api/pois` instead of Overpass statuses.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: POI service degradation handling
|
||||
The Planner SHALL handle POI endpoint failures gracefully.
|
||||
|
||||
#### Scenario: Rate limited response
|
||||
- **WHEN** `/api/pois` returns a 429 status
|
||||
- **THEN** the Planner shows a temporary "POI data unavailable — try again shortly" message and sets a backoff delay before the next request
|
||||
|
||||
#### Scenario: POI service unavailable
|
||||
- **WHEN** `/api/pois` is unreachable or returns a server error
|
||||
- **THEN** the Planner shows a message and tile overlays continue to function normally
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
## ADDED Requirements
|
||||
|
||||
### Requirement: POI index table
|
||||
The instance SHALL maintain a `planner.pois` table of OSM points of interest for the planner's POI categories, storing OSM type and id, category, optional name, centroid point geometry, the element's tags, and import timestamp, indexed for bbox-and-category lookup. Category membership SHALL be derived from the structured tag selectors defined in `@trails-cool/map-core`, which are the single source of truth for the pipeline filter and import-time classification.
|
||||
|
||||
#### Scenario: Bbox query by category
|
||||
- **WHEN** the serving layer queries the index for "drinking water" within a viewport bbox
|
||||
- **THEN** all indexed elements whose tags match that category's selectors and whose centroid lies in the bbox are returned
|
||||
|
||||
#### Scenario: Way POIs appear as points
|
||||
- **WHEN** an OSM way (e.g. a campground area) matches a category selector
|
||||
- **THEN** it is indexed with its centroid as the point geometry
|
||||
|
||||
### Requirement: Periodic import with atomic swap
|
||||
The instance SHALL refresh the POI index on a schedule (monthly by default) by filtering an OSM data file to the category selectors, loading the result into a staging table, and swapping it in atomically so serving is never interrupted. The import SHALL abort without swapping when the staging row count falls below 70% of the live table's row count, unless bootstrap mode is explicitly enabled.
|
||||
|
||||
#### Scenario: Refresh replaces data without downtime
|
||||
- **WHEN** a scheduled import completes successfully
|
||||
- **THEN** the live table reflects the new dataset and no serving request observed an empty or partial table
|
||||
|
||||
#### Scenario: Shrunken dataset refused
|
||||
- **WHEN** an import produces fewer than 70% of the live table's rows (e.g. truncated download)
|
||||
- **THEN** no swap occurs, the live data stays untouched, and the failure is logged and visible in metrics
|
||||
|
||||
#### Scenario: Regional extract for self-hosters
|
||||
- **WHEN** an operator configures a regional extract URL instead of the planet file
|
||||
- **THEN** the same pipeline imports that region only
|
||||
|
||||
### Requirement: POI serving endpoint
|
||||
The Planner SHALL serve POIs from the index via `/api/pois`, accepting a bbox and category list, enforcing the same-origin and session checks and the per-IP rate limit that the Overpass proxy enforced, and capping results at 100 per request. The response SHALL provide the fields the client's `Poi` shape requires (id, coordinates, name, category, tags). POI requests SHALL NOT contact any third-party service.
|
||||
|
||||
#### Scenario: Happy-path query
|
||||
- **WHEN** the browser requests two enabled categories for the current viewport
|
||||
- **THEN** the endpoint returns matching POIs from the local index with their tags
|
||||
|
||||
#### Scenario: Missing index degrades gracefully
|
||||
- **WHEN** the POI table is empty or absent (fresh self-hosted instance)
|
||||
- **THEN** the endpoint returns an empty result or a service-unavailable status without errors that break the map
|
||||
|
||||
### Requirement: Index freshness observability
|
||||
The instance SHALL expose metrics for index size per category, index age, import job outcome, and serving request counts, and the monitoring stack SHALL alert when the index age indicates a missed refresh.
|
||||
|
||||
#### Scenario: Stale index alerts
|
||||
- **WHEN** the index age exceeds the alert threshold (~6 weeks)
|
||||
- **THEN** an alert fires identifying the POI import as unhealthy
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
## REMOVED Requirements
|
||||
|
||||
### Requirement: Overpass API rate limit
|
||||
**Reason**: The `/api/overpass` proxy and its third-party upstream are removed; POIs are served from the instance's own index.
|
||||
**Migration**: The equivalent limit continues on the replacement endpoint — see "POI API rate limit" below. No client changes; the same error UI handles 429s.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: POI API rate limit
|
||||
The Planner SHALL limit `/api/pois` requests to 120 per IP per minute to protect the instance's database from abusive clients.
|
||||
|
||||
#### Scenario: POI rate limit exceeded
|
||||
- **WHEN** a single IP exceeds 120 POI requests in one minute
|
||||
- **THEN** `/api/pois` responds with 429 Too Many Requests
|
||||
- **AND** no database query is executed for rejected requests
|
||||
51
openspec/changes/archive/2026-07-13-poi-index/tasks.md
Normal file
51
openspec/changes/archive/2026-07-13-poi-index/tasks.md
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
## 1. Category selectors (single source of truth)
|
||||
|
||||
- [x] 1.1 Refactor `packages/map-core/src/poi.ts`: replace Overpass QL `query` strings with structured `selectors: Array<{key, value}>` per category; update `poi.test.ts`
|
||||
- [x] 1.2 Add a helper that renders the selectors as an osmium tag-filter expression list (used by the pipeline; unit-tested against the nine categories)
|
||||
|
||||
## 2. Schema
|
||||
|
||||
- [x] 2.1 Add `planner.pois` to `packages/db` (osm_type, osm_id, category, name, geom Point 4326, tags jsonb, imported_at; PK (osm_type, osm_id, category); GiST on geom, btree on category) + migration
|
||||
- [x] 2.2 `pnpm db:push` locally and verify bbox+category query plans use the indexes _(verified: serving query drives off `pois_geom_idx` for the bbox; `pois_category_idx` used for category filter; PK created as `pois_pkey`)_
|
||||
|
||||
## 3. Extract pipeline (BRouter host)
|
||||
|
||||
- [x] 3.1 Create `infrastructure/brouter-host/poi-extract/`: script that downloads the configured PBF (`POI_PBF_URL`, planet by default), runs `osmium tags-filter` + `osmium export` (centroids) into gzipped NDJSON, writes a manifest (sha256 + timestamp + row count), and cleans up working files _(NDJSON is `{osm_type, osm_id, name, lat, lon, tags}`; the importer derives `categories` from tags via map-core, so selector logic isn't duplicated on the Node-free host)_
|
||||
- [x] 3.2 Publish artifact + manifest via the existing Caddy sidecar at a vSwitch-only address (`/poi/*` on `:17777`, same `X-BRouter-Auth` gate) _(public-unreachability follows from the existing vSwitch-only bind + UFW; verify on the host)_
|
||||
- [x] 3.3 Add a monthly systemd timer under the `trails` user; document disk/bandwidth footprint in `infrastructure/brouter-host/poi-extract/README.md`
|
||||
- [x] 3.4 Dry-run on a small Geofabrik extract first; then a full planet run — record row counts per category and artifact size _(Malta smoke 2947 rows; Germany 733,986 → 41 MB; **planet 8,623,453 rows → 468 MB gz**, filtered PBF 619 MB, source planet.osm.org)_
|
||||
|
||||
## 4. Import job (flagship)
|
||||
|
||||
- [x] 4.1 Create the import script: fetch manifest + artifact over the vSwitch, verify checksum, COPY into `planner.pois_staging`, build indexes, atomic rename swap in one transaction
|
||||
- [x] 4.2 Implement the 70% row-count guard with a `--bootstrap` override; abort loudly (log + metric) when tripped
|
||||
- [x] 4.3 Schedule via systemd timer (`poi-import.timer`, offset a day after the extract); wire into `cd-infra.yml` deploy sources (`infrastructure/scripts` added to the SCP list)
|
||||
- [x] 4.4 First production import with `--bootstrap` (germany), then planet re-import (guard germany→planet passed): live `planner.pois` = **8,434,518 rows** across all nine categories; spot-checked Berlin/Fehmarn/Tokyo/NYC viewports return expected POIs. _(Two import-script bugs found + fixed in prod: `\i` selectors path inside the postgres container → stream via stdin (#563); multi-selector same-category PK collision → `DISTINCT ON` (#564).)_
|
||||
|
||||
## 5. Serving route
|
||||
|
||||
- [x] 5.1 Create `apps/planner/app/routes/api.pois.ts` (GET bbox + categories): same-origin + session checks, 120/IP/min rate limit (no DB query on rejection), 100-result cap, short Cache-Control; register in `apps/planner/app/routes.ts`
|
||||
- [x] 5.2 Unit tests: happy path, invalid bbox/categories 400, 429 path, empty-table graceful response
|
||||
|
||||
## 6. Client switch
|
||||
|
||||
- [x] 6.1 Rewrite `apps/planner/app/lib/overpass.ts` as the `/api/pois` client (keep `Poi` shape, `quantizeBbox`, error mapping; drop QL building); rename file to `pois.ts` and update imports (`use-pois`, `use-nearby-pois`, snap-to-poi)
|
||||
- [x] 6.2 Update client tests; verify the existing rate-limit banner and unavailable message fire on 429/5xx from `/api/pois`
|
||||
- [x] 6.3 Delete `apps/planner/app/routes/api.overpass.ts` and its tests; remove the route registration and the private.coffee configuration (`OVERPASS_URLS` kept for the Journal surface backfill; removed from the planner service in both compose files)
|
||||
|
||||
## 7. Observability
|
||||
|
||||
- [x] 7.1 Add `poi_index_rows{category}`, `poi_index_age_seconds`, import outcome, and `poi_api_requests_total{status}` metrics; remove `overpass_upstream_*` metrics from `metrics.server.ts` (planner exposes rows/age via scrape-time DB collect + `poi_api_requests_total`; import outcome is emitted by the import job via node_exporter textfile — see Task 4)
|
||||
- [x] 7.2 Update the Grafana planner dashboard: replace Overpass upstream panels with index freshness + serving panels; add the stale-index alert (~6 weeks) (`poi-index-stale` in alerts.yml)
|
||||
|
||||
## 8. Docs & cleanup
|
||||
|
||||
- [x] 8.1 Update `docs/architecture.md` (POI data flow) and the privacy documentation (Overpass now Journal-surface-backfill-only; Planner POIs served same-origin)
|
||||
- [x] 8.2 Document the self-hoster story: optional pipeline, regional extract URL, graceful behavior with an empty index (poi-extract README)
|
||||
- [x] 8.3 Update `docs/ideas/self-host-overpass/README.md`: superseded by `poi-index`, with the revive-criteria note kept for the QL-compatibility case (+ roadmap pointer)
|
||||
|
||||
## 9. Verification
|
||||
|
||||
- [x] 9.1 E2E: POI overlay flow (enable category → markers appear) — `e2e/planner-overlays.test.ts` + nearby-snap in `planner-routing.test.ts` now mock `/api/pois` (network-mock is the house pattern; a DB-seeded variant is possible but the route's SQL path is covered by `api.pois.test.ts`)
|
||||
- [x] 9.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` _(typecheck + lint + unit green; full planner e2e project green incl. all three POI tests. Journal e2e specs need `E2E=true` + the `trails_e2e` scratch DB — unrelated to this planner-only change; CI runs them with that harness.)_
|
||||
- [x] 9.3 Post-cutover production check: verified via the cmux browser on a live session (Brandenburg) — POI markers + clusters render incl. camping tents; `/api/pois` returns data for busy (Berlin/Tokyo/NYC) and rural (Fehmarn) viewports; all nine categories populated. _(Remaining manual op step: enable the two monthly systemd timers — `poi-extract.timer` on the BRouter host, `poi-import.timer` on the flagship — a persistence change to apply by hand; see the poi-extract + scripts READMEs.)_
|
||||
Loading…
Add table
Add a link
Reference in a new issue