trails-cool/trails
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
trails/infrastructure/scripts
History
Ullrich Schäfer 55796e7d15
poi-index: dedupe classify INSERT (element matching a category via 2 selectors) 
The import hit a PK violation on (osm_type, osm_id, category): an OSM element
that matches one category through two selectors on different keys (e.g. shelter
= amenity=shelter OR tourism=wilderness_hut, both present) produced two rows for
that category. Add DISTINCT ON (osm_type, osm_id, category) so classification is
one row per (element, category), matching matchingCategoryIds semantics.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 00:33:17 +02:00
..
backup-postgres.sh Add infrastructure: Terraform, Docker Compose, BRouter, CD pipeline (#2) 2026-03-22 13:32:01 +01:00
gen-poi-selectors-sql.ts poi-index: extract pipeline (BRouter host) + import job (flagship) + e2e 2026-07-12 22:48:32 +02:00
poi-import.service poi-index: extract pipeline (BRouter host) + import job (flagship) + e2e 2026-07-12 22:48:32 +02:00
poi-import.sh poi-index: dedupe classify INSERT (element matching a category via 2 selectors) 2026-07-13 00:33:17 +02:00
poi-import.timer poi-index: extract pipeline (BRouter host) + import job (flagship) + e2e 2026-07-12 22:48:32 +02:00
poi-selectors.sql poi-index: extract pipeline (BRouter host) + import job (flagship) + e2e 2026-07-12 22:48:32 +02:00
README.md poi-index: extract pipeline (BRouter host) + import job (flagship) + e2e 2026-07-12 22:48:32 +02:00

README.md

Flagship operational scripts

Deployed to /opt/trails-cool/scripts/ by cd-infra.yml.

backup-postgres.sh

Daily pg_dump + retention sweep. Run via cron (see the header).

POI index import (poi-import.sh)

The "load where the database is" half of the poi-index change. Pulls the artifact the BRouter host published, classifies each element into planner.pois category rows, and swaps it in atomically.

  • poi-import.sh — fetch (vSwitch) → checksum → load → classify → guarded atomic rename swap. --bootstrap skips the 70% row-count guard for the first import (when the live table is legitimately empty).
  • poi-selectors.sql — generated from @trails-cool/map-core selectors; the classifier JOINs on it. Regenerate with npx tsx infrastructure/scripts/gen-poi-selectors-sql.ts (CI sync test: packages/map-core/src/poi-selectors-sql.sync.test.ts).
  • poi-import.service / .timer — monthly, offset a day after the extract.

First-time setup on the flagship

# 1. Create the empty live table from the Drizzle schema (idempotent).
#    (Run from a checkout with DATABASE_URL pointed at production, or via the
#    normal db:push deploy path.)
pnpm db:push

# 2. Install the timer.
cp /opt/trails-cool/scripts/poi-import.{service,timer} /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now poi-import.timer

# 3. Bootstrap the first import (guard disabled — live table is empty).
BROUTER_AUTH_TOKEN=$(grep BROUTER_AUTH_TOKEN /opt/trails-cool/.env | cut -d= -f2-) \
  /opt/trails-cool/scripts/poi-import.sh --bootstrap

Prometheus picks up import outcome from node_exporter's textfile collector (poi_import_last_status, …_last_success_timestamp_seconds, …_last_rows) when NODE_EXPORTER_TEXTFILE_DIR is set (the unit sets it). Index size and age are additionally exposed by the planner at /metrics (poi_index_rows{category}, poi_index_age_seconds).