trails-cool/trails
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
trails/infrastructure/scripts
History
Ullrich Schäfer e703843b9b
poi-index: extract pipeline (BRouter host) + import job (flagship) + e2e 
- poi-extract.sh: PBF download -> osmium tags-filter -> centroid NDJSON +
  manifest, published via the Caddy sidecar at vSwitch-only /poi/*
- osmium-filters.txt + poi-selectors.sql generated from map-core selectors,
  guarded by sync tests so poiCategories stays the single source of truth
- poi-import.sh: checksum verify -> classify into category rows ->
  70% guard (--bootstrap override) -> atomic rename swap; node_exporter
  textfile metric for import outcome
- systemd service+timer units for both halves (monthly, offset)
- cd-infra.yml: ship infrastructure/scripts to the flagship
- e2e: mock /api/pois instead of /api/overpass

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 22:48:32 +02:00
..
backup-postgres.sh Add infrastructure: Terraform, Docker Compose, BRouter, CD pipeline (#2) 2026-03-22 13:32:01 +01:00
gen-poi-selectors-sql.ts poi-index: extract pipeline (BRouter host) + import job (flagship) + e2e 2026-07-12 22:48:32 +02:00
poi-import.service poi-index: extract pipeline (BRouter host) + import job (flagship) + e2e 2026-07-12 22:48:32 +02:00
poi-import.sh poi-index: extract pipeline (BRouter host) + import job (flagship) + e2e 2026-07-12 22:48:32 +02:00
poi-import.timer poi-index: extract pipeline (BRouter host) + import job (flagship) + e2e 2026-07-12 22:48:32 +02:00
poi-selectors.sql poi-index: extract pipeline (BRouter host) + import job (flagship) + e2e 2026-07-12 22:48:32 +02:00
README.md poi-index: extract pipeline (BRouter host) + import job (flagship) + e2e 2026-07-12 22:48:32 +02:00

README.md

Flagship operational scripts

Deployed to /opt/trails-cool/scripts/ by cd-infra.yml.

backup-postgres.sh

Daily pg_dump + retention sweep. Run via cron (see the header).

POI index import (poi-import.sh)

The "load where the database is" half of the poi-index change. Pulls the artifact the BRouter host published, classifies each element into planner.pois category rows, and swaps it in atomically.

  • poi-import.sh — fetch (vSwitch) → checksum → load → classify → guarded atomic rename swap. --bootstrap skips the 70% row-count guard for the first import (when the live table is legitimately empty).
  • poi-selectors.sql — generated from @trails-cool/map-core selectors; the classifier JOINs on it. Regenerate with npx tsx infrastructure/scripts/gen-poi-selectors-sql.ts (CI sync test: packages/map-core/src/poi-selectors-sql.sync.test.ts).
  • poi-import.service / .timer — monthly, offset a day after the extract.

First-time setup on the flagship

# 1. Create the empty live table from the Drizzle schema (idempotent).
#    (Run from a checkout with DATABASE_URL pointed at production, or via the
#    normal db:push deploy path.)
pnpm db:push

# 2. Install the timer.
cp /opt/trails-cool/scripts/poi-import.{service,timer} /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now poi-import.timer

# 3. Bootstrap the first import (guard disabled — live table is empty).
BROUTER_AUTH_TOKEN=$(grep BROUTER_AUTH_TOKEN /opt/trails-cool/.env | cut -d= -f2-) \
  /opt/trails-cool/scripts/poi-import.sh --bootstrap

Prometheus picks up import outcome from node_exporter's textfile collector (poi_import_last_status, …_last_success_timestamp_seconds, …_last_rows) when NODE_EXPORTER_TEXTFILE_DIR is set (the unit sets it). Index size and age are additionally exposed by the planner at /metrics (poi_index_rows{category}, poi_index_age_seconds).