trails/infrastructure/brouter-host/poi-extract

POI extract pipeline (BRouter host)

Monthly job that turns an OSM PBF into the compact POI artifact the flagship's planner.pois index is built from. This is the "filter where the disk is" half of the poi-index change — it replaces the third-party Overpass dependency for the Planner's POI overlays.

Runs on the dedicated BRouter host (ullrich.is, the box with 1.8 TB free) under the non-root trails user. The flagship import job (infrastructure/scripts/poi-import.sh) pulls the result over the vSwitch.

Files

File	Role
poi-extract.sh	Download PBF → osmium tags-filter → osmium export → to-ndjson.py → gzip + manifest
to-ndjson.py	GeoJSONSeq → NDJSON, reducing ways/relations to their bbox centre (Overpass out center equivalent)
osmium-filters.txt	osmium tag filters, generated from @trails-cool/map-core POI selectors
gen-osmium-filters.ts	Regenerates osmium-filters.txt (run after changing categories)
poi-extract.service / .timer	systemd units (monthly)

Single source of truth

osmium-filters.txt is derived from poiCategories in packages/map-core/src/poi.ts. After changing categories, regenerate it:

npx tsx infrastructure/brouter-host/poi-extract/gen-osmium-filters.ts

packages/map-core/src/osmium-filters.sync.test.ts fails CI if the committed file drifts from the selectors, so the host never needs a Node runtime.

Artifact contract

poi-extract.sh writes to the publish dir (default ./publish, mounted into the Caddy sidecar at /srv/poi):

• pois.ndjson.gz — one POI per line: {osm_type, osm_id, name, lat, lon, tags}. Category is not stored here; the flagship importer classifies each element from its tags using the same map-core selectors (so the selector logic is never duplicated on this host).
• manifest.json — {artifact, sha256, row_count, generated_at, source}.

Caddy serves both under /poi/* on the existing vSwitch-only :17777 listener, gated by the same X-BRouter-Auth shared secret as BRouter — so the artifact is unreachable from the public internet.

Prerequisites

Install on the host: osmium-tool, python3, curl, gzip, coreutils (sha256sum). On Debian/Ubuntu: apt install osmium-tool python3.

Install the timer

Deployed under ~trails/brouter/poi-extract/. As a user unit with lingering:

loginctl enable-linger trails
mkdir -p ~/.config/systemd/user
cp poi-extract.service poi-extract.timer ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now poi-extract.timer
systemctl --user list-timers poi-extract.timer

Trigger a run by hand (e.g. the first bootstrap): systemctl --user start poi-extract.service then journalctl --user -u poi-extract.service -f.

Footprint

• Bandwidth: the planet PBF is ~80 GB downloaded per run (monthly). Point POI_PBF_URL at a Geofabrik mirror or a regional/continental extract to cut this — e.g. europe-latest.osm.pbf (~30 GB) or germany-latest.osm.pbf (~4 GB) for a smaller instance.
• Disk: the planet PBF + filtered PBF live transiently in ./work and are removed on exit (even on failure) via a trap. Peak transient usage ≈ size of the PBF. The published artifact is a few hundred MB gzipped.
• CPU/time: a full planet filter is IO-bound and can take a few hours; the unit runs at Nice=10 / idle IO priority so it doesn't disturb BRouter.

Dry run first

Before the first planet run, validate the whole chain on a small extract:

POI_PBF_URL=https://download.geofabrik.de/europe/germany-latest.osm.pbf \
  ./poi-extract.sh
cat publish/manifest.json          # row_count sane? sha256 present?
zcat publish/pois.ndjson.gz | head # records well-formed?
zcat publish/pois.ndjson.gz | wc -l