poi-index: run osmium via Docker (host has no osmium + no sudo)
The BRouter host's trails user is non-root without sudo, so osmium-tool can't be apt-installed. Ship osmium.Dockerfile (built on first use) and run osmium in a container with the work dir bind-mounted (native I/O, negligible overhead on Linux). python3/curl/gzip/sha256sum remain on the host. Correct the README disk figure to the /home free space (~595 GB). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer
parent
157a4fad64
commit
1b5f48f118
3 changed files with 66 additions and 21 deletions
|
|
@ -48,8 +48,12 @@ artifact is unreachable from the public internet.
|
|||
|
||||
## Prerequisites
|
||||
|
||||
Install on the host: `osmium-tool`, `python3`, `curl`, `gzip`, `coreutils`
|
||||
(`sha256sum`). On Debian/Ubuntu: `apt install osmium-tool python3`.
|
||||
- **Docker** — osmium runs in a container (`osmium.Dockerfile`, built on first
|
||||
use as `trails-osmium:local`). The BRouter host's `trails` user is non-root
|
||||
with no sudo, so osmium can't be apt-installed; docker-group rights cover it.
|
||||
On Linux this adds no meaningful overhead — big files are bind-mounted, so I/O
|
||||
is native.
|
||||
- **On the host** (already present): `python3`, `curl`, `gzip`, `sha256sum`.
|
||||
|
||||
## Install the timer
|
||||
|
||||
|
|
@ -75,7 +79,8 @@ poi-extract.service` then `journalctl --user -u poi-extract.service -f`.
|
|||
(~4 GB) for a smaller instance.
|
||||
- **Disk**: the planet PBF + filtered PBF live transiently in `./work` and are
|
||||
removed on exit (even on failure) via a trap. Peak transient usage ≈ size of
|
||||
the PBF. The published artifact is a few hundred MB gzipped.
|
||||
the PBF. The `trails` user's `/home` has ~595 GB free (as of 2026-07) — ample
|
||||
for an ~80 GB planet run. The published artifact is a few hundred MB gzipped.
|
||||
- **CPU/time**: a full planet filter is IO-bound and can take a few hours; the
|
||||
unit runs at `Nice=10` / idle IO priority so it doesn't disturb BRouter.
|
||||
|
||||
|
|
|
|||
15
infrastructure/brouter-host/poi-extract/osmium.Dockerfile
Normal file
15
infrastructure/brouter-host/poi-extract/osmium.Dockerfile
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
# Minimal osmium-tool image for the POI extract pipeline. The BRouter host runs
|
||||
# a non-root `trails` user with no sudo, so osmium can't be apt-installed on the
|
||||
# host — poi-extract.sh runs it in this container instead (docker-group rights
|
||||
# are available). Big files are bind-mounted, so I/O is native; the container
|
||||
# adds no meaningful CPU/throughput overhead on Linux.
|
||||
#
|
||||
# Built on first use by poi-extract.sh (tag: trails-osmium:local). Self-hosters
|
||||
# get identical behavior without touching system packages.
|
||||
FROM debian:bookworm-slim
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends osmium-tool \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
ENTRYPOINT ["osmium"]
|
||||
|
|
@ -1,16 +1,19 @@
|
|||
#!/bin/bash
|
||||
# POI extract pipeline — runs on the dedicated BRouter host under the `trails`
|
||||
# user (the box with 1.8 TB of disk). Downloads an OSM PBF, filters it to the
|
||||
# planner's POI category selectors with osmium, reduces every element to a
|
||||
# centroid, and publishes a compact NDJSON artifact + manifest that the
|
||||
# flagship import job pulls over the vSwitch.
|
||||
# user. Downloads an OSM PBF, filters it to the planner's POI category selectors
|
||||
# with osmium, reduces every element to a centroid, and publishes a compact
|
||||
# NDJSON artifact + manifest that the flagship import job pulls over the vSwitch.
|
||||
#
|
||||
# Nothing here is planner/journal code — it's plain osmium + python so the host
|
||||
# needs no Node runtime. The tag filter comes from `osmium-filters.txt`, which
|
||||
# is generated from `@trails-cool/map-core`'s POI selectors (single source of
|
||||
# truth; kept in sync by osmium-filters.sync.test.ts).
|
||||
# Nothing here is planner/journal code — the host needs no Node runtime. The tag
|
||||
# filter comes from `osmium-filters.txt`, generated from `@trails-cool/map-core`
|
||||
# POI selectors (single source of truth; kept in sync by
|
||||
# osmium-filters.sync.test.ts).
|
||||
#
|
||||
# Prerequisites on the host: osmium-tool, python3, curl, gzip, sha256sum.
|
||||
# osmium runs in a container: the `trails` user is non-root with no sudo, so
|
||||
# osmium-tool can't be apt-installed on the host, but docker-group rights are
|
||||
# available. Big files are bind-mounted into the container, so I/O is native and
|
||||
# the container adds no meaningful CPU overhead on Linux. The image is built on
|
||||
# first use from osmium.Dockerfile. python3/curl/gzip/sha256sum run on the host.
|
||||
#
|
||||
# Usage:
|
||||
# ./poi-extract.sh
|
||||
|
|
@ -24,6 +27,8 @@
|
|||
# POI_PUBLISH_DIR Where the artifact + manifest are written for Caddy to
|
||||
# serve. Default: <script dir>/publish (mounted into the Caddy
|
||||
# sidecar at /srv/poi — see docker-compose.yml + Caddyfile).
|
||||
# OSMIUM_IMAGE osmium container image. Default: trails-osmium:local (built
|
||||
# on first use from osmium.Dockerfile).
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
|
@ -33,6 +38,7 @@ TRANSFORMER="$SCRIPT_DIR/to-ndjson.py"
|
|||
POI_PBF_URL="${POI_PBF_URL:-https://planet.osm.org/pbf/planet-latest.osm.pbf}"
|
||||
POI_WORK_DIR="${POI_WORK_DIR:-$SCRIPT_DIR/work}"
|
||||
POI_PUBLISH_DIR="${POI_PUBLISH_DIR:-$SCRIPT_DIR/publish}"
|
||||
OSMIUM_IMAGE="${OSMIUM_IMAGE:-trails-osmium:local}"
|
||||
|
||||
PLANET_PBF="$POI_WORK_DIR/planet.osm.pbf"
|
||||
FILTERED_PBF="$POI_WORK_DIR/filtered.osm.pbf"
|
||||
|
|
@ -41,6 +47,16 @@ MANIFEST="$POI_PUBLISH_DIR/manifest.json"
|
|||
|
||||
mkdir -p "$POI_WORK_DIR" "$POI_PUBLISH_DIR"
|
||||
|
||||
# Run osmium inside the container with the work dir bind-mounted at /data, as the
|
||||
# host uid/gid so output files are owned by `trails`, not root. Big-file I/O goes
|
||||
# through the bind mount (native), never the container's overlay layer.
|
||||
osmium_run() {
|
||||
docker run --rm \
|
||||
-u "$(id -u):$(id -g)" \
|
||||
-v "$POI_WORK_DIR:/data" \
|
||||
"$OSMIUM_IMAGE" "$@"
|
||||
}
|
||||
|
||||
# Working files can be huge (planet PBF ~80 GB); always clean them up, even on
|
||||
# failure, so a broken run can't fill the shared host's disk.
|
||||
cleanup() { rm -f "$PLANET_PBF" "$FILTERED_PBF"; }
|
||||
|
|
@ -56,26 +72,35 @@ fi
|
|||
mapfile -t FILTERS < "$FILTERS_FILE"
|
||||
log "tag filters: ${FILTERS[*]}"
|
||||
|
||||
# --- 2. Download the PBF -----------------------------------------------------
|
||||
# --- 2. Ensure the osmium image exists (build once from osmium.Dockerfile) ---
|
||||
if ! docker image inspect "$OSMIUM_IMAGE" >/dev/null 2>&1; then
|
||||
log "building osmium image $OSMIUM_IMAGE"
|
||||
docker build -t "$OSMIUM_IMAGE" -f "$SCRIPT_DIR/osmium.Dockerfile" "$SCRIPT_DIR"
|
||||
fi
|
||||
|
||||
# --- 3. Download the PBF -----------------------------------------------------
|
||||
log "downloading $POI_PBF_URL"
|
||||
curl --fail --location --show-error --silent --output "$PLANET_PBF" "$POI_PBF_URL"
|
||||
log "downloaded $(du -h "$PLANET_PBF" | cut -f1)"
|
||||
|
||||
# --- 3. Filter to our categories --------------------------------------------
|
||||
# --- 4. Filter to our categories --------------------------------------------
|
||||
# Paths are inside the container's /data bind mount. tags-filter keeps the nodes
|
||||
# referenced by matched ways/relations (default), so export can build geometry.
|
||||
log "filtering with osmium tags-filter"
|
||||
osmium tags-filter --overwrite --output "$FILTERED_PBF" "$PLANET_PBF" "${FILTERS[@]}"
|
||||
osmium_run tags-filter --overwrite --output /data/filtered.osm.pbf /data/planet.osm.pbf "${FILTERS[@]}"
|
||||
log "filtered to $(du -h "$FILTERED_PBF" | cut -f1)"
|
||||
|
||||
# --- 4. Export to centroid NDJSON -------------------------------------------
|
||||
# osmium export emits one GeoJSON feature per line; to-ndjson.py reduces each to
|
||||
# a bbox centre + compact record and reports the row count on stderr.
|
||||
# --- 5. Export to centroid NDJSON -------------------------------------------
|
||||
# osmium export emits one GeoJSON feature per line to stdout (forwarded out of
|
||||
# the container); to-ndjson.py reduces each to a bbox centre + compact record on
|
||||
# the host and reports the row count on stderr.
|
||||
log "exporting + transforming to NDJSON"
|
||||
rowcount_file="$(mktemp)"
|
||||
osmium export "$FILTERED_PBF" \
|
||||
osmium_run export /data/filtered.osm.pbf \
|
||||
--output-format=geojsonseq \
|
||||
--add-unique-id=type_id \
|
||||
--geometry-types=point,linestring,polygon \
|
||||
--overwrite --output - \
|
||||
--output - \
|
||||
| python3 "$TRANSFORMER" 2>"$rowcount_file" \
|
||||
| gzip -c > "$ARTIFACT"
|
||||
ROW_COUNT="$(cat "$rowcount_file")"
|
||||
|
|
@ -87,7 +112,7 @@ if [ "$ROW_COUNT" -eq 0 ]; then
|
|||
exit 1
|
||||
fi
|
||||
|
||||
# --- 5. Manifest (checksum + timestamp + row count) -------------------------
|
||||
# --- 6. Manifest (checksum + timestamp + row count) -------------------------
|
||||
SHA256="$(sha256sum "$ARTIFACT" | cut -d' ' -f1)"
|
||||
GENERATED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
cat > "$MANIFEST" <<EOF
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue