trails-cool/trails

Watch

Fork 0

POI index never auto-refreshes — neither refresh timer is installed #19

New issue

Closed

opened 2026-07-26 18:26:12 +00:00 by ullrich · 2 comments

ullrich commented

2026-07-26 18:26:12 +00:00

Owner

The poi-index change shipped 2026-07-13 with a documented remaining manual step: enable the two monthly refresh timers. That step was never done, so the POI index has not refreshed since the one-off manual planet import — OSM changes have not appeared in /api/pois for the ~2 weeks since.

Verified on both hosts today, and the units are not merely disabled — they are not installed at all:

BRouter host (ullrich.is, user systemd):

$ systemctl --user is-enabled poi-extract.timer
Failed to get unit file state for poi-extract.timer: No such file or directory
$ systemctl --user list-timers --all
0 timers listed.
$ loginctl show-user trails | grep -i linger
Linger=no

Flagship (trails.cool, system systemd):

$ systemctl is-enabled poi-import.timer
not-found

The unit files are deployed by cd-brouter — ~/brouter/poi-extract/poi-extract.{service,timer} both exist — they were simply never installed into a systemd unit path. ~/.config/systemd/user/ does not exist on the BRouter host.

Why it needs doing by hand

This is a persistence change on two machines, not something the deploy workflows cover:

BRouter host — user systemd, so it also needs loginctl enable-linger trails. Without linger the timer would not run when nobody is logged in, which is the normal state. Linger=no today.
Flagship — /etc/systemd/system/poi-import.timer, needs root.

Runbooks already exist: infrastructure/brouter-host/poi-extract/README.md and infrastructure/scripts/README.md.

Worth deciding while doing it

Monitoring. A silently-not-running refresh is exactly the class of failure that took down cd-apps sourcemaps (#5) and the SOPS restart gap (#13): no signal, just staleness. Consider a Grafana alert on POI index age, or emitting the extract's completion into Loki, so a failed monthly run is visible rather than discovered months later.
Timer vs workflow. The extract is HDD-bound and takes ~1.5h on an ~87GB planet PBF, which is why it is a host timer rather than CI. That reasoning still holds, but it does mean the schedule lives outside the repo and outside Forgejo's view — nothing in CI/CD will ever tell you it stopped.

Context: the BRouter host has no osmium and no sudo for the trails user, so the extract runs osmium in a container (osmium.Dockerfile, built on first use).

The `poi-index` change shipped 2026-07-13 with a documented remaining manual step: enable the two monthly refresh timers. That step was never done, so **the POI index has not refreshed since the one-off manual planet import** — OSM changes have not appeared in `/api/pois` for the ~2 weeks since. Verified on both hosts today, and the units are not merely *disabled* — they are **not installed at all**: **BRouter host (`ullrich.is`, user systemd):** ``` $ systemctl --user is-enabled poi-extract.timer Failed to get unit file state for poi-extract.timer: No such file or directory $ systemctl --user list-timers --all 0 timers listed. $ loginctl show-user trails | grep -i linger Linger=no ``` **Flagship (`trails.cool`, system systemd):** ``` $ systemctl is-enabled poi-import.timer not-found ``` The unit files *are* deployed by `cd-brouter` — `~/brouter/poi-extract/poi-extract.{service,timer}` both exist — they were simply never installed into a systemd unit path. `~/.config/systemd/user/` does not exist on the BRouter host. ## Why it needs doing by hand This is a persistence change on two machines, not something the deploy workflows cover: - **BRouter host** — user systemd, so it also needs `loginctl enable-linger trails`. Without linger the timer would not run when nobody is logged in, which is the normal state. `Linger=no` today. - **Flagship** — `/etc/systemd/system/poi-import.timer`, needs root. Runbooks already exist: `infrastructure/brouter-host/poi-extract/README.md` and `infrastructure/scripts/README.md`. ## Worth deciding while doing it - **Monitoring.** A silently-not-running refresh is exactly the class of failure that took down `cd-apps` sourcemaps (#5) and the SOPS restart gap (#13): no signal, just staleness. Consider a Grafana alert on POI index age, or emitting the extract's completion into Loki, so a failed monthly run is visible rather than discovered months later. - **Timer vs workflow.** The extract is HDD-bound and takes ~1.5h on an ~87GB planet PBF, which is why it is a host timer rather than CI. That reasoning still holds, but it does mean the schedule lives outside the repo and outside Forgejo's view — nothing in CI/CD will ever tell you it stopped. Context: the BRouter host has no osmium and no sudo for the `trails` user, so the extract runs osmium in a container (`osmium.Dockerfile`, built on first use).

ullrich commented

2026-07-26 18:40:18 +00:00

Author

Owner

Correction to my "worth deciding" note above: the monitoring already exists. I suggested considering a Grafana alert on POI index age without checking first — it was already built, and I have now verified it is live end to end.

uid: poi-index-stale        (infrastructure/grafana/provisioning/alerting/alerts.yml)
expr: max(poi_index_age_seconds) > 3628800   # 6 weeks, "one missed monthly refresh"
for:  1h

The metric poi_index_age_seconds is published by the Planner from its own DB (extract(epoch from now() - max(pois.imported_at))).
The rule is present in the running Grafana on the flagship, and Prometheus is scraping the metric.

Current value: 1,191,795 s ≈ 13.8 days. Threshold is 42 days, so the alert has not fired — correctly. Given the timers are not installed, the index will keep aging and this alert will trip on its own at roughly 2026-08-23 unless the timers are enabled before then.

So the safety net is real, just slow by design. That changes the urgency of this issue rather than the substance: nothing is silently unmonitored, but the index is stale today and will stay stale until the timers exist.

Worth noting the alert also covers the failure mode where the import runs but refuses to swap — the guard leaves the old table in place, so age keeps climbing even though the job "ran". Detail is on the Planner dashboard (poi_import_last_status).

Correction to my "worth deciding" note above: **the monitoring already exists.** I suggested considering a Grafana alert on POI index age without checking first — it was already built, and I have now verified it is live end to end. ``` uid: poi-index-stale (infrastructure/grafana/provisioning/alerting/alerts.yml) expr: max(poi_index_age_seconds) > 3628800 # 6 weeks, "one missed monthly refresh" for: 1h ``` - The metric `poi_index_age_seconds` is published by the Planner from its own DB (`extract(epoch from now() - max(pois.imported_at))`). - The rule is present in the running Grafana on the flagship, and Prometheus is scraping the metric. **Current value: 1,191,795 s ≈ 13.8 days.** Threshold is 42 days, so the alert has not fired — correctly. Given the timers are not installed, the index will keep aging and this alert will trip on its own at roughly **2026-08-23** unless the timers are enabled before then. So the safety net is real, just slow by design. That changes the urgency of this issue rather than the substance: nothing is silently unmonitored, but the index is stale today and will stay stale until the timers exist. Worth noting the alert also covers the failure mode where the import runs but refuses to swap — the guard leaves the old table in place, so age keeps climbing even though the job "ran". Detail is on the Planner dashboard (`poi_import_last_status`).

ullrich referenced this issue from a commit

2026-07-27 12:00:01 +00:00

fix(poi): stop a successful import reporting itself as failed; wire the textfile collector

ullrich referenced this issue

2026-07-27 12:00:28 +00:00

fix(poi): stop a successful import reporting itself as failed; wire the textfile collector #22

ullrich commented

2026-07-27 12:18:57 +00:00

Author

Owner

Done — both timers are installed, enabled, and their prerequisites verified.

BRouter host (trails, user systemd + linger):

NEXT                        LEFT        LAST PASSED UNIT              ACTIVATES
Sat 2026-08-01 02:44:31 UTC 4 days left n/a  n/a    poi-extract.timer poi-extract.service

Flagship (root, system systemd):

NEXT                        LEFT   LAST PASSED UNIT             ACTIVATES
Sun 2026-08-02 04:11:40 UTC 5 days -    -      poi-import.timer poi-import.service

The one-day offset the units were designed around is intact, and LAST is unset on both — Persistent=true did not fire a catch-up run on enable, so nothing started early.

Prerequisites checked rather than assumed

A monthly timer that fails at 02:00 would not surface for weeks, so the extract unit was checked end-to-end:

%h resolved correctly → ExecStart=/home/trails/brouter/poi-extract/poi-extract.sh
poi-extract.sh and to-ndjson.py both -rwxr-xr-x
docker ps works as trails; trails-osmium:local already built, so the first run will not stop to build it
python3, curl, gzip, sha256sum present; 587G free on /home; timeout 6h

Note for anyone else doing this: su trails does not register a logind session, so systemctl --user fails with Failed to connect to bus. export XDG_RUNTIME_DIR=/run/user/$(id -u) fixes it, or SSH in as trails directly.

The monitoring question this issue raised

The issue asked whether a silently-not-running refresh would be visible. Two fixes went in, and the answer is now yes — verified rather than assumed:

poi-index-stale (Grafana, 6-week threshold on poi_index_age_seconds) already existed and is live. Index age currently reads ~14.5 days, so if the Aug 1–2 cycle fails, it alerts around Aug 23.

Per-run outcome detail was broken in two ways, both fixed in #22:

emit_metric is the last command a successful import runs, and its mv targeted a directory that did not exist on the flagship. Under set -euo pipefail that failing mv became the exit status — a fully successful import would have been recorded by systemd as a failed unit. Verified against the pre-fix script: exit 1, with all the work already done.
node-exporter ran without --collector.textfile.directory and without that path mounted, so the metrics were written and silently dropped. It also runs as nobody (uid 65534) while mktemp creates 0600 root-owned files, so the file would have been unreadable even once the collector was wired.

The chain is now confirmed working: a synthetic .prom written on the host was exposed by node_exporter (node_textfile_scrape_error 0) and scraped into Prometheus, then removed.

Follow-up filed

Verifying that deploy surfaced #23 — cd-infra killed Prometheus one second after starting it, taking monitoring down for ~16 minutes. Unrelated to POI, but found here and tracked there.

The remaining unknown is simply whether the first real cycle produces fresh data on Aug 1–2. The stale-index alert covers it if not, which is the outcome this issue was asking for.

Done — both timers are installed, enabled, and their prerequisites verified. **BRouter host** (`trails`, user systemd + linger): ``` NEXT LEFT LAST PASSED UNIT ACTIVATES Sat 2026-08-01 02:44:31 UTC 4 days left n/a n/a poi-extract.timer poi-extract.service ``` **Flagship** (root, system systemd): ``` NEXT LEFT LAST PASSED UNIT ACTIVATES Sun 2026-08-02 04:11:40 UTC 5 days - - poi-import.timer poi-import.service ``` The one-day offset the units were designed around is intact, and `LAST` is unset on both — `Persistent=true` did not fire a catch-up run on enable, so nothing started early. ## Prerequisites checked rather than assumed A monthly timer that fails at 02:00 would not surface for weeks, so the extract unit was checked end-to-end: - `%h` resolved correctly → `ExecStart=/home/trails/brouter/poi-extract/poi-extract.sh` - `poi-extract.sh` and `to-ndjson.py` both `-rwxr-xr-x` - `docker ps` works as `trails`; `trails-osmium:local` already built, so the first run will not stop to build it - `python3`, `curl`, `gzip`, `sha256sum` present; 587G free on `/home`; timeout 6h Note for anyone else doing this: `su trails` does not register a logind session, so `systemctl --user` fails with `Failed to connect to bus`. `export XDG_RUNTIME_DIR=/run/user/$(id -u)` fixes it, or SSH in as `trails` directly. ## The monitoring question this issue raised The issue asked whether a silently-not-running refresh would be visible. Two fixes went in, and the answer is now yes — verified rather than assumed: `poi-index-stale` (Grafana, 6-week threshold on `poi_index_age_seconds`) already existed and is live. Index age currently reads ~14.5 days, so if the Aug 1–2 cycle fails, it alerts around Aug 23. Per-run outcome detail was broken in two ways, both fixed in #22: 1. `emit_metric` is the last command a successful import runs, and its `mv` targeted a directory that did not exist on the flagship. Under `set -euo pipefail` that failing `mv` became the exit status — **a fully successful import would have been recorded by systemd as a failed unit.** Verified against the pre-fix script: exit 1, with all the work already done. 2. `node-exporter` ran without `--collector.textfile.directory` and without that path mounted, so the metrics were written and silently dropped. It also runs as `nobody` (uid 65534) while `mktemp` creates `0600` root-owned files, so the file would have been unreadable even once the collector was wired. The chain is now confirmed working: a synthetic `.prom` written on the host was exposed by node_exporter (`node_textfile_scrape_error 0`) and scraped into Prometheus, then removed. ## Follow-up filed Verifying that deploy surfaced #23 — `cd-infra` killed Prometheus one second after starting it, taking monitoring down for ~16 minutes. Unrelated to POI, but found here and tracked there. The remaining unknown is simply whether the first real cycle produces fresh data on Aug 1–2. The stale-index alert covers it if not, which is the outcome this issue was asking for.

ullrich closed this issue

2026-07-27 12:18:57 +00:00

No labels

dependencies

major

security-pin

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

trails-cool/trails#19

No description provided.

Rows
Columns