trails-cool/trails
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

openspec: revise route-surface-breakdown — add async Overpass backfill + SSE

Browse source 
Builds on the merged A-only proposal. Broadens the capability:
- breakdown now on routes AND activities;
- Path 1 (sync) unchanged: BRouter waytags → breakdown at Planner save;
- Path 2 (async): a `surface-backfill` pg-boss job map-matches geometry to OSM
  via Overpass for imports/uploads/older rows, stores the breakdown, and pushes
  a `surface_breakdown` SSE event so an open detail page fills in live;
- privacy note: backfill sends geometry to Overpass → route via the proxy /
  self-hostable Overpass; documented in the manifest.

Validates with `openspec validate --strict`. Specs only; no code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-06-14 17:54:50 +02:00
parent a81134332b
commit e1e3dc5c81
No known key found for this signature in database
GPG key ID: A32FF691A0F752D9
4 changed files with 167 additions and 105 deletions
Download patch file Download diff file Expand all files Collapse all files

123
openspec/changes/route-surface-breakdown/design.md
View file

@ -1,73 +1,90 @@
## Context ## Context
The Planner computes accurate per-coordinate surface/highway from BRouter's The Planner computes accurate per-coordinate surface/highway from BRouter's
`messages`/`WayTags` (`apps/planner/.../route-merge.ts` → `EnrichedRoute`), and `WayTags` (`apps/planner/.../route-merge.ts` → `EnrichedRoute`) and colours the
`ColoredRoute` already renders the route coloured by surface. But this metadata line with it, but it's session-only: `api.save-to-journal` POSTs only `{ gpx }`,
is **session-only**: the planner→journal handoff (`api.save-to-journal`) POSTs `routes` has no surface column, GPX carries no surface tags. The repo already
only `{ gpx }` to the Journal callback (`api.routes.$id.callback`), the `routes` has a typed pg-boss job system (`apps/journal/app/jobs/payloads.ts`,
table has no surface column, and GPX carries no surface extensions. So today no `@trails-cool/jobs`), an SSE broker (`/api/events` + EventSource hooks, e.g.
saved route has surface data. `useUnreadNotifications`), and an Overpass proxy used for Planner POIs
(`apps/planner/.../overpass.ts`, `/api/overpass`).
## Goals / Non-Goals ## Goals / Non-Goals
**Goals** **Goals**
- Surface + waytype proportion bars on route detail, using data we already - Surface + waytype bars on route **and** activity detail.
compute, with no new external dependency. - Two derivation paths: free/accurate for Planner routes; async backfill for
everything else, with a live SSE update.
**Non-Goals (v1)** **Non-Goals (v1)**
- **Activities / imported / manually-uploaded GPX** — they have no waytags - Colouring the Journal **map line** by surface (needs per-segment data, not the
(bare GPX). They show no breakdown for now (see "Data source" → future summary; a clean follow-up).
Overpass). - Re-backfilling on every geometry edit (backfill is one-shot per row unless
- **Colouring the Journal map line by surface** — needs per-segment data, not re-triggered).
just the summary; a clean follow-up once the bars land.
- Selectable surface-vs-waytype toggle UI beyond showing both bars.
## Decisions ## Decisions
### D1: Data source — persist BRouter waytags at save (Planner routes only) ### D1: Two derivation paths into one stored summary
Two ways to get a breakdown onto the Journal detail page: Both paths produce the same compact, distance-weighted summary —
`{ surface: { asphalt: <m>, gravel: <m>, … }, highway: { … } }` (metres per
category) — stored as nullable `surface_breakdown` jsonb on `routes` and
`activities`. The bars need proportions, not geometry, so we store a summary,
not per-segment data (per-segment is a future map-colouring concern).
- **(A, chosen) BRouter-at-save** — the Planner already has 100%-accurate - **Path 1 — synchronous, Planner routes (accurate, free).** The Planner has
per-segment surface/highway; compute the distribution there at save and 100%-accurate per-segment tags; compute the summary at save and persist via an
persist it. Accurate, cheap, no external call. **Limitation:** only extra optional `surfaceBreakdown` field in the handoff. No external call.
Planner-created routes (activities/uploads have no waytags). - **Path 2 — async backfill, everything else (best-effort).** For a row with
- **(B, future) Overpass map-matching** — snap arbitrary route geometry to OSM geometry but no summary, a job map-matches to OSM and computes it (D2).
ways and read `surface`/`highway` server-side. Covers imports/uploads, but
it's expensive, approximate, rate-limited, and an external dependency. Parked
for when we want imports covered (the repo already proxies Overpass for POIs).
v1 ships (A). Routes without the data degrade to no bars. ### D2: The async backfill job
A typed `surface-backfill` pg-boss job keyed by `{ kind: "route" | "activity",
id }`:
1. Load the geometry; bail if absent or already has a summary.
2. Query Overpass for `way[highway]` within the route bbox (a journal-side
client mirroring the Planner's `buildQuery`/`parseResponse` + rate-limit
handling).
3. **Map-match**: for each route segment, find the nearest OSM way and read its
`surface`/`highway`; bucket the segment's length. Unmatched/untagged →
`unknown`.
4. Store the summary; emit the SSE event (D3).
- **Triggers:** enqueued after a GPX import (Komoot/Garmin) and after a manual
upload; plus a one-off sweep job to backfill pre-existing rows. Optionally
enqueued on-demand when a detail page without a summary is opened.
- **Resilience:** pg-boss retries cover Overpass rate-limits/outages; the job is
idempotent (skips if a summary already exists) and best-effort (failure leaves
the row with no bars, not an error).
### D2: Store a compact distance-weighted summary, not per-segment data ### D3: SSE live update
The bars need **proportions**, not geometry. At save, sum the haversine length On backfill completion the job publishes a `surface_breakdown` SSE event
of each coordinate segment into its surface bucket (and highway bucket), (payload: `{ kind, id }`) via the existing broker. The detail page subscribes
yielding `{ surface: { asphalt: <m>, gravel: <m>, … }, highway: { … } }` (EventSource hook) and, when the event matches the row it's showing, re-fetches
(metres per category). This is tiny (a handful of keys), stored as a nullable the breakdown and fills the bars — so a user who opened the page before the job
`surface_breakdown` jsonb on `routes`. Per-segment data (for future map line finished sees it appear without reloading. While a backfill is known to be
colouring) is deliberately *not* persisted in v1 — that's a bigger payload and in-flight, the page may show a small "computing…" affordance.
a separate feature.
### D3: Handoff carries it
`api.save-to-journal` adds `surfaceBreakdown` to the POST body alongside `gpx`;
the Journal callback validates (Zod, all values ≥ 0) and stores it. The field
is optional, so the handoff stays backward-compatible and non-Planner callers
(if any) are unaffected. Re-saving a route recomputes/overwrites it.
### D4: Rendering ### D4: Rendering
A `SurfaceBreakdown` component renders one horizontal stacked bar per dimension A `SurfaceBreakdown` component: one horizontal stacked bar per dimension
(surface, then waytype): segments sized by percentage, coloured via (surface, then waytype), segments sized by percentage and coloured via
`SURFACE_COLORS` / `HIGHWAY_COLORS` (`DEFAULT_*` for unknown), with a legend `SURFACE_COLORS`/`HIGHWAY_COLORS` (`DEFAULT_*` for unknown), with a legend
listing each category + km + percent (largest first). Unknown/unmapped tags (category · km · %, largest first). Unknown/unmapped tags collapse into "other".
collapse into an "other" segment. Hidden entirely when `surface_breakdown` is Hidden when the summary is null/empty. Labels via i18n; distances via `stats.ts`.
null or empty. Labels via i18n; distances via `stats.ts`.
### D5: Privacy (backfill sends geometry to Overpass)
Path 1 sends nothing externally. Path 2's Overpass query exposes a route's
bounding box to the Overpass endpoint — a real consideration for a privacy-first
app. Mitigations: route the backfill through the **proxied / self-hostable**
Overpass (the parked `self-host-overpass` idea removes the third-party entirely),
and only the bbox + geometry needed for matching is sent. We document the
backfill's external lookup in the privacy manifest; a self-hosted Overpass is the
clean long-term answer.
## Risks / Trade-offs ## Risks / Trade-offs
- **Coverage gap** — only Planner routes get bars in v1. Acceptable: routes are - **Map-matching is approximate** — nearest-way snapping mismatches at junctions
*designed* in the Planner, which is exactly where "how much gravel" matters. / parallel ways; acceptable for a proportional overview, and Path 1 (exact) is
Imports get bars later via (B). preferred whenever available.
- **Tag sparsity** — OSM `surface` is often missing; those segments bucket as - **Overpass cost/limits** — bounded by the async queue + retries; the sweep
`unknown`/"other". We show it honestly rather than guessing. should throttle. A self-hosted Overpass removes the ceiling.
- **Stale on edit** — editing a route's geometry elsewhere without re-routing - **Two code paths** — both converge on one stored summary + one renderer, so
could leave the breakdown stale; recomputing on every Planner save keeps it only the *derivation* differs.
fresh for the normal flow.

71
openspec/changes/route-surface-breakdown/proposal.md
View file

@ -1,46 +1,57 @@
## Why ## Why
"How much of this is gravel?" is a top-three question for anyone choosing a "How much of this is gravel?" is a top-three question for choosing a route, and
route — Komoot's surface/waytype breakdown bars are one of its most-used Komoot's surface/waytype bars are one of its most-used features.
features, and `@trails-cool/map-core` already ships the surface/highway colour `@trails-cool/map-core` already ships the surface/highway colour palettes, and
palettes. We also already *compute* accurate per-segment surface and highway the Planner already *computes* accurate per-segment surface/highway from BRouter
tags from BRouter while planning (`route-merge` → `EnrichedRoute.surfaces[]`), (`route-merge` → `EnrichedRoute.surfaces[]`) — then **discards it on save**.
then **throw them away on save**. This change captures that data and shows it. This change captures that data, shows it as proportion bars, and **backfills
everything else** (imports, uploads, older rows) asynchronously so the bars
aren't limited to Planner routes.
## What Changes ## What Changes
- On **route detail**, show surface and waytype **proportion bars** (e.g. - On **route and activity detail**, show surface and waytype **proportion bars**
"62% asphalt · 28% gravel · 10% path"), coloured with the `map-core` (e.g. "62% asphalt · 28% gravel · 10% path"), coloured with the `map-core`
`SURFACE_COLORS` / `HIGHWAY_COLORS` palettes, with per-segment km/percent. `SURFACE_COLORS` / `HIGHWAY_COLORS` palettes, with per-category km/percent.
- **Persist the breakdown for Planner-created routes**: compute a - **Path 1 — synchronous (Planner routes):** compute a distance-weighted
distance-weighted surface + waytype distribution from the BRouter waytags at surface + waytype distribution from the BRouter waytags at save time, send it
save time, send it in the planner→journal handoff, and store it on the route. in the planner→journal handoff, store it. Accurate, free, no external call.
- Degrade gracefully: routes without breakdown data (older routes, or activities - **Path 2 — async backfill (everything else):** for a route/activity that has
/ manually-imported GPX that never had waytags) simply don't show the bars. geometry but no breakdown (imports, manual uploads, pre-existing rows), a
background job map-matches the geometry to OSM ways via Overpass, computes the
breakdown, stores it, and **pushes an SSE event** so an open detail page fills
the bars in live.
- Degrade gracefully: no geometry / not-yet-backfilled → no bars (or a
"computing…" state while a backfill is in flight).
## Capabilities ## Capabilities
### New Capabilities ### New Capabilities
- `route-surface-breakdown`: the surface/waytype proportion bars on route - `route-surface-breakdown`: the surface/waytype proportion bars on route &
detail, how the distribution is derived from BRouter waytags, persisted at activity detail — derived synchronously from BRouter waytags (Planner) or
save, and rendered. asynchronously via an Overpass backfill job with an SSE live-update.
### Modified Capabilities ### Modified Capabilities
<!-- `planner-journal-handoff` gains one optional field in its save payload; <!-- `planner-journal-handoff` gains one optional payload field; `background-jobs`
noted in design, but the breakdown is a new capability, not a change to the and `sse-broker` gain a new job + event, but those are additive uses of
handoff's existing requirements. --> existing transports, owned by this new capability. -->
## Impact ## Impact
- **Schema**: a nullable `surface_breakdown` jsonb column on `journal.routes` - **Schema**: nullable `surface_breakdown` jsonb on `journal.routes` **and**
(additive — `pnpm db:push`, no data migration). `journal.activities` (additive — `pnpm db:push`).
- **Planner**: `api.save-to-journal` computes the distance-weighted breakdown - **Planner**: `api.save-to-journal` computes + sends the breakdown.
from the session `EnrichedRoute` (surfaces/highways + coordinates) and adds it - **Journal**:
to the POST body. - route callback stores the synchronous breakdown;
- **Journal**: the route callback (`api.routes.$id.callback`) accepts + stores - a typed `surface-backfill` pg-boss job (Overpass map-match → breakdown →
it; the route detail loader exposes it; a `SurfaceBreakdown` component renders store → SSE), enqueued after imports/uploads and available as a sweep;
the bars (reusing `map-core` palettes + `stats.ts` distance formatting). - a journal-side Overpass way/surface client (reusing the existing
- **No new external dependency** — BRouter already returns the tags; no Overpass `/api/overpass` proxy pattern);
call. Imported activities/uploads are out of scope for v1 (see design). - an SSE `surface_breakdown` event + a detail-page subscription;
- a `SurfaceBreakdown` component (reuses `map-core` palettes + `stats.ts`).
- **i18n**: `journal.surface.*` (en + de).
- **Privacy**: the backfill sends route geometry to Overpass — see design for
the proxy/self-host + visibility considerations.

36
openspec/changes/route-surface-breakdown/specs/route-surface-breakdown/spec.md
View file

@ -1,21 +1,43 @@
## ADDED Requirements ## ADDED Requirements
### Requirement: Surface breakdown derived from BRouter waytags ### Requirement: Surface breakdown derived from BRouter waytags
The system SHALL compute a distance-weighted surface and waytype distribution for a Planner-created route from the per-segment BRouter waytags at save time, and persist it with the route. The system SHALL compute a distance-weighted surface and waytype distribution for a Planner-created route from its per-segment BRouter waytags at save time, and persist it with the route.
#### Scenario: Breakdown captured on save #### Scenario: Breakdown captured on save
- **WHEN** a route is saved from the Planner with per-segment surface/highway waytags - **WHEN** a route is saved from the Planner with per-segment surface/highway waytags
- **THEN** the route stores a distance-weighted breakdown of metres per surface category and per waytype category - **THEN** the route stores a distance-weighted breakdown of metres per surface category and per waytype category
#### Scenario: No waytags available #### Scenario: No waytags available
- **WHEN** a route is saved without waytags (no surface data) - **WHEN** a route is saved without waytags
- **THEN** no breakdown is stored, and the route detail shows no breakdown bars - **THEN** no breakdown is stored synchronously (an async backfill may fill it later)
### Requirement: Async surface backfill via Overpass
The system SHALL asynchronously derive a surface breakdown for a route or activity that has geometry but no stored breakdown, by map-matching its geometry to OpenStreetMap ways via Overpass in a background job, and SHALL store the resulting distance-weighted breakdown.
#### Scenario: Imported activity is backfilled
- **WHEN** an activity is imported (e.g. from Komoot) with geometry but no waytags
- **THEN** a backfill job map-matches its geometry to OSM ways and stores a surface/waytype breakdown
#### Scenario: Backfill is best-effort and idempotent
- **WHEN** a backfill job runs for a row that already has a breakdown
- **THEN** it does no work; and **WHEN** Overpass is unavailable, the row is left without a breakdown rather than erroring
#### Scenario: Untagged segments
- **WHEN** map-matched ways have no surface tag
- **THEN** those segments' distance is bucketed as unknown
### Requirement: Live update when a backfill completes
The detail page SHALL update its breakdown without a reload when a backfill for the row being viewed completes, via a server-sent event.
#### Scenario: Bars appear after backfill
- **WHEN** a user is viewing a route/activity detail page whose backfill finishes
- **THEN** the page receives a `surface_breakdown` event for that row and renders the bars without a manual reload
### Requirement: Surface and waytype proportion bars ### Requirement: Surface and waytype proportion bars
The route detail page SHALL render the surface and waytype breakdown as proportion bars when the route has breakdown data, coloured with the shared surface/highway palettes, each segment labelled with its category, distance, and percentage. The route and activity detail pages SHALL render the surface and waytype breakdown as proportion bars when the row has breakdown data, coloured with the shared surface/highway palettes, each segment labelled with its category, distance, and percentage.
#### Scenario: Bars shown for a route with a breakdown #### Scenario: Bars shown for a row with a breakdown
- **WHEN** a user views a route that has a stored surface breakdown - **WHEN** a user views a route or activity that has a stored surface breakdown
- **THEN** a surface proportion bar and a waytype proportion bar are shown, each segment coloured by category with the largest category first - **THEN** a surface proportion bar and a waytype proportion bar are shown, each segment coloured by category with the largest category first
#### Scenario: Percentages reflect distance share #### Scenario: Percentages reflect distance share
@ -27,7 +49,7 @@ The route detail page SHALL render the surface and waytype breakdown as proporti
- **THEN** their distance is shown as an "other" segment rather than omitted - **THEN** their distance is shown as an "other" segment rather than omitted
#### Scenario: Hidden without data #### Scenario: Hidden without data
- **WHEN** a route has no stored breakdown - **WHEN** a row has no stored breakdown
- **THEN** no breakdown bars are rendered - **THEN** no breakdown bars are rendered
### Requirement: Localized breakdown labels ### Requirement: Localized breakdown labels

42
openspec/changes/route-surface-breakdown/tasks.md
View file

@ -1,26 +1,38 @@
## 1. Breakdown derivation (shared, pure) ## 1. Shared derivation + shape
- [ ] 1.1 Add a pure `surfaceBreakdown(coordinates, surfaces, highways)` helper (in `@trails-cool/map-core` or a planner lib): sum haversine segment lengths into per-surface and per-waytype buckets → `{ surface: Record<string, number>, highway: Record<string, number> }` (metres). Unit-test it. - [ ] 1.1 Pure `surfaceBreakdown(coordinates, surfaces, highways)` helper: haversine-weight each segment into per-surface + per-waytype buckets → `{ surface: Record<string, number>, highway: Record<string, number> }` (metres). Unit-test.
- [ ] 1.2 Define the shared wire shape (Zod) for the breakdown in `@trails-cool/api` (`SurfaceBreakdownSchema`, all values ≥ 0). - [ ] 1.2 `SurfaceBreakdownSchema` (Zod) in `@trails-cool/api` (record of category → metres ≥ 0, surface + highway).
## 2. Capture at save (Planner → Journal) ## 2. Schema
- [ ] 2.1 In `apps/planner/.../api.save-to-journal`, read the session `EnrichedRoute` (coordinates + surfaces + highways), compute the breakdown, and add `surfaceBreakdown` to the POST body. - [ ] 2.1 Nullable `surfaceBreakdown` jsonb on `journal.routes` **and** `journal.activities`; `pnpm db:push`.
- [ ] 2.2 Journal callback (`api.routes.$id.callback`) validates the optional `surfaceBreakdown` and persists it; re-save overwrites.
## 3. Schema ## 3. Path 1 — synchronous (Planner routes)
- [ ] 3.1 Add nullable `surfaceBreakdown` jsonb to `journal.routes`; `pnpm db:push`. - [ ] 3.1 `apps/planner/.../api.save-to-journal`: compute the breakdown from the session `EnrichedRoute` (coords + surfaces + highways) and add `surfaceBreakdown` to the POST body.
- [ ] 3.2 Journal route callback validates the optional field and persists it (overwrite on re-save).
## 4. Render ## 4. Render
- [ ] 4.1 Route detail loader exposes `surfaceBreakdown`. - [ ] 4.1 Route + activity detail loaders expose `surfaceBreakdown`.
- [ ] 4.2 `SurfaceBreakdown` component: stacked proportion bar per dimension (surface, waytype), segments coloured via `SURFACE_COLORS`/`HIGHWAY_COLORS` (DEFAULT for unknown), legend with category · km · % (largest first), unknown → "other"; hidden when empty. Mount on `routes.$id.tsx`. - [ ] 4.2 `SurfaceBreakdown` component: stacked bar per dimension (surface, waytype), `map-core` `SURFACE_COLORS`/`HIGHWAY_COLORS` (DEFAULT → "other"), legend category · km · % (largest first), hidden when empty. Mount on `routes.$id.tsx` and `activities.$id.tsx`.
- [ ] 4.3 i18n `journal.surface.*` (label + category names + "other") en + de. - [ ] 4.3 i18n `journal.surface.*` (label + category names + "other") en + de.
## 5. Tests & checks ## 5. Path 2 — async backfill + SSE
- [ ] 5.1 Unit: `surfaceBreakdown` (distance weighting, multiple surfaces, unknown bucket). - [ ] 5.1 Journal-side Overpass client: `way[highway]` in bbox + parse (mirror the planner's `buildQuery`/`parseResponse` + rate-limit handling; route via the `/api/overpass` proxy).
- [ ] 5.2 Component (jsdom): bars render with correct proportions; "other" for unknowns; hidden when empty. - [ ] 5.2 Map-match helper: nearest OSM way per route segment → surface/highway; unmatched → unknown. Unit-test the matcher on a small fixture.
- [ ] 5.3 E2E: a Planner-saved route with waytags shows the breakdown bars on its detail page (or assert via a seeded route if the planner round-trip is too heavy for e2e). - [ ] 5.3 Typed `surface-backfill` pg-boss job (`{ kind, id }`): load geometry → skip if breakdown exists → Overpass → map-match → store. Idempotent, retry-safe, best-effort.
- [ ] 5.4 `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` green. - [ ] 5.4 Enqueue the job after GPX import (Komoot/Garmin) and manual upload; add a one-off sweep entrypoint for pre-existing rows.
- [ ] 5.5 Emit a `surface_breakdown` SSE event (`{ kind, id }`) on completion; detail page subscribes (EventSource hook) and re-fetches the breakdown when it matches; optional "computing…" state.
## 6. Privacy
- [ ] 6.1 Document the backfill's Overpass lookup in the privacy manifest; route via the proxy (self-hostable Overpass is the long-term answer — see `docs/ideas/self-host-overpass`).
## 7. Tests & checks
- [ ] 7.1 Unit: `surfaceBreakdown` weighting; map-matcher nearest-way + unknown bucket.
- [ ] 7.2 Component (jsdom): bars/proportions, "other" bucket, hidden when empty.
- [ ] 7.3 E2E: a route/activity with a stored breakdown shows the bars (seed the breakdown to avoid a live Overpass call in CI).
- [ ] 7.4 `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` green.