From e1e3dc5c812f17afef1f18125263bae15d82882f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ullrich=20Sch=C3=A4fer?= Date: Sun, 14 Jun 2026 17:54:50 +0200 Subject: [PATCH] =?UTF-8?q?openspec:=20revise=20route-surface-breakdown=20?= =?UTF-8?q?=E2=80=94=20add=20async=20Overpass=20backfill=20+=20SSE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Builds on the merged A-only proposal. Broadens the capability: - breakdown now on routes AND activities; - Path 1 (sync) unchanged: BRouter waytags → breakdown at Planner save; - Path 2 (async): a `surface-backfill` pg-boss job map-matches geometry to OSM via Overpass for imports/uploads/older rows, stores the breakdown, and pushes a `surface_breakdown` SSE event so an open detail page fills in live; - privacy note: backfill sends geometry to Overpass → route via the proxy / self-hostable Overpass; documented in the manifest. Validates with `openspec validate --strict`. Specs only; no code. Co-Authored-By: Claude Opus 4.8 --- .../changes/route-surface-breakdown/design.md | 123 ++++++++++-------- .../route-surface-breakdown/proposal.md | 71 +++++----- .../specs/route-surface-breakdown/spec.md | 36 ++++- .../changes/route-surface-breakdown/tasks.md | 42 +++--- 4 files changed, 167 insertions(+), 105 deletions(-) diff --git a/openspec/changes/route-surface-breakdown/design.md b/openspec/changes/route-surface-breakdown/design.md index 203b7ed..c9d936a 100644 --- a/openspec/changes/route-surface-breakdown/design.md +++ b/openspec/changes/route-surface-breakdown/design.md @@ -1,73 +1,90 @@ ## Context The Planner computes accurate per-coordinate surface/highway from BRouter's -`messages`/`WayTags` (`apps/planner/.../route-merge.ts` → `EnrichedRoute`), and -`ColoredRoute` already renders the route coloured by surface. But this metadata -is **session-only**: the planner→journal handoff (`api.save-to-journal`) POSTs -only `{ gpx }` to the Journal callback (`api.routes.$id.callback`), the `routes` -table has no surface column, and GPX carries no surface extensions. So today no -saved route has surface data. +`WayTags` (`apps/planner/.../route-merge.ts` → `EnrichedRoute`) and colours the +line with it, but it's session-only: `api.save-to-journal` POSTs only `{ gpx }`, +`routes` has no surface column, GPX carries no surface tags. The repo already +has a typed pg-boss job system (`apps/journal/app/jobs/payloads.ts`, +`@trails-cool/jobs`), an SSE broker (`/api/events` + EventSource hooks, e.g. +`useUnreadNotifications`), and an Overpass proxy used for Planner POIs +(`apps/planner/.../overpass.ts`, `/api/overpass`). ## Goals / Non-Goals **Goals** -- Surface + waytype proportion bars on route detail, using data we already - compute, with no new external dependency. +- Surface + waytype bars on route **and** activity detail. +- Two derivation paths: free/accurate for Planner routes; async backfill for + everything else, with a live SSE update. **Non-Goals (v1)** -- **Activities / imported / manually-uploaded GPX** — they have no waytags - (bare GPX). They show no breakdown for now (see "Data source" → future - Overpass). -- **Colouring the Journal map line by surface** — needs per-segment data, not - just the summary; a clean follow-up once the bars land. -- Selectable surface-vs-waytype toggle UI beyond showing both bars. +- Colouring the Journal **map line** by surface (needs per-segment data, not the + summary; a clean follow-up). +- Re-backfilling on every geometry edit (backfill is one-shot per row unless + re-triggered). ## Decisions -### D1: Data source — persist BRouter waytags at save (Planner routes only) -Two ways to get a breakdown onto the Journal detail page: +### D1: Two derivation paths into one stored summary +Both paths produce the same compact, distance-weighted summary — +`{ surface: { asphalt: , gravel: , … }, highway: { … } }` (metres per +category) — stored as nullable `surface_breakdown` jsonb on `routes` and +`activities`. The bars need proportions, not geometry, so we store a summary, +not per-segment data (per-segment is a future map-colouring concern). -- **(A, chosen) BRouter-at-save** — the Planner already has 100%-accurate - per-segment surface/highway; compute the distribution there at save and - persist it. Accurate, cheap, no external call. **Limitation:** only - Planner-created routes (activities/uploads have no waytags). -- **(B, future) Overpass map-matching** — snap arbitrary route geometry to OSM - ways and read `surface`/`highway` server-side. Covers imports/uploads, but - it's expensive, approximate, rate-limited, and an external dependency. Parked - for when we want imports covered (the repo already proxies Overpass for POIs). +- **Path 1 — synchronous, Planner routes (accurate, free).** The Planner has + 100%-accurate per-segment tags; compute the summary at save and persist via an + extra optional `surfaceBreakdown` field in the handoff. No external call. +- **Path 2 — async backfill, everything else (best-effort).** For a row with + geometry but no summary, a job map-matches to OSM and computes it (D2). -v1 ships (A). Routes without the data degrade to no bars. +### D2: The async backfill job +A typed `surface-backfill` pg-boss job keyed by `{ kind: "route" | "activity", +id }`: +1. Load the geometry; bail if absent or already has a summary. +2. Query Overpass for `way[highway]` within the route bbox (a journal-side + client mirroring the Planner's `buildQuery`/`parseResponse` + rate-limit + handling). +3. **Map-match**: for each route segment, find the nearest OSM way and read its + `surface`/`highway`; bucket the segment's length. Unmatched/untagged → + `unknown`. +4. Store the summary; emit the SSE event (D3). +- **Triggers:** enqueued after a GPX import (Komoot/Garmin) and after a manual + upload; plus a one-off sweep job to backfill pre-existing rows. Optionally + enqueued on-demand when a detail page without a summary is opened. +- **Resilience:** pg-boss retries cover Overpass rate-limits/outages; the job is + idempotent (skips if a summary already exists) and best-effort (failure leaves + the row with no bars, not an error). -### D2: Store a compact distance-weighted summary, not per-segment data -The bars need **proportions**, not geometry. At save, sum the haversine length -of each coordinate segment into its surface bucket (and highway bucket), -yielding `{ surface: { asphalt: , gravel: , … }, highway: { … } }` -(metres per category). This is tiny (a handful of keys), stored as a nullable -`surface_breakdown` jsonb on `routes`. Per-segment data (for future map line -colouring) is deliberately *not* persisted in v1 — that's a bigger payload and -a separate feature. - -### D3: Handoff carries it -`api.save-to-journal` adds `surfaceBreakdown` to the POST body alongside `gpx`; -the Journal callback validates (Zod, all values ≥ 0) and stores it. The field -is optional, so the handoff stays backward-compatible and non-Planner callers -(if any) are unaffected. Re-saving a route recomputes/overwrites it. +### D3: SSE live update +On backfill completion the job publishes a `surface_breakdown` SSE event +(payload: `{ kind, id }`) via the existing broker. The detail page subscribes +(EventSource hook) and, when the event matches the row it's showing, re-fetches +the breakdown and fills the bars — so a user who opened the page before the job +finished sees it appear without reloading. While a backfill is known to be +in-flight, the page may show a small "computing…" affordance. ### D4: Rendering -A `SurfaceBreakdown` component renders one horizontal stacked bar per dimension -(surface, then waytype): segments sized by percentage, coloured via -`SURFACE_COLORS` / `HIGHWAY_COLORS` (`DEFAULT_*` for unknown), with a legend -listing each category + km + percent (largest first). Unknown/unmapped tags -collapse into an "other" segment. Hidden entirely when `surface_breakdown` is -null or empty. Labels via i18n; distances via `stats.ts`. +A `SurfaceBreakdown` component: one horizontal stacked bar per dimension +(surface, then waytype), segments sized by percentage and coloured via +`SURFACE_COLORS`/`HIGHWAY_COLORS` (`DEFAULT_*` for unknown), with a legend +(category · km · %, largest first). Unknown/unmapped tags collapse into "other". +Hidden when the summary is null/empty. Labels via i18n; distances via `stats.ts`. + +### D5: Privacy (backfill sends geometry to Overpass) +Path 1 sends nothing externally. Path 2's Overpass query exposes a route's +bounding box to the Overpass endpoint — a real consideration for a privacy-first +app. Mitigations: route the backfill through the **proxied / self-hostable** +Overpass (the parked `self-host-overpass` idea removes the third-party entirely), +and only the bbox + geometry needed for matching is sent. We document the +backfill's external lookup in the privacy manifest; a self-hosted Overpass is the +clean long-term answer. ## Risks / Trade-offs -- **Coverage gap** — only Planner routes get bars in v1. Acceptable: routes are - *designed* in the Planner, which is exactly where "how much gravel" matters. - Imports get bars later via (B). -- **Tag sparsity** — OSM `surface` is often missing; those segments bucket as - `unknown`/"other". We show it honestly rather than guessing. -- **Stale on edit** — editing a route's geometry elsewhere without re-routing - could leave the breakdown stale; recomputing on every Planner save keeps it - fresh for the normal flow. +- **Map-matching is approximate** — nearest-way snapping mismatches at junctions + / parallel ways; acceptable for a proportional overview, and Path 1 (exact) is + preferred whenever available. +- **Overpass cost/limits** — bounded by the async queue + retries; the sweep + should throttle. A self-hosted Overpass removes the ceiling. +- **Two code paths** — both converge on one stored summary + one renderer, so + only the *derivation* differs. diff --git a/openspec/changes/route-surface-breakdown/proposal.md b/openspec/changes/route-surface-breakdown/proposal.md index 3ce0b00..d0fef08 100644 --- a/openspec/changes/route-surface-breakdown/proposal.md +++ b/openspec/changes/route-surface-breakdown/proposal.md @@ -1,46 +1,57 @@ ## Why -"How much of this is gravel?" is a top-three question for anyone choosing a -route — Komoot's surface/waytype breakdown bars are one of its most-used -features, and `@trails-cool/map-core` already ships the surface/highway colour -palettes. We also already *compute* accurate per-segment surface and highway -tags from BRouter while planning (`route-merge` → `EnrichedRoute.surfaces[]`), -then **throw them away on save**. This change captures that data and shows it. +"How much of this is gravel?" is a top-three question for choosing a route, and +Komoot's surface/waytype bars are one of its most-used features. +`@trails-cool/map-core` already ships the surface/highway colour palettes, and +the Planner already *computes* accurate per-segment surface/highway from BRouter +(`route-merge` → `EnrichedRoute.surfaces[]`) — then **discards it on save**. +This change captures that data, shows it as proportion bars, and **backfills +everything else** (imports, uploads, older rows) asynchronously so the bars +aren't limited to Planner routes. ## What Changes -- On **route detail**, show surface and waytype **proportion bars** (e.g. - "62% asphalt · 28% gravel · 10% path"), coloured with the `map-core` - `SURFACE_COLORS` / `HIGHWAY_COLORS` palettes, with per-segment km/percent. -- **Persist the breakdown for Planner-created routes**: compute a - distance-weighted surface + waytype distribution from the BRouter waytags at - save time, send it in the planner→journal handoff, and store it on the route. -- Degrade gracefully: routes without breakdown data (older routes, or activities - / manually-imported GPX that never had waytags) simply don't show the bars. +- On **route and activity detail**, show surface and waytype **proportion bars** + (e.g. "62% asphalt · 28% gravel · 10% path"), coloured with the `map-core` + `SURFACE_COLORS` / `HIGHWAY_COLORS` palettes, with per-category km/percent. +- **Path 1 — synchronous (Planner routes):** compute a distance-weighted + surface + waytype distribution from the BRouter waytags at save time, send it + in the planner→journal handoff, store it. Accurate, free, no external call. +- **Path 2 — async backfill (everything else):** for a route/activity that has + geometry but no breakdown (imports, manual uploads, pre-existing rows), a + background job map-matches the geometry to OSM ways via Overpass, computes the + breakdown, stores it, and **pushes an SSE event** so an open detail page fills + the bars in live. +- Degrade gracefully: no geometry / not-yet-backfilled → no bars (or a + "computing…" state while a backfill is in flight). ## Capabilities ### New Capabilities -- `route-surface-breakdown`: the surface/waytype proportion bars on route - detail, how the distribution is derived from BRouter waytags, persisted at - save, and rendered. +- `route-surface-breakdown`: the surface/waytype proportion bars on route & + activity detail — derived synchronously from BRouter waytags (Planner) or + asynchronously via an Overpass backfill job with an SSE live-update. ### Modified Capabilities - + ## Impact -- **Schema**: a nullable `surface_breakdown` jsonb column on `journal.routes` - (additive — `pnpm db:push`, no data migration). -- **Planner**: `api.save-to-journal` computes the distance-weighted breakdown - from the session `EnrichedRoute` (surfaces/highways + coordinates) and adds it - to the POST body. -- **Journal**: the route callback (`api.routes.$id.callback`) accepts + stores - it; the route detail loader exposes it; a `SurfaceBreakdown` component renders - the bars (reusing `map-core` palettes + `stats.ts` distance formatting). -- **No new external dependency** — BRouter already returns the tags; no Overpass - call. Imported activities/uploads are out of scope for v1 (see design). +- **Schema**: nullable `surface_breakdown` jsonb on `journal.routes` **and** + `journal.activities` (additive — `pnpm db:push`). +- **Planner**: `api.save-to-journal` computes + sends the breakdown. +- **Journal**: + - route callback stores the synchronous breakdown; + - a typed `surface-backfill` pg-boss job (Overpass map-match → breakdown → + store → SSE), enqueued after imports/uploads and available as a sweep; + - a journal-side Overpass way/surface client (reusing the existing + `/api/overpass` proxy pattern); + - an SSE `surface_breakdown` event + a detail-page subscription; + - a `SurfaceBreakdown` component (reuses `map-core` palettes + `stats.ts`). +- **i18n**: `journal.surface.*` (en + de). +- **Privacy**: the backfill sends route geometry to Overpass — see design for + the proxy/self-host + visibility considerations. diff --git a/openspec/changes/route-surface-breakdown/specs/route-surface-breakdown/spec.md b/openspec/changes/route-surface-breakdown/specs/route-surface-breakdown/spec.md index 6a8dac7..5a31075 100644 --- a/openspec/changes/route-surface-breakdown/specs/route-surface-breakdown/spec.md +++ b/openspec/changes/route-surface-breakdown/specs/route-surface-breakdown/spec.md @@ -1,21 +1,43 @@ ## ADDED Requirements ### Requirement: Surface breakdown derived from BRouter waytags -The system SHALL compute a distance-weighted surface and waytype distribution for a Planner-created route from the per-segment BRouter waytags at save time, and persist it with the route. +The system SHALL compute a distance-weighted surface and waytype distribution for a Planner-created route from its per-segment BRouter waytags at save time, and persist it with the route. #### Scenario: Breakdown captured on save - **WHEN** a route is saved from the Planner with per-segment surface/highway waytags - **THEN** the route stores a distance-weighted breakdown of metres per surface category and per waytype category #### Scenario: No waytags available -- **WHEN** a route is saved without waytags (no surface data) -- **THEN** no breakdown is stored, and the route detail shows no breakdown bars +- **WHEN** a route is saved without waytags +- **THEN** no breakdown is stored synchronously (an async backfill may fill it later) + +### Requirement: Async surface backfill via Overpass +The system SHALL asynchronously derive a surface breakdown for a route or activity that has geometry but no stored breakdown, by map-matching its geometry to OpenStreetMap ways via Overpass in a background job, and SHALL store the resulting distance-weighted breakdown. + +#### Scenario: Imported activity is backfilled +- **WHEN** an activity is imported (e.g. from Komoot) with geometry but no waytags +- **THEN** a backfill job map-matches its geometry to OSM ways and stores a surface/waytype breakdown + +#### Scenario: Backfill is best-effort and idempotent +- **WHEN** a backfill job runs for a row that already has a breakdown +- **THEN** it does no work; and **WHEN** Overpass is unavailable, the row is left without a breakdown rather than erroring + +#### Scenario: Untagged segments +- **WHEN** map-matched ways have no surface tag +- **THEN** those segments' distance is bucketed as unknown + +### Requirement: Live update when a backfill completes +The detail page SHALL update its breakdown without a reload when a backfill for the row being viewed completes, via a server-sent event. + +#### Scenario: Bars appear after backfill +- **WHEN** a user is viewing a route/activity detail page whose backfill finishes +- **THEN** the page receives a `surface_breakdown` event for that row and renders the bars without a manual reload ### Requirement: Surface and waytype proportion bars -The route detail page SHALL render the surface and waytype breakdown as proportion bars when the route has breakdown data, coloured with the shared surface/highway palettes, each segment labelled with its category, distance, and percentage. +The route and activity detail pages SHALL render the surface and waytype breakdown as proportion bars when the row has breakdown data, coloured with the shared surface/highway palettes, each segment labelled with its category, distance, and percentage. -#### Scenario: Bars shown for a route with a breakdown -- **WHEN** a user views a route that has a stored surface breakdown +#### Scenario: Bars shown for a row with a breakdown +- **WHEN** a user views a route or activity that has a stored surface breakdown - **THEN** a surface proportion bar and a waytype proportion bar are shown, each segment coloured by category with the largest category first #### Scenario: Percentages reflect distance share @@ -27,7 +49,7 @@ The route detail page SHALL render the surface and waytype breakdown as proporti - **THEN** their distance is shown as an "other" segment rather than omitted #### Scenario: Hidden without data -- **WHEN** a route has no stored breakdown +- **WHEN** a row has no stored breakdown - **THEN** no breakdown bars are rendered ### Requirement: Localized breakdown labels diff --git a/openspec/changes/route-surface-breakdown/tasks.md b/openspec/changes/route-surface-breakdown/tasks.md index 9eaf300..68d50a6 100644 --- a/openspec/changes/route-surface-breakdown/tasks.md +++ b/openspec/changes/route-surface-breakdown/tasks.md @@ -1,26 +1,38 @@ -## 1. Breakdown derivation (shared, pure) +## 1. Shared derivation + shape -- [ ] 1.1 Add a pure `surfaceBreakdown(coordinates, surfaces, highways)` helper (in `@trails-cool/map-core` or a planner lib): sum haversine segment lengths into per-surface and per-waytype buckets → `{ surface: Record, highway: Record }` (metres). Unit-test it. -- [ ] 1.2 Define the shared wire shape (Zod) for the breakdown in `@trails-cool/api` (`SurfaceBreakdownSchema`, all values ≥ 0). +- [ ] 1.1 Pure `surfaceBreakdown(coordinates, surfaces, highways)` helper: haversine-weight each segment into per-surface + per-waytype buckets → `{ surface: Record, highway: Record }` (metres). Unit-test. +- [ ] 1.2 `SurfaceBreakdownSchema` (Zod) in `@trails-cool/api` (record of category → metres ≥ 0, surface + highway). -## 2. Capture at save (Planner → Journal) +## 2. Schema -- [ ] 2.1 In `apps/planner/.../api.save-to-journal`, read the session `EnrichedRoute` (coordinates + surfaces + highways), compute the breakdown, and add `surfaceBreakdown` to the POST body. -- [ ] 2.2 Journal callback (`api.routes.$id.callback`) validates the optional `surfaceBreakdown` and persists it; re-save overwrites. +- [ ] 2.1 Nullable `surfaceBreakdown` jsonb on `journal.routes` **and** `journal.activities`; `pnpm db:push`. -## 3. Schema +## 3. Path 1 — synchronous (Planner routes) -- [ ] 3.1 Add nullable `surfaceBreakdown` jsonb to `journal.routes`; `pnpm db:push`. +- [ ] 3.1 `apps/planner/.../api.save-to-journal`: compute the breakdown from the session `EnrichedRoute` (coords + surfaces + highways) and add `surfaceBreakdown` to the POST body. +- [ ] 3.2 Journal route callback validates the optional field and persists it (overwrite on re-save). ## 4. Render -- [ ] 4.1 Route detail loader exposes `surfaceBreakdown`. -- [ ] 4.2 `SurfaceBreakdown` component: stacked proportion bar per dimension (surface, waytype), segments coloured via `SURFACE_COLORS`/`HIGHWAY_COLORS` (DEFAULT for unknown), legend with category · km · % (largest first), unknown → "other"; hidden when empty. Mount on `routes.$id.tsx`. +- [ ] 4.1 Route + activity detail loaders expose `surfaceBreakdown`. +- [ ] 4.2 `SurfaceBreakdown` component: stacked bar per dimension (surface, waytype), `map-core` `SURFACE_COLORS`/`HIGHWAY_COLORS` (DEFAULT → "other"), legend category · km · % (largest first), hidden when empty. Mount on `routes.$id.tsx` and `activities.$id.tsx`. - [ ] 4.3 i18n `journal.surface.*` (label + category names + "other") en + de. -## 5. Tests & checks +## 5. Path 2 — async backfill + SSE -- [ ] 5.1 Unit: `surfaceBreakdown` (distance weighting, multiple surfaces, unknown bucket). -- [ ] 5.2 Component (jsdom): bars render with correct proportions; "other" for unknowns; hidden when empty. -- [ ] 5.3 E2E: a Planner-saved route with waytags shows the breakdown bars on its detail page (or assert via a seeded route if the planner round-trip is too heavy for e2e). -- [ ] 5.4 `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` green. +- [ ] 5.1 Journal-side Overpass client: `way[highway]` in bbox + parse (mirror the planner's `buildQuery`/`parseResponse` + rate-limit handling; route via the `/api/overpass` proxy). +- [ ] 5.2 Map-match helper: nearest OSM way per route segment → surface/highway; unmatched → unknown. Unit-test the matcher on a small fixture. +- [ ] 5.3 Typed `surface-backfill` pg-boss job (`{ kind, id }`): load geometry → skip if breakdown exists → Overpass → map-match → store. Idempotent, retry-safe, best-effort. +- [ ] 5.4 Enqueue the job after GPX import (Komoot/Garmin) and manual upload; add a one-off sweep entrypoint for pre-existing rows. +- [ ] 5.5 Emit a `surface_breakdown` SSE event (`{ kind, id }`) on completion; detail page subscribes (EventSource hook) and re-fetches the breakdown when it matches; optional "computing…" state. + +## 6. Privacy + +- [ ] 6.1 Document the backfill's Overpass lookup in the privacy manifest; route via the proxy (self-hostable Overpass is the long-term answer — see `docs/ideas/self-host-overpass`). + +## 7. Tests & checks + +- [ ] 7.1 Unit: `surfaceBreakdown` weighting; map-matcher nearest-way + unknown bucket. +- [ ] 7.2 Component (jsdom): bars/proportions, "other" bucket, hidden when empty. +- [ ] 7.3 E2E: a route/activity with a stored breakdown shows the bars (seed the breakdown to avoid a live Overpass call in CI). +- [ ] 7.4 `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` green.