openspec: revise route-surface-breakdown — add async Overpass backfill + SSE

Builds on the merged A-only proposal. Broadens the capability:
- breakdown now on routes AND activities;
- Path 1 (sync) unchanged: BRouter waytags → breakdown at Planner save;
- Path 2 (async): a `surface-backfill` pg-boss job map-matches geometry to OSM
  via Overpass for imports/uploads/older rows, stores the breakdown, and pushes
  a `surface_breakdown` SSE event so an open detail page fills in live;
- privacy note: backfill sends geometry to Overpass → route via the proxy /
  self-hostable Overpass; documented in the manifest.

Validates with `openspec validate --strict`. Specs only; no code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-06-14 17:54:50 +02:00
parent a81134332b
commit e1e3dc5c81
No known key found for this signature in database
GPG key ID: A32FF691A0F752D9
4 changed files with 167 additions and 105 deletions

View file

@ -1,73 +1,90 @@
## Context
The Planner computes accurate per-coordinate surface/highway from BRouter's
`messages`/`WayTags` (`apps/planner/.../route-merge.ts``EnrichedRoute`), and
`ColoredRoute` already renders the route coloured by surface. But this metadata
is **session-only**: the planner→journal handoff (`api.save-to-journal`) POSTs
only `{ gpx }` to the Journal callback (`api.routes.$id.callback`), the `routes`
table has no surface column, and GPX carries no surface extensions. So today no
saved route has surface data.
`WayTags` (`apps/planner/.../route-merge.ts``EnrichedRoute`) and colours the
line with it, but it's session-only: `api.save-to-journal` POSTs only `{ gpx }`,
`routes` has no surface column, GPX carries no surface tags. The repo already
has a typed pg-boss job system (`apps/journal/app/jobs/payloads.ts`,
`@trails-cool/jobs`), an SSE broker (`/api/events` + EventSource hooks, e.g.
`useUnreadNotifications`), and an Overpass proxy used for Planner POIs
(`apps/planner/.../overpass.ts`, `/api/overpass`).
## Goals / Non-Goals
**Goals**
- Surface + waytype proportion bars on route detail, using data we already
compute, with no new external dependency.
- Surface + waytype bars on route **and** activity detail.
- Two derivation paths: free/accurate for Planner routes; async backfill for
everything else, with a live SSE update.
**Non-Goals (v1)**
- **Activities / imported / manually-uploaded GPX** — they have no waytags
(bare GPX). They show no breakdown for now (see "Data source" → future
Overpass).
- **Colouring the Journal map line by surface** — needs per-segment data, not
just the summary; a clean follow-up once the bars land.
- Selectable surface-vs-waytype toggle UI beyond showing both bars.
- Colouring the Journal **map line** by surface (needs per-segment data, not the
summary; a clean follow-up).
- Re-backfilling on every geometry edit (backfill is one-shot per row unless
re-triggered).
## Decisions
### D1: Data source — persist BRouter waytags at save (Planner routes only)
Two ways to get a breakdown onto the Journal detail page:
### D1: Two derivation paths into one stored summary
Both paths produce the same compact, distance-weighted summary —
`{ surface: { asphalt: <m>, gravel: <m>, … }, highway: { … } }` (metres per
category) — stored as nullable `surface_breakdown` jsonb on `routes` and
`activities`. The bars need proportions, not geometry, so we store a summary,
not per-segment data (per-segment is a future map-colouring concern).
- **(A, chosen) BRouter-at-save** — the Planner already has 100%-accurate
per-segment surface/highway; compute the distribution there at save and
persist it. Accurate, cheap, no external call. **Limitation:** only
Planner-created routes (activities/uploads have no waytags).
- **(B, future) Overpass map-matching** — snap arbitrary route geometry to OSM
ways and read `surface`/`highway` server-side. Covers imports/uploads, but
it's expensive, approximate, rate-limited, and an external dependency. Parked
for when we want imports covered (the repo already proxies Overpass for POIs).
- **Path 1 — synchronous, Planner routes (accurate, free).** The Planner has
100%-accurate per-segment tags; compute the summary at save and persist via an
extra optional `surfaceBreakdown` field in the handoff. No external call.
- **Path 2 — async backfill, everything else (best-effort).** For a row with
geometry but no summary, a job map-matches to OSM and computes it (D2).
v1 ships (A). Routes without the data degrade to no bars.
### D2: The async backfill job
A typed `surface-backfill` pg-boss job keyed by `{ kind: "route" | "activity",
id }`:
1. Load the geometry; bail if absent or already has a summary.
2. Query Overpass for `way[highway]` within the route bbox (a journal-side
client mirroring the Planner's `buildQuery`/`parseResponse` + rate-limit
handling).
3. **Map-match**: for each route segment, find the nearest OSM way and read its
`surface`/`highway`; bucket the segment's length. Unmatched/untagged →
`unknown`.
4. Store the summary; emit the SSE event (D3).
- **Triggers:** enqueued after a GPX import (Komoot/Garmin) and after a manual
upload; plus a one-off sweep job to backfill pre-existing rows. Optionally
enqueued on-demand when a detail page without a summary is opened.
- **Resilience:** pg-boss retries cover Overpass rate-limits/outages; the job is
idempotent (skips if a summary already exists) and best-effort (failure leaves
the row with no bars, not an error).
### D2: Store a compact distance-weighted summary, not per-segment data
The bars need **proportions**, not geometry. At save, sum the haversine length
of each coordinate segment into its surface bucket (and highway bucket),
yielding `{ surface: { asphalt: <m>, gravel: <m>, … }, highway: { … } }`
(metres per category). This is tiny (a handful of keys), stored as a nullable
`surface_breakdown` jsonb on `routes`. Per-segment data (for future map line
colouring) is deliberately *not* persisted in v1 — that's a bigger payload and
a separate feature.
### D3: Handoff carries it
`api.save-to-journal` adds `surfaceBreakdown` to the POST body alongside `gpx`;
the Journal callback validates (Zod, all values ≥ 0) and stores it. The field
is optional, so the handoff stays backward-compatible and non-Planner callers
(if any) are unaffected. Re-saving a route recomputes/overwrites it.
### D3: SSE live update
On backfill completion the job publishes a `surface_breakdown` SSE event
(payload: `{ kind, id }`) via the existing broker. The detail page subscribes
(EventSource hook) and, when the event matches the row it's showing, re-fetches
the breakdown and fills the bars — so a user who opened the page before the job
finished sees it appear without reloading. While a backfill is known to be
in-flight, the page may show a small "computing…" affordance.
### D4: Rendering
A `SurfaceBreakdown` component renders one horizontal stacked bar per dimension
(surface, then waytype): segments sized by percentage, coloured via
`SURFACE_COLORS` / `HIGHWAY_COLORS` (`DEFAULT_*` for unknown), with a legend
listing each category + km + percent (largest first). Unknown/unmapped tags
collapse into an "other" segment. Hidden entirely when `surface_breakdown` is
null or empty. Labels via i18n; distances via `stats.ts`.
A `SurfaceBreakdown` component: one horizontal stacked bar per dimension
(surface, then waytype), segments sized by percentage and coloured via
`SURFACE_COLORS`/`HIGHWAY_COLORS` (`DEFAULT_*` for unknown), with a legend
(category · km · %, largest first). Unknown/unmapped tags collapse into "other".
Hidden when the summary is null/empty. Labels via i18n; distances via `stats.ts`.
### D5: Privacy (backfill sends geometry to Overpass)
Path 1 sends nothing externally. Path 2's Overpass query exposes a route's
bounding box to the Overpass endpoint — a real consideration for a privacy-first
app. Mitigations: route the backfill through the **proxied / self-hostable**
Overpass (the parked `self-host-overpass` idea removes the third-party entirely),
and only the bbox + geometry needed for matching is sent. We document the
backfill's external lookup in the privacy manifest; a self-hosted Overpass is the
clean long-term answer.
## Risks / Trade-offs
- **Coverage gap** — only Planner routes get bars in v1. Acceptable: routes are
*designed* in the Planner, which is exactly where "how much gravel" matters.
Imports get bars later via (B).
- **Tag sparsity** — OSM `surface` is often missing; those segments bucket as
`unknown`/"other". We show it honestly rather than guessing.
- **Stale on edit** — editing a route's geometry elsewhere without re-routing
could leave the breakdown stale; recomputing on every Planner save keeps it
fresh for the normal flow.
- **Map-matching is approximate** — nearest-way snapping mismatches at junctions
/ parallel ways; acceptable for a proportional overview, and Path 1 (exact) is
preferred whenever available.
- **Overpass cost/limits** — bounded by the async queue + retries; the sweep
should throttle. A self-hosted Overpass removes the ceiling.
- **Two code paths** — both converge on one stored summary + one renderer, so
only the *derivation* differs.

View file

@ -1,46 +1,57 @@
## Why
"How much of this is gravel?" is a top-three question for anyone choosing a
route — Komoot's surface/waytype breakdown bars are one of its most-used
features, and `@trails-cool/map-core` already ships the surface/highway colour
palettes. We also already *compute* accurate per-segment surface and highway
tags from BRouter while planning (`route-merge``EnrichedRoute.surfaces[]`),
then **throw them away on save**. This change captures that data and shows it.
"How much of this is gravel?" is a top-three question for choosing a route, and
Komoot's surface/waytype bars are one of its most-used features.
`@trails-cool/map-core` already ships the surface/highway colour palettes, and
the Planner already *computes* accurate per-segment surface/highway from BRouter
(`route-merge``EnrichedRoute.surfaces[]`) — then **discards it on save**.
This change captures that data, shows it as proportion bars, and **backfills
everything else** (imports, uploads, older rows) asynchronously so the bars
aren't limited to Planner routes.
## What Changes
- On **route detail**, show surface and waytype **proportion bars** (e.g.
"62% asphalt · 28% gravel · 10% path"), coloured with the `map-core`
`SURFACE_COLORS` / `HIGHWAY_COLORS` palettes, with per-segment km/percent.
- **Persist the breakdown for Planner-created routes**: compute a
distance-weighted surface + waytype distribution from the BRouter waytags at
save time, send it in the planner→journal handoff, and store it on the route.
- Degrade gracefully: routes without breakdown data (older routes, or activities
/ manually-imported GPX that never had waytags) simply don't show the bars.
- On **route and activity detail**, show surface and waytype **proportion bars**
(e.g. "62% asphalt · 28% gravel · 10% path"), coloured with the `map-core`
`SURFACE_COLORS` / `HIGHWAY_COLORS` palettes, with per-category km/percent.
- **Path 1 — synchronous (Planner routes):** compute a distance-weighted
surface + waytype distribution from the BRouter waytags at save time, send it
in the planner→journal handoff, store it. Accurate, free, no external call.
- **Path 2 — async backfill (everything else):** for a route/activity that has
geometry but no breakdown (imports, manual uploads, pre-existing rows), a
background job map-matches the geometry to OSM ways via Overpass, computes the
breakdown, stores it, and **pushes an SSE event** so an open detail page fills
the bars in live.
- Degrade gracefully: no geometry / not-yet-backfilled → no bars (or a
"computing…" state while a backfill is in flight).
## Capabilities
### New Capabilities
- `route-surface-breakdown`: the surface/waytype proportion bars on route
detail, how the distribution is derived from BRouter waytags, persisted at
save, and rendered.
- `route-surface-breakdown`: the surface/waytype proportion bars on route &
activity detail — derived synchronously from BRouter waytags (Planner) or
asynchronously via an Overpass backfill job with an SSE live-update.
### Modified Capabilities
<!-- `planner-journal-handoff` gains one optional field in its save payload;
noted in design, but the breakdown is a new capability, not a change to the
handoff's existing requirements. -->
<!-- `planner-journal-handoff` gains one optional payload field; `background-jobs`
and `sse-broker` gain a new job + event, but those are additive uses of
existing transports, owned by this new capability. -->
## Impact
- **Schema**: a nullable `surface_breakdown` jsonb column on `journal.routes`
(additive — `pnpm db:push`, no data migration).
- **Planner**: `api.save-to-journal` computes the distance-weighted breakdown
from the session `EnrichedRoute` (surfaces/highways + coordinates) and adds it
to the POST body.
- **Journal**: the route callback (`api.routes.$id.callback`) accepts + stores
it; the route detail loader exposes it; a `SurfaceBreakdown` component renders
the bars (reusing `map-core` palettes + `stats.ts` distance formatting).
- **No new external dependency** — BRouter already returns the tags; no Overpass
call. Imported activities/uploads are out of scope for v1 (see design).
- **Schema**: nullable `surface_breakdown` jsonb on `journal.routes` **and**
`journal.activities` (additive — `pnpm db:push`).
- **Planner**: `api.save-to-journal` computes + sends the breakdown.
- **Journal**:
- route callback stores the synchronous breakdown;
- a typed `surface-backfill` pg-boss job (Overpass map-match → breakdown →
store → SSE), enqueued after imports/uploads and available as a sweep;
- a journal-side Overpass way/surface client (reusing the existing
`/api/overpass` proxy pattern);
- an SSE `surface_breakdown` event + a detail-page subscription;
- a `SurfaceBreakdown` component (reuses `map-core` palettes + `stats.ts`).
- **i18n**: `journal.surface.*` (en + de).
- **Privacy**: the backfill sends route geometry to Overpass — see design for
the proxy/self-host + visibility considerations.

View file

@ -1,21 +1,43 @@
## ADDED Requirements
### Requirement: Surface breakdown derived from BRouter waytags
The system SHALL compute a distance-weighted surface and waytype distribution for a Planner-created route from the per-segment BRouter waytags at save time, and persist it with the route.
The system SHALL compute a distance-weighted surface and waytype distribution for a Planner-created route from its per-segment BRouter waytags at save time, and persist it with the route.
#### Scenario: Breakdown captured on save
- **WHEN** a route is saved from the Planner with per-segment surface/highway waytags
- **THEN** the route stores a distance-weighted breakdown of metres per surface category and per waytype category
#### Scenario: No waytags available
- **WHEN** a route is saved without waytags (no surface data)
- **THEN** no breakdown is stored, and the route detail shows no breakdown bars
- **WHEN** a route is saved without waytags
- **THEN** no breakdown is stored synchronously (an async backfill may fill it later)
### Requirement: Async surface backfill via Overpass
The system SHALL asynchronously derive a surface breakdown for a route or activity that has geometry but no stored breakdown, by map-matching its geometry to OpenStreetMap ways via Overpass in a background job, and SHALL store the resulting distance-weighted breakdown.
#### Scenario: Imported activity is backfilled
- **WHEN** an activity is imported (e.g. from Komoot) with geometry but no waytags
- **THEN** a backfill job map-matches its geometry to OSM ways and stores a surface/waytype breakdown
#### Scenario: Backfill is best-effort and idempotent
- **WHEN** a backfill job runs for a row that already has a breakdown
- **THEN** it does no work; and **WHEN** Overpass is unavailable, the row is left without a breakdown rather than erroring
#### Scenario: Untagged segments
- **WHEN** map-matched ways have no surface tag
- **THEN** those segments' distance is bucketed as unknown
### Requirement: Live update when a backfill completes
The detail page SHALL update its breakdown without a reload when a backfill for the row being viewed completes, via a server-sent event.
#### Scenario: Bars appear after backfill
- **WHEN** a user is viewing a route/activity detail page whose backfill finishes
- **THEN** the page receives a `surface_breakdown` event for that row and renders the bars without a manual reload
### Requirement: Surface and waytype proportion bars
The route detail page SHALL render the surface and waytype breakdown as proportion bars when the route has breakdown data, coloured with the shared surface/highway palettes, each segment labelled with its category, distance, and percentage.
The route and activity detail pages SHALL render the surface and waytype breakdown as proportion bars when the row has breakdown data, coloured with the shared surface/highway palettes, each segment labelled with its category, distance, and percentage.
#### Scenario: Bars shown for a route with a breakdown
- **WHEN** a user views a route that has a stored surface breakdown
#### Scenario: Bars shown for a row with a breakdown
- **WHEN** a user views a route or activity that has a stored surface breakdown
- **THEN** a surface proportion bar and a waytype proportion bar are shown, each segment coloured by category with the largest category first
#### Scenario: Percentages reflect distance share
@ -27,7 +49,7 @@ The route detail page SHALL render the surface and waytype breakdown as proporti
- **THEN** their distance is shown as an "other" segment rather than omitted
#### Scenario: Hidden without data
- **WHEN** a route has no stored breakdown
- **WHEN** a row has no stored breakdown
- **THEN** no breakdown bars are rendered
### Requirement: Localized breakdown labels

View file

@ -1,26 +1,38 @@
## 1. Breakdown derivation (shared, pure)
## 1. Shared derivation + shape
- [ ] 1.1 Add a pure `surfaceBreakdown(coordinates, surfaces, highways)` helper (in `@trails-cool/map-core` or a planner lib): sum haversine segment lengths into per-surface and per-waytype buckets → `{ surface: Record<string, number>, highway: Record<string, number> }` (metres). Unit-test it.
- [ ] 1.2 Define the shared wire shape (Zod) for the breakdown in `@trails-cool/api` (`SurfaceBreakdownSchema`, all values ≥ 0).
- [ ] 1.1 Pure `surfaceBreakdown(coordinates, surfaces, highways)` helper: haversine-weight each segment into per-surface + per-waytype buckets → `{ surface: Record<string, number>, highway: Record<string, number> }` (metres). Unit-test.
- [ ] 1.2 `SurfaceBreakdownSchema` (Zod) in `@trails-cool/api` (record of category → metres ≥ 0, surface + highway).
## 2. Capture at save (Planner → Journal)
## 2. Schema
- [ ] 2.1 In `apps/planner/.../api.save-to-journal`, read the session `EnrichedRoute` (coordinates + surfaces + highways), compute the breakdown, and add `surfaceBreakdown` to the POST body.
- [ ] 2.2 Journal callback (`api.routes.$id.callback`) validates the optional `surfaceBreakdown` and persists it; re-save overwrites.
- [ ] 2.1 Nullable `surfaceBreakdown` jsonb on `journal.routes` **and** `journal.activities`; `pnpm db:push`.
## 3. Schema
## 3. Path 1 — synchronous (Planner routes)
- [ ] 3.1 Add nullable `surfaceBreakdown` jsonb to `journal.routes`; `pnpm db:push`.
- [ ] 3.1 `apps/planner/.../api.save-to-journal`: compute the breakdown from the session `EnrichedRoute` (coords + surfaces + highways) and add `surfaceBreakdown` to the POST body.
- [ ] 3.2 Journal route callback validates the optional field and persists it (overwrite on re-save).
## 4. Render
- [ ] 4.1 Route detail loader exposes `surfaceBreakdown`.
- [ ] 4.2 `SurfaceBreakdown` component: stacked proportion bar per dimension (surface, waytype), segments coloured via `SURFACE_COLORS`/`HIGHWAY_COLORS` (DEFAULT for unknown), legend with category · km · % (largest first), unknown → "other"; hidden when empty. Mount on `routes.$id.tsx`.
- [ ] 4.1 Route + activity detail loaders expose `surfaceBreakdown`.
- [ ] 4.2 `SurfaceBreakdown` component: stacked bar per dimension (surface, waytype), `map-core` `SURFACE_COLORS`/`HIGHWAY_COLORS` (DEFAULT → "other"), legend category · km · % (largest first), hidden when empty. Mount on `routes.$id.tsx` and `activities.$id.tsx`.
- [ ] 4.3 i18n `journal.surface.*` (label + category names + "other") en + de.
## 5. Tests & checks
## 5. Path 2 — async backfill + SSE
- [ ] 5.1 Unit: `surfaceBreakdown` (distance weighting, multiple surfaces, unknown bucket).
- [ ] 5.2 Component (jsdom): bars render with correct proportions; "other" for unknowns; hidden when empty.
- [ ] 5.3 E2E: a Planner-saved route with waytags shows the breakdown bars on its detail page (or assert via a seeded route if the planner round-trip is too heavy for e2e).
- [ ] 5.4 `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` green.
- [ ] 5.1 Journal-side Overpass client: `way[highway]` in bbox + parse (mirror the planner's `buildQuery`/`parseResponse` + rate-limit handling; route via the `/api/overpass` proxy).
- [ ] 5.2 Map-match helper: nearest OSM way per route segment → surface/highway; unmatched → unknown. Unit-test the matcher on a small fixture.
- [ ] 5.3 Typed `surface-backfill` pg-boss job (`{ kind, id }`): load geometry → skip if breakdown exists → Overpass → map-match → store. Idempotent, retry-safe, best-effort.
- [ ] 5.4 Enqueue the job after GPX import (Komoot/Garmin) and manual upload; add a one-off sweep entrypoint for pre-existing rows.
- [ ] 5.5 Emit a `surface_breakdown` SSE event (`{ kind, id }`) on completion; detail page subscribes (EventSource hook) and re-fetches the breakdown when it matches; optional "computing…" state.
## 6. Privacy
- [ ] 6.1 Document the backfill's Overpass lookup in the privacy manifest; route via the proxy (self-hostable Overpass is the long-term answer — see `docs/ideas/self-host-overpass`).
## 7. Tests & checks
- [ ] 7.1 Unit: `surfaceBreakdown` weighting; map-matcher nearest-way + unknown bucket.
- [ ] 7.2 Component (jsdom): bars/proportions, "other" bucket, hidden when empty.
- [ ] 7.3 E2E: a route/activity with a stored breakdown shows the bars (seed the breakdown to avoid a live Overpass call in CI).
- [ ] 7.4 `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` green.