openspec: revise route-surface-breakdown — add async Overpass backfill + SSE

Builds on the merged A-only proposal. Broadens the capability:
- breakdown now on routes AND activities;
- Path 1 (sync) unchanged: BRouter waytags → breakdown at Planner save;
- Path 2 (async): a `surface-backfill` pg-boss job map-matches geometry to OSM
  via Overpass for imports/uploads/older rows, stores the breakdown, and pushes
  a `surface_breakdown` SSE event so an open detail page fills in live;
- privacy note: backfill sends geometry to Overpass → route via the proxy /
  self-hostable Overpass; documented in the manifest.

Validates with `openspec validate --strict`. Specs only; no code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

openspec/changes/route-surface-breakdown/design.md

@@ -1,73 +1,90 @@
 ## Context
 
 The Planner computes accurate per-coordinate surface/highway from BRouter's
-`messages`/`WayTags` (`apps/planner/.../route-merge.ts` → `EnrichedRoute`), and
-`ColoredRoute` already renders the route coloured by surface. But this metadata
-is **session-only**: the planner→journal handoff (`api.save-to-journal`) POSTs
-only `{ gpx }` to the Journal callback (`api.routes.$id.callback`), the `routes`
-table has no surface column, and GPX carries no surface extensions. So today no
-saved route has surface data.
+`WayTags` (`apps/planner/.../route-merge.ts` → `EnrichedRoute`) and colours the
+line with it, but it's session-only: `api.save-to-journal` POSTs only `{ gpx }`,
+`routes` has no surface column, GPX carries no surface tags. The repo already
+has a typed pg-boss job system (`apps/journal/app/jobs/payloads.ts`,
+`@trails-cool/jobs`), an SSE broker (`/api/events` + EventSource hooks, e.g.
+`useUnreadNotifications`), and an Overpass proxy used for Planner POIs
+(`apps/planner/.../overpass.ts`, `/api/overpass`).
 
 ## Goals / Non-Goals
 
 **Goals**
-- Surface + waytype proportion bars on route detail, using data we already
-  compute, with no new external dependency.
+- Surface + waytype bars on route **and** activity detail.
+- Two derivation paths: free/accurate for Planner routes; async backfill for
+  everything else, with a live SSE update.
 
 **Non-Goals (v1)**
-- **Activities / imported / manually-uploaded GPX** — they have no waytags
-  (bare GPX). They show no breakdown for now (see "Data source" → future
-  Overpass).
-- **Colouring the Journal map line by surface** — needs per-segment data, not
-  just the summary; a clean follow-up once the bars land.
-- Selectable surface-vs-waytype toggle UI beyond showing both bars.
+- Colouring the Journal **map line** by surface (needs per-segment data, not the
+  summary; a clean follow-up).
+- Re-backfilling on every geometry edit (backfill is one-shot per row unless
+  re-triggered).
 
 ## Decisions
 
-### D1: Data source — persist BRouter waytags at save (Planner routes only)
-Two ways to get a breakdown onto the Journal detail page:
+### D1: Two derivation paths into one stored summary
+Both paths produce the same compact, distance-weighted summary —
+`{ surface: { asphalt: <m>, gravel: <m>, … }, highway: { … } }` (metres per
+category) — stored as nullable `surface_breakdown` jsonb on `routes` and
+`activities`. The bars need proportions, not geometry, so we store a summary,
+not per-segment data (per-segment is a future map-colouring concern).
 
-- **(A, chosen) BRouter-at-save** — the Planner already has 100%-accurate
-  per-segment surface/highway; compute the distribution there at save and
-  persist it. Accurate, cheap, no external call. **Limitation:** only
-  Planner-created routes (activities/uploads have no waytags).
-- **(B, future) Overpass map-matching** — snap arbitrary route geometry to OSM
-  ways and read `surface`/`highway` server-side. Covers imports/uploads, but
-  it's expensive, approximate, rate-limited, and an external dependency. Parked
-  for when we want imports covered (the repo already proxies Overpass for POIs).
+- **Path 1 — synchronous, Planner routes (accurate, free).** The Planner has
+  100%-accurate per-segment tags; compute the summary at save and persist via an
+  extra optional `surfaceBreakdown` field in the handoff. No external call.
+- **Path 2 — async backfill, everything else (best-effort).** For a row with
+  geometry but no summary, a job map-matches to OSM and computes it (D2).
 
-v1 ships (A). Routes without the data degrade to no bars.
+### D2: The async backfill job
+A typed `surface-backfill` pg-boss job keyed by `{ kind: "route" | "activity",
+id }`:
+1. Load the geometry; bail if absent or already has a summary.
+2. Query Overpass for `way[highway]` within the route bbox (a journal-side
+   client mirroring the Planner's `buildQuery`/`parseResponse` + rate-limit
+   handling).
+3. **Map-match**: for each route segment, find the nearest OSM way and read its
+   `surface`/`highway`; bucket the segment's length. Unmatched/untagged →
+   `unknown`.
+4. Store the summary; emit the SSE event (D3).
+- **Triggers:** enqueued after a GPX import (Komoot/Garmin) and after a manual
+  upload; plus a one-off sweep job to backfill pre-existing rows. Optionally
+  enqueued on-demand when a detail page without a summary is opened.
+- **Resilience:** pg-boss retries cover Overpass rate-limits/outages; the job is
+  idempotent (skips if a summary already exists) and best-effort (failure leaves
+  the row with no bars, not an error).
 
-### D2: Store a compact distance-weighted summary, not per-segment data
-The bars need **proportions**, not geometry. At save, sum the haversine length
-of each coordinate segment into its surface bucket (and highway bucket),
-yielding `{ surface: { asphalt: <m>, gravel: <m>, … }, highway: { … } }`
-(metres per category). This is tiny (a handful of keys), stored as a nullable
-`surface_breakdown` jsonb on `routes`. Per-segment data (for future map line
-colouring) is deliberately *not* persisted in v1 — that's a bigger payload and
-a separate feature.
-
-### D3: Handoff carries it
-`api.save-to-journal` adds `surfaceBreakdown` to the POST body alongside `gpx`;
-the Journal callback validates (Zod, all values ≥ 0) and stores it. The field
-is optional, so the handoff stays backward-compatible and non-Planner callers
-(if any) are unaffected. Re-saving a route recomputes/overwrites it.
+### D3: SSE live update
+On backfill completion the job publishes a `surface_breakdown` SSE event
+(payload: `{ kind, id }`) via the existing broker. The detail page subscribes
+(EventSource hook) and, when the event matches the row it's showing, re-fetches
+the breakdown and fills the bars — so a user who opened the page before the job
+finished sees it appear without reloading. While a backfill is known to be
+in-flight, the page may show a small "computing…" affordance.
 
 ### D4: Rendering
-A `SurfaceBreakdown` component renders one horizontal stacked bar per dimension
-(surface, then waytype): segments sized by percentage, coloured via
-`SURFACE_COLORS` / `HIGHWAY_COLORS` (`DEFAULT_*` for unknown), with a legend
-listing each category + km + percent (largest first). Unknown/unmapped tags
-collapse into an "other" segment. Hidden entirely when `surface_breakdown` is
-null or empty. Labels via i18n; distances via `stats.ts`.
+A `SurfaceBreakdown` component: one horizontal stacked bar per dimension
+(surface, then waytype), segments sized by percentage and coloured via
+`SURFACE_COLORS`/`HIGHWAY_COLORS` (`DEFAULT_*` for unknown), with a legend
+(category · km · %, largest first). Unknown/unmapped tags collapse into "other".
+Hidden when the summary is null/empty. Labels via i18n; distances via `stats.ts`.
+
+### D5: Privacy (backfill sends geometry to Overpass)
+Path 1 sends nothing externally. Path 2's Overpass query exposes a route's
+bounding box to the Overpass endpoint — a real consideration for a privacy-first
+app. Mitigations: route the backfill through the **proxied / self-hostable**
+Overpass (the parked `self-host-overpass` idea removes the third-party entirely),
+and only the bbox + geometry needed for matching is sent. We document the
+backfill's external lookup in the privacy manifest; a self-hosted Overpass is the
+clean long-term answer.
 
 ## Risks / Trade-offs
 
-- **Coverage gap** — only Planner routes get bars in v1. Acceptable: routes are
-  *designed* in the Planner, which is exactly where "how much gravel" matters.
-  Imports get bars later via (B).
-- **Tag sparsity** — OSM `surface` is often missing; those segments bucket as
-  `unknown`/"other". We show it honestly rather than guessing.
-- **Stale on edit** — editing a route's geometry elsewhere without re-routing
-  could leave the breakdown stale; recomputing on every Planner save keeps it
-  fresh for the normal flow.
+- **Map-matching is approximate** — nearest-way snapping mismatches at junctions
+  / parallel ways; acceptable for a proportional overview, and Path 1 (exact) is
+  preferred whenever available.
+- **Overpass cost/limits** — bounded by the async queue + retries; the sweep
+  should throttle. A self-hosted Overpass removes the ceiling.
+- **Two code paths** — both converge on one stored summary + one renderer, so
+  only the *derivation* differs.

openspec/changes/route-surface-breakdown/proposal.md

@@ -1,46 +1,57 @@
 ## Why
 
-"How much of this is gravel?" is a top-three question for anyone choosing a
-route — Komoot's surface/waytype breakdown bars are one of its most-used
-features, and `@trails-cool/map-core` already ships the surface/highway colour
-palettes. We also already *compute* accurate per-segment surface and highway
-tags from BRouter while planning (`route-merge` → `EnrichedRoute.surfaces[]`),
-then **throw them away on save**. This change captures that data and shows it.
+"How much of this is gravel?" is a top-three question for choosing a route, and
+Komoot's surface/waytype bars are one of its most-used features.
+`@trails-cool/map-core` already ships the surface/highway colour palettes, and
+the Planner already *computes* accurate per-segment surface/highway from BRouter
+(`route-merge` → `EnrichedRoute.surfaces[]`) — then **discards it on save**.
+This change captures that data, shows it as proportion bars, and **backfills
+everything else** (imports, uploads, older rows) asynchronously so the bars
+aren't limited to Planner routes.
 
 ## What Changes
 
-- On **route detail**, show surface and waytype **proportion bars** (e.g.
-  "62% asphalt · 28% gravel · 10% path"), coloured with the `map-core`
-  `SURFACE_COLORS` / `HIGHWAY_COLORS` palettes, with per-segment km/percent.
-- **Persist the breakdown for Planner-created routes**: compute a
-  distance-weighted surface + waytype distribution from the BRouter waytags at
-  save time, send it in the planner→journal handoff, and store it on the route.
-- Degrade gracefully: routes without breakdown data (older routes, or activities
-  / manually-imported GPX that never had waytags) simply don't show the bars.
+- On **route and activity detail**, show surface and waytype **proportion bars**
+  (e.g. "62% asphalt · 28% gravel · 10% path"), coloured with the `map-core`
+  `SURFACE_COLORS` / `HIGHWAY_COLORS` palettes, with per-category km/percent.
+- **Path 1 — synchronous (Planner routes):** compute a distance-weighted
+  surface + waytype distribution from the BRouter waytags at save time, send it
+  in the planner→journal handoff, store it. Accurate, free, no external call.
+- **Path 2 — async backfill (everything else):** for a route/activity that has
+  geometry but no breakdown (imports, manual uploads, pre-existing rows), a
+  background job map-matches the geometry to OSM ways via Overpass, computes the
+  breakdown, stores it, and **pushes an SSE event** so an open detail page fills
+  the bars in live.
+- Degrade gracefully: no geometry / not-yet-backfilled → no bars (or a
+  "computing…" state while a backfill is in flight).
 
 ## Capabilities
 
 ### New Capabilities
 
-- `route-surface-breakdown`: the surface/waytype proportion bars on route
-  detail, how the distribution is derived from BRouter waytags, persisted at
-  save, and rendered.
+- `route-surface-breakdown`: the surface/waytype proportion bars on route &
+  activity detail — derived synchronously from BRouter waytags (Planner) or
+  asynchronously via an Overpass backfill job with an SSE live-update.
 
 ### Modified Capabilities
 
-<!-- `planner-journal-handoff` gains one optional field in its save payload;
-     noted in design, but the breakdown is a new capability, not a change to the
-     handoff's existing requirements. -->
+<!-- `planner-journal-handoff` gains one optional payload field; `background-jobs`
+     and `sse-broker` gain a new job + event, but those are additive uses of
+     existing transports, owned by this new capability. -->
 
 ## Impact
 
-- **Schema**: a nullable `surface_breakdown` jsonb column on `journal.routes`
-  (additive — `pnpm db:push`, no data migration).
-- **Planner**: `api.save-to-journal` computes the distance-weighted breakdown
-  from the session `EnrichedRoute` (surfaces/highways + coordinates) and adds it
-  to the POST body.
-- **Journal**: the route callback (`api.routes.$id.callback`) accepts + stores
-  it; the route detail loader exposes it; a `SurfaceBreakdown` component renders
-  the bars (reusing `map-core` palettes + `stats.ts` distance formatting).
-- **No new external dependency** — BRouter already returns the tags; no Overpass
-  call. Imported activities/uploads are out of scope for v1 (see design).
+- **Schema**: nullable `surface_breakdown` jsonb on `journal.routes` **and**
+  `journal.activities` (additive — `pnpm db:push`).
+- **Planner**: `api.save-to-journal` computes + sends the breakdown.
+- **Journal**:
+  - route callback stores the synchronous breakdown;
+  - a typed `surface-backfill` pg-boss job (Overpass map-match → breakdown →
+    store → SSE), enqueued after imports/uploads and available as a sweep;
+  - a journal-side Overpass way/surface client (reusing the existing
+    `/api/overpass` proxy pattern);
+  - an SSE `surface_breakdown` event + a detail-page subscription;
+  - a `SurfaceBreakdown` component (reuses `map-core` palettes + `stats.ts`).
+- **i18n**: `journal.surface.*` (en + de).
+- **Privacy**: the backfill sends route geometry to Overpass — see design for
+  the proxy/self-host + visibility considerations.

openspec/changes/route-surface-breakdown/specs/route-surface-breakdown/spec.md

@@ -1,21 +1,43 @@
 ## ADDED Requirements
 
 ### Requirement: Surface breakdown derived from BRouter waytags
-The system SHALL compute a distance-weighted surface and waytype distribution for a Planner-created route from the per-segment BRouter waytags at save time, and persist it with the route.
+The system SHALL compute a distance-weighted surface and waytype distribution for a Planner-created route from its per-segment BRouter waytags at save time, and persist it with the route.
 
 #### Scenario: Breakdown captured on save
 - **WHEN** a route is saved from the Planner with per-segment surface/highway waytags
 - **THEN** the route stores a distance-weighted breakdown of metres per surface category and per waytype category
 
 #### Scenario: No waytags available
-- **WHEN** a route is saved without waytags (no surface data)
-- **THEN** no breakdown is stored, and the route detail shows no breakdown bars
+- **WHEN** a route is saved without waytags
+- **THEN** no breakdown is stored synchronously (an async backfill may fill it later)
+
+### Requirement: Async surface backfill via Overpass
+The system SHALL asynchronously derive a surface breakdown for a route or activity that has geometry but no stored breakdown, by map-matching its geometry to OpenStreetMap ways via Overpass in a background job, and SHALL store the resulting distance-weighted breakdown.
+
+#### Scenario: Imported activity is backfilled
+- **WHEN** an activity is imported (e.g. from Komoot) with geometry but no waytags
+- **THEN** a backfill job map-matches its geometry to OSM ways and stores a surface/waytype breakdown
+
+#### Scenario: Backfill is best-effort and idempotent
+- **WHEN** a backfill job runs for a row that already has a breakdown
+- **THEN** it does no work; and **WHEN** Overpass is unavailable, the row is left without a breakdown rather than erroring
+
+#### Scenario: Untagged segments
+- **WHEN** map-matched ways have no surface tag
+- **THEN** those segments' distance is bucketed as unknown
+
+### Requirement: Live update when a backfill completes
+The detail page SHALL update its breakdown without a reload when a backfill for the row being viewed completes, via a server-sent event.
+
+#### Scenario: Bars appear after backfill
+- **WHEN** a user is viewing a route/activity detail page whose backfill finishes
+- **THEN** the page receives a `surface_breakdown` event for that row and renders the bars without a manual reload
 
 ### Requirement: Surface and waytype proportion bars
-The route detail page SHALL render the surface and waytype breakdown as proportion bars when the route has breakdown data, coloured with the shared surface/highway palettes, each segment labelled with its category, distance, and percentage.
+The route and activity detail pages SHALL render the surface and waytype breakdown as proportion bars when the row has breakdown data, coloured with the shared surface/highway palettes, each segment labelled with its category, distance, and percentage.
 
-#### Scenario: Bars shown for a route with a breakdown
-- **WHEN** a user views a route that has a stored surface breakdown
+#### Scenario: Bars shown for a row with a breakdown
+- **WHEN** a user views a route or activity that has a stored surface breakdown
 - **THEN** a surface proportion bar and a waytype proportion bar are shown, each segment coloured by category with the largest category first
 
 #### Scenario: Percentages reflect distance share
@@ -27,7 +49,7 @@ The route detail page SHALL render the surface and waytype breakdown as proporti
 - **THEN** their distance is shown as an "other" segment rather than omitted
 
 #### Scenario: Hidden without data
-- **WHEN** a route has no stored breakdown
+- **WHEN** a row has no stored breakdown
 - **THEN** no breakdown bars are rendered
 
 ### Requirement: Localized breakdown labels

openspec/changes/route-surface-breakdown/tasks.md

@@ -1,26 +1,38 @@
-## 1. Breakdown derivation (shared, pure)
+## 1. Shared derivation + shape
 
-- [ ] 1.1 Add a pure `surfaceBreakdown(coordinates, surfaces, highways)` helper (in `@trails-cool/map-core` or a planner lib): sum haversine segment lengths into per-surface and per-waytype buckets → `{ surface: Record<string, number>, highway: Record<string, number> }` (metres). Unit-test it.
-- [ ] 1.2 Define the shared wire shape (Zod) for the breakdown in `@trails-cool/api` (`SurfaceBreakdownSchema`, all values ≥ 0).
+- [ ] 1.1 Pure `surfaceBreakdown(coordinates, surfaces, highways)` helper: haversine-weight each segment into per-surface + per-waytype buckets → `{ surface: Record<string, number>, highway: Record<string, number> }` (metres). Unit-test.
+- [ ] 1.2 `SurfaceBreakdownSchema` (Zod) in `@trails-cool/api` (record of category → metres ≥ 0, surface + highway).
 
-## 2. Capture at save (Planner → Journal)
+## 2. Schema
 
-- [ ] 2.1 In `apps/planner/.../api.save-to-journal`, read the session `EnrichedRoute` (coordinates + surfaces + highways), compute the breakdown, and add `surfaceBreakdown` to the POST body.
-- [ ] 2.2 Journal callback (`api.routes.$id.callback`) validates the optional `surfaceBreakdown` and persists it; re-save overwrites.
+- [ ] 2.1 Nullable `surfaceBreakdown` jsonb on `journal.routes` **and** `journal.activities`; `pnpm db:push`.
 
-## 3. Schema
+## 3. Path 1 — synchronous (Planner routes)
 
-- [ ] 3.1 Add nullable `surfaceBreakdown` jsonb to `journal.routes`; `pnpm db:push`.
+- [ ] 3.1 `apps/planner/.../api.save-to-journal`: compute the breakdown from the session `EnrichedRoute` (coords + surfaces + highways) and add `surfaceBreakdown` to the POST body.
+- [ ] 3.2 Journal route callback validates the optional field and persists it (overwrite on re-save).
 
 ## 4. Render
 
-- [ ] 4.1 Route detail loader exposes `surfaceBreakdown`.
-- [ ] 4.2 `SurfaceBreakdown` component: stacked proportion bar per dimension (surface, waytype), segments coloured via `SURFACE_COLORS`/`HIGHWAY_COLORS` (DEFAULT for unknown), legend with category · km · % (largest first), unknown → "other"; hidden when empty. Mount on `routes.$id.tsx`.
+- [ ] 4.1 Route + activity detail loaders expose `surfaceBreakdown`.
+- [ ] 4.2 `SurfaceBreakdown` component: stacked bar per dimension (surface, waytype), `map-core` `SURFACE_COLORS`/`HIGHWAY_COLORS` (DEFAULT → "other"), legend category · km · % (largest first), hidden when empty. Mount on `routes.$id.tsx` and `activities.$id.tsx`.
 - [ ] 4.3 i18n `journal.surface.*` (label + category names + "other") en + de.
 
-## 5. Tests & checks
+## 5. Path 2 — async backfill + SSE
 
-- [ ] 5.1 Unit: `surfaceBreakdown` (distance weighting, multiple surfaces, unknown bucket).
-- [ ] 5.2 Component (jsdom): bars render with correct proportions; "other" for unknowns; hidden when empty.
-- [ ] 5.3 E2E: a Planner-saved route with waytags shows the breakdown bars on its detail page (or assert via a seeded route if the planner round-trip is too heavy for e2e).
-- [ ] 5.4 `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` green.
+- [ ] 5.1 Journal-side Overpass client: `way[highway]` in bbox + parse (mirror the planner's `buildQuery`/`parseResponse` + rate-limit handling; route via the `/api/overpass` proxy).
+- [ ] 5.2 Map-match helper: nearest OSM way per route segment → surface/highway; unmatched → unknown. Unit-test the matcher on a small fixture.
+- [ ] 5.3 Typed `surface-backfill` pg-boss job (`{ kind, id }`): load geometry → skip if breakdown exists → Overpass → map-match → store. Idempotent, retry-safe, best-effort.
+- [ ] 5.4 Enqueue the job after GPX import (Komoot/Garmin) and manual upload; add a one-off sweep entrypoint for pre-existing rows.
+- [ ] 5.5 Emit a `surface_breakdown` SSE event (`{ kind, id }`) on completion; detail page subscribes (EventSource hook) and re-fetches the breakdown when it matches; optional "computing…" state.
+
+## 6. Privacy
+
+- [ ] 6.1 Document the backfill's Overpass lookup in the privacy manifest; route via the proxy (self-hostable Overpass is the long-term answer — see `docs/ideas/self-host-overpass`).
+
+## 7. Tests & checks
+
+- [ ] 7.1 Unit: `surfaceBreakdown` weighting; map-matcher nearest-way + unknown bucket.
+- [ ] 7.2 Component (jsdom): bars/proportions, "other" bucket, hidden when empty.
+- [ ] 7.3 E2E: a route/activity with a stored breakdown shows the bars (seed the breakdown to avoid a live Overpass call in CI).
+- [ ] 7.4 `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` green.