Merge pull request #396 from trails-cool/stigi/komoot-import-two-mode-spec

Komoot import: add public mode via bio verification
This commit is contained in:
Ullrich Schäfer 2026-05-23 10:00:21 +02:00 committed by GitHub
commit b63fd1a303
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 155 additions and 125 deletions

View file

@ -1,9 +1,10 @@
## Context
trails.cool Journal supports manual route creation and GPX import. Users coming
from Komoot have hundreds of tours they'd lose by switching. The old trails
project had a working Komoot integration using basic auth against Komoot's
undocumented API (`api.komoot.de`).
from Komoot have hundreds of tours they'd lose by switching. The Komoot public
API (`api.komoot.de/v007`) exposes public tours and user profiles without
authentication. Profile fields (`content_text`, `content_link`) are readable
unauthenticated after a short cache delay (~minutes).
> **Note (added 2026-05-08, post `deepen-connected-services`)**:
> Earlier drafts of this design proposed a separate `journal.integrations`
@ -13,87 +14,87 @@ undocumented API (`api.komoot.de`).
> revisited, Komoot must implement:
>
> - A row in `journal.connected_services` with `credential_kind = 'web-login'`
> and a `credentials` JSONB blob carrying `{ email, encrypted_password,
> session_jar }`.
> - A `web-login` `CredentialAdapter` at
> `apps/journal/app/lib/connected-services/credential-adapters/web-login.ts`
> implementing `relogin(creds) → creds | InvalidCredentials`. Web-login
> breakage (form changes, captcha, password rotation) surfaces at the
> import layer, not the credential layer (see ADR-0001 / CONTEXT.md).
> (authenticated mode) or `credential_kind = 'public'` (public mode).
> - A `KomootImporter` in
> `apps/journal/app/lib/connected-services/providers/komoot/importer.ts`
> that goes through `ctx.withFreshCredentials` like the Wahoo importer.
> Komoot does not have webhooks or push, so its manifest declares only
> the `Importer` capability — no `routePusher`, no `webhookReceiver`.
> that branches on credential kind.
> - A manifest at `providers/komoot/manifest.ts` registered via
> `providers/index.ts`.
>
> Don't add a `journal.integrations` table. The user-facing "Connected
> Services" list at `/settings/connections` should show Komoot alongside
> Wahoo, which only works if both share `connected_services`.
> Services" list at `/settings/connections` should show Komoot alongside Wahoo.
## Goals / Non-Goals
**Goals:**
- Connect Komoot account via email + password
- Public import: verify Komoot profile ownership via bio field, import public tours, store no passwords
- Authenticated import: connect via email + password, import all tours including private
- Import all tours (paginated) as activities + routes
- Track import progress with batch status
- Deduplicate on re-import (same tour never imported twice)
- Fetch GPX geometry per tour (not just metadata)
- Cross-link: store Komoot username on the connection so the Journal profile can show "also on Komoot"
**Non-Goals:**
- OAuth flow (Komoot has no public OAuth — basic auth only)
- OAuth flow
- Real-time sync or webhook-based updates
- Strava/other providers (future iteration, but design the schema generically)
- Background job queue like Inngest (keep it simple — synchronous import with progress)
- Other providers (future iteration)
## Import Modes
### Public mode
No credentials stored. Ownership is verified by checking that the user's
trails.cool profile URL appears in their Komoot `content_text` (bio/"Über dich")
field, which is readable via the unauthenticated public API.
Verification flow:
1. User enters their Komoot profile URL (e.g. `komoot.com/user/27595800585`)
2. trails.cool shows: *"Add your trails.cool profile link (`https://trails.cool/users/ullrich`) to your Komoot bio ('Über dich'), then click Verify"*
3. trails.cool fetches `api.komoot.de/v007/users/{id}/` and checks `content_text` contains their trails.cool profile URL
4. On success: store Komoot username in `connected_services` with `credential_kind = 'public'`, no password
5. Fetch `api.komoot.de/v007/users/{id}/tours/?status=public` (paginated), import
### Authenticated mode
User provides email + password. Credentials validated and stored AES-256-GCM
encrypted. All tours (public and private) imported.
## Decisions
### D1: Generic integrations table with provider column
### D1: Two connection modes in `connected_services`
Store connections in a `journal.integrations` table with a `provider` enum
(`komoot`, and later `strava` etc). Credentials encrypted at rest. This avoids
a separate table per provider.
A `mode` column (`'public' | 'authenticated'`) on the connection row controls
which import path runs. Public mode rows have no `encryptedCredentials`.
### D2: Import batches for progress tracking
### D2: Bio field for ownership verification
Each import creates an `import_batches` row tracking: status (running,
completed, failed), total found, imported count, duplicate count, error message.
The UI polls this for progress.
`api.komoot.de/v007/users/{id}/` returns `content_text` (bio) unauthenticated.
Checking for the user's own trails.cool profile URL proves they control the
Komoot account without any credential exchange. The profile link stays in the
bio permanently, providing cross-platform discovery.
### D3: Deduplication via composite key
### D3: Import batches for progress tracking
Activities get a `dedupeKey` column. For Komoot: `komoot:{tourId}`. Combined
with `ownerId`, a unique constraint prevents duplicates. Insert uses
`onConflictDoNothing`.
Each import creates a batch row tracking: status, total found, imported count,
duplicate count, error message. UI polls for live progress.
### D4: Synchronous import with streaming progress
### D4: Deduplication via dedupe key
No background job queue. The import runs in an API route action that:
1. Fetches all tour pages from Komoot
2. For each tour, fetches GPX and creates activity + route
3. Updates the batch row with progress
4. Client polls `/api/integrations/komoot/import-status` for live updates
Activities get a `dedupeKey` of `komoot:{tourId}`. Unique constraint on
`(ownerId, dedupeKey)` prevents duplicates on re-import.
This keeps the architecture simple. If imports are too slow (>100 tours), we
can add a background worker later.
### D5: AES-256-GCM for credential encryption (authenticated mode only)
### D5: Encrypt credentials with AES-256-GCM
Use Node's `crypto.createCipheriv` with a server-side key derived from
`INTEGRATION_SECRET` env var. Decrypt on use, never log plaintext.
**Alternative considered**: Store only the API token (not email+password).
Rejected because the token may expire and re-auth requires the original
credentials.
`INTEGRATION_SECRET` env var → scrypt-derived key. Decrypt on use, never log.
## Risks / Trade-offs
- **Komoot API is undocumented** → Could break without notice. Mitigation: wrap
all API calls in error handling, mark integration as "needs reauth" on 401.
- **Storing user passwords for third-party service** → Security risk.
Mitigation: AES-256-GCM encryption, separate `INTEGRATION_SECRET`, document
in privacy manifest.
- **Synchronous import may timeout for large accounts** → Mitigation: paginate
and commit per-page. If a page fails, the batch is marked partial and can be
resumed.
- **Komoot API is undocumented** → Could change without notice. All calls
wrapped in error handling; connection marked as needing reauth on 401.
- **Public mode: only public tours importable** → Expected and documented.
Users who want private tours use authenticated mode.
- **Bio field cache delay** → Verification shows a clear "allow a few minutes
for changes to propagate" message and a retry button.
- **Authenticated mode: storing third-party passwords** → AES-256-GCM,
separate `INTEGRATION_SECRET`, documented in privacy manifest.

View file

@ -2,31 +2,41 @@
Users switching to trails.cool have years of tours on Komoot. Without import,
they start with an empty Journal — no history, no motivation to switch. A
one-click Komoot import lets users bring their existing tours and start using
Komoot import lets users bring their existing tours and start using
trails.cool immediately.
## What Changes
- Add Komoot account connection flow (email + password → API credentials)
- Import Komoot tours as Journal activities with route data
- Batch import with progress tracking, deduplication, and error handling
- Integration settings page to manage connected accounts and trigger imports
- Fetch full tour GPX (not just metadata) so imported routes have geometry
- **Public import** (no credentials): user proves ownership of their Komoot
profile by placing their trails.cool profile URL in their Komoot bio field.
trails.cool verifies via the public Komoot API, then imports all public tours.
- **Authenticated import** (email + password): imports all tours including
private ones. Credentials stored encrypted.
- Both modes share the same batch import engine, progress tracking, and
deduplication logic.
## Capabilities
### New Capabilities
- `komoot-import`: Connect a Komoot account, import tours as activities with routes, track import progress, deduplicate on re-import
- `komoot-import`: Two-mode Komoot import — public (bio verification, no
credential storage) and authenticated (email + password, all tours including
private). Both track batch progress and deduplicate on re-import.
### Modified Capabilities
- `route-management`: Routes can now be created via import (not just manual creation), with an external source tracking field
- `route-management`: Routes can now be created via import (not just manual
creation), with an external source tracking field.
## Impact
- **Database**: New tables for integration connections and import batches in `journal` schema
- **Files**: New routes (`/integrations`, `/api/integrations/*`), new server utilities for Komoot API, new DB schema tables
- **Dependencies**: No new packages needed — uses fetch for Komoot API, existing Drizzle for DB
- **Privacy**: Komoot credentials stored encrypted. Must be documented in privacy manifest.
- **External**: Komoot API (api.komoot.de) — basic auth, no official developer program
- **Database**: New tables for integration connections and import batches in
`journal` schema. Connections can be credential-free (public mode).
- **Files**: New routes (`/integrations`, `/api/integrations/*`), new server
utilities for Komoot API, new DB schema tables
- **Dependencies**: No new packages — uses fetch for Komoot API, existing
Drizzle for DB
- **Privacy**: Only authenticated mode stores credentials (encrypted). Public
mode stores only the Komoot username. Both documented in privacy manifest.
- **External**: Komoot public API (`api.komoot.de`) — unauthenticated for
public tours and profile lookup; basic auth for authenticated mode

View file

@ -1,40 +1,68 @@
## ADDED Requirements
### Requirement: Connect Komoot account
Users SHALL be able to connect their Komoot account by providing email and password. Credentials are validated against the Komoot API and stored encrypted.
### Requirement: Connect Komoot account — public mode
Users SHALL be able to connect their Komoot account without providing credentials
by verifying ownership via their Komoot bio field.
#### Scenario: Successful connection
- **WHEN** user enters valid Komoot email and password on the integrations page
- **THEN** the system validates credentials via `api.komoot.de`, stores them encrypted, and shows the connection as active
#### Scenario: Initiate public connection
- **WHEN** a user enters their Komoot profile URL on the integrations page
- **THEN** the system displays their trails.cool profile URL and instructs them to add it to their Komoot bio ("Über dich" field)
#### Scenario: Verify public ownership
- **WHEN** the user clicks Verify after updating their Komoot bio
- **THEN** the system fetches the public Komoot profile, checks that `content_text` contains the user's trails.cool profile URL, and on success marks the connection as active with mode "public"
#### Scenario: Bio not yet updated
- **WHEN** verification is attempted but the trails.cool URL is not found in the Komoot bio
- **THEN** the system shows an error explaining that changes may take a few minutes to propagate, with a retry button
#### Scenario: Disconnect public account
- **WHEN** a user disconnects their public Komoot integration
- **THEN** the stored Komoot username is removed and no further imports can occur
### Requirement: Connect Komoot account — authenticated mode
Users SHALL be able to connect their Komoot account by providing email and
password to access all tours including private ones.
#### Scenario: Successful authenticated connection
- **WHEN** a user enters valid Komoot email and password
- **THEN** the system validates credentials via the Komoot API, stores them encrypted, and shows the connection as active with mode "authenticated"
#### Scenario: Invalid credentials
- **WHEN** user enters incorrect Komoot credentials
- **THEN** the system shows an error message and does not store credentials
- **WHEN** a user enters incorrect Komoot credentials
- **THEN** the system shows an error and does not store credentials
#### Scenario: Disconnect account
- **WHEN** user disconnects their Komoot integration
#### Scenario: Disconnect authenticated account
- **WHEN** a user disconnects their authenticated Komoot integration
- **THEN** stored credentials are deleted and no further imports can occur
### Requirement: Import Komoot tours
Users SHALL be able to import all their Komoot tours as Journal activities with linked routes.
Users SHALL be able to import their Komoot tours as Journal activities with
linked routes. Public mode imports public tours only; authenticated mode imports
all tours.
#### Scenario: Import all tours
- **WHEN** user clicks "Import" on the integrations page
- **THEN** the system fetches all tours (paginated), creates an activity and route for each, and shows progress
#### Scenario: Import public tours
- **WHEN** a user with a public-mode connection triggers import
- **THEN** the system fetches all public tours from the Komoot API without credentials, creates an activity and route for each, and shows progress
#### Scenario: Import all tours (authenticated)
- **WHEN** a user with an authenticated-mode connection triggers import
- **THEN** the system fetches all tours (public and private, paginated) using stored credentials, creates an activity and route for each, and shows progress
#### Scenario: Tour with GPX geometry
- **WHEN** a Komoot tour is imported
- **THEN** the system fetches the tour's GPX/geometry and creates a route with the full track data
- **THEN** the system fetches the tour's GPX and creates a route with the full track data
#### Scenario: Import progress
- **WHEN** an import is running
- **THEN** the UI shows: total tours found, imported so far, duplicates skipped, and current status
### Requirement: Deduplication
The system SHALL NOT create duplicate activities when re-importing tours that were previously imported.
The system SHALL NOT create duplicate activities when re-importing tours that
were previously imported.
#### Scenario: Re-import skips existing tours
- **WHEN** user runs import and a tour was already imported previously
- **WHEN** a user runs import and a tour was already imported previously
- **THEN** the tour is skipped and counted as a duplicate in the import summary
### Requirement: Import batch tracking
@ -42,17 +70,17 @@ Each import run SHALL be tracked as a batch with status and statistics.
#### Scenario: Batch lifecycle
- **WHEN** an import starts
- **THEN** a batch record is created with status "running", and updated to "completed" or "failed" when done
- **THEN** a batch record is created with status "running", updated to "completed" or "failed" when done
#### Scenario: Batch statistics
- **WHEN** an import completes
- **THEN** the batch shows: totalFound, importedCount, duplicateCount, and duration
### Requirement: Credential encryption
### Requirement: Credential encryption (authenticated mode)
Komoot credentials SHALL be encrypted at rest using AES-256-GCM.
#### Scenario: Credentials stored securely
- **WHEN** a user connects their Komoot account
- **WHEN** a user connects via authenticated mode
- **THEN** the password is encrypted before storage and only decrypted when making API calls
### Requirement: Privacy disclosure
@ -60,4 +88,4 @@ The Komoot integration SHALL be documented in the privacy manifest.
#### Scenario: Privacy manifest updated
- **WHEN** Komoot import is available
- **THEN** the /privacy page documents: what credentials are stored, what data is imported, and that credentials are encrypted
- **THEN** the /privacy page documents: that public mode stores only the Komoot username, that authenticated mode stores an encrypted password, and what tour data is imported

View file

@ -1,51 +1,42 @@
## 1. Database Schema
## 1. Komoot API Client
- [ ] 1.1 Add `journal.integrations` table (id, userId, provider enum, encryptedCredentials, apiUsername, status, lastSyncedAt, createdAt)
- [ ] 1.2 Add `journal.import_batches` table (id, userId, integrationId, status enum, totalFound, importedCount, duplicateCount, errorMessage, startedAt, completedAt)
- [ ] 1.3 Add `dedupeKey` column to `journal.activities` table with unique constraint on (ownerId, dedupeKey)
- [ ] 1.4 Add `source` column to `journal.routes` table (nullable, e.g. "komoot", "manual", "gpx-upload")
- [ ] 1.5 Run `pnpm db:push` and verify schema locally
- [ ] 1.1 Add `fetchPublicProfile(komootUserId)` to `komoot.server.ts` — unauthenticated GET of `api.komoot.de/v007/users/{id}/`, returns `{ displayName, contentText, contentLink }`
- [ ] 1.2 Add `fetchPublicTours(komootUserId, page)` to `komoot.server.ts` — unauthenticated GET with `?status=public&limit=50`
- [ ] 1.3 Add `fetchPublicTourGpx(tourId)` — unauthenticated GPX fetch for public tours
- [ ] 1.4 Update `KomootCredentials` type to be a discriminated union: `{ mode: 'public'; username: string } | { mode: 'authenticated'; email: string; password: string; username: string; token: string }`
- [ ] 1.5 Update `fetchTours` / `fetchTourGpx` to branch on credential mode (use auth header only in authenticated mode)
- [ ] 1.6 Add unit tests for `fetchPublicProfile` response parsing and bio verification logic
## 2. Credential Encryption
## 2. Verification Logic
- [ ] 2.1 Create `apps/journal/app/lib/crypto.server.ts` with AES-256-GCM encrypt/decrypt using `INTEGRATION_SECRET` env var
- [ ] 2.2 Write unit tests for encrypt/decrypt roundtrip
- [ ] 2.1 Add `verifyKomootOwnership(komootUserId, trailsProfileUrl)` in `komoot.server.ts` — fetches public profile, checks `content_text` contains the trails.cool URL (case-insensitive, trimmed)
- [ ] 2.2 Write unit tests for verification: match, no match, null bio, trailing slash variations
## 3. Komoot API Client
## 3. Database Schema
- [ ] 3.1 Create `apps/journal/app/lib/komoot.server.ts` with login function (email + password → username + token)
- [ ] 3.2 Add fetchTours function (paginated, fetches all pages)
- [ ] 3.3 Add fetchTourGpx function (fetch GPX geometry for a single tour)
- [ ] 3.4 Write unit tests for API response parsing (mock fetch)
- [ ] 3.1 Add `mode` column (`'public' | 'authenticated'`) to `journal.sync_connections` (or connected_services table per the connected-services architecture)
- [ ] 3.2 Make `encryptedCredentials` nullable (public mode has none)
- [ ] 3.3 Run `pnpm db:push` and verify schema locally
## 4. Import Logic
## 4. API Routes
- [ ] 4.1 Create `apps/journal/app/lib/import.server.ts` with importKomootTours function that: creates batch, pages through tours, fetches GPX, creates activities + routes, deduplicates, updates batch
- [ ] 4.2 Write integration test for import with mock Komoot responses
- [ ] 4.1 Create `POST /api/integrations/komoot/verify` — accepts `{ komootProfileUrl }`, calls `verifyKomootOwnership`, on success stores public connection (no credentials)
- [ ] 4.2 Update `POST /api/integrations/komoot/connect` — now handles authenticated mode only; accepts `{ email, password }`
- [ ] 4.3 Update `POST /api/integrations/komoot/import` — branches on connection mode to use public or authenticated fetch path
- [ ] 4.4 Register new verify route in `apps/journal/app/routes.ts`
## 5. API Routes
## 5. Import Logic
- [ ] 5.1 Create `POST /api/integrations/komoot/connect` — validate credentials, store encrypted
- [ ] 5.2 Create `POST /api/integrations/komoot/disconnect` — delete credentials
- [ ] 5.3 Create `POST /api/integrations/komoot/import` — trigger import, return batch ID
- [ ] 5.4 Create `GET /api/integrations/komoot/import-status` — return current batch progress
- [ ] 5.1 Update `importKomootTours` in `komoot-import.server.ts` to accept the discriminated union credential type and route to public or authenticated fetch functions accordingly
- [ ] 5.2 Update `komootImportJob` background job handler to reconstruct credentials as `{ mode: 'public', username }` when `encryptedCredentials` is null
## 6. UI
- [ ] 6.1 Create `/integrations` route with Komoot connection form (email + password)
- [ ] 6.2 Show connected status, last sync time, and import button when connected
- [ ] 6.3 Import progress UI — poll import-status, show counts (found, imported, duplicated)
- [ ] 6.4 Add link to integrations page from Journal navigation
- [ ] 6.5 Add i18n keys for all integration strings (en + de)
- [ ] 6.1 Add public import section to `/integrations` page above the authenticated form: input for Komoot profile URL, instructions showing the user's trails.cool profile URL to copy, Verify button
- [ ] 6.2 Show verification state: pending instructions → verifying spinner → success (connected, public) or error with retry
- [ ] 6.3 Show mode badge ("Public tours only" vs "All tours") on the connected state UI
- [ ] 6.4 Add i18n keys for all new public-mode strings (en + de)
## 7. Privacy & Config
## 7. Privacy
- [ ] 7.1 Update /privacy page to document Komoot integration (credentials stored encrypted, what data is imported)
- [ ] 7.2 Add `INTEGRATION_SECRET` env var to docker-compose.yml and CI
- [ ] 7.3 Add `INTEGRATION_SECRET` to deploy secrets documentation
## 8. Verify
- [ ] 8.1 Test full flow locally: connect Komoot → import tours → verify activities + routes created
- [ ] 8.2 Verify deduplication: re-import and confirm no duplicates
- [ ] 8.3 Verify disconnect removes credentials
- [ ] 7.1 Update `/privacy` page to document both modes: public (stores Komoot username only) and authenticated (stores encrypted password)