Merge branch 'main' into dependabot/npm_and_yarn/production-6f66162dcb
This commit is contained in:
commit
84cf0490d2
25 changed files with 231 additions and 12 deletions
|
|
@ -41,5 +41,5 @@
|
||||||
## 7. Rollout
|
## 7. Rollout
|
||||||
|
|
||||||
- [x] 7.1 Merge + deploy — nothing changes because default persona equals current behaviour
|
- [x] 7.1 Merge + deploy — nothing changes because default persona equals current behaviour
|
||||||
- [ ] 7.2 Validate on prod: `/users/bruno` unchanged, synthetic content cadence unchanged, metrics unchanged
|
- [x] 7.2 Validate on prod: `/users/bruno` unchanged, synthetic content cadence unchanged, metrics unchanged
|
||||||
- [ ] 7.3 (Optional demo) On a staging or second instance, ship a non-Bruno persona via `DEMO_BOT_PERSONA=file:...` and confirm the demo user renders with the new identity + voice
|
- [x] 7.3 (Optional demo) On a staging or second instance, ship a non-Bruno persona via `DEMO_BOT_PERSONA=file:...` and confirm the demo user renders with the new identity + voice
|
||||||
|
|
@ -68,6 +68,6 @@
|
||||||
## 11. Rollout
|
## 11. Rollout
|
||||||
|
|
||||||
- [x] 11.1 Merge schema + code; deploy — the bot stays disabled everywhere because no env flag is set
|
- [x] 11.1 Merge schema + code; deploy — the bot stays disabled everywhere because no env flag is set
|
||||||
- [ ] 11.2 Flip `DEMO_BOT_ENABLED=true` on prod via the SOPS env file + redeploy
|
- [x] 11.2 Flip `DEMO_BOT_ENABLED=true` on prod via the SOPS env file + redeploy
|
||||||
- [ ] 11.3 On next worker start: verify `ensureDemoUser` created the `bruno` user, the backfill produced 3–5 items, and `/users/bruno` renders them publicly
|
- [x] 11.3 On next worker start: verify `ensureDemoUser` created the `bruno` user, the backfill produced 3–5 items, and `/users/bruno` renders them publicly
|
||||||
- [ ] 11.4 After 24 h: verify cadence looks right and the prune job ran without deleting anything yet
|
- [x] 11.4 After 24 h: verify cadence looks right and the prune job ran without deleting anything yet
|
||||||
|
|
@ -16,7 +16,7 @@
|
||||||
|
|
||||||
- [x] 3.1 Create `apps/planner/app/jobs/expire-sessions.ts` — job handler that calls `expireSessions(7)` and returns the count
|
- [x] 3.1 Create `apps/planner/app/jobs/expire-sessions.ts` — job handler that calls `expireSessions(7)` and returns the count
|
||||||
- [x] 3.2 Register the job in planner's `server.ts` with cron `0 * * * *` (hourly), retryLimit 2, expireInSeconds 60
|
- [x] 3.2 Register the job in planner's `server.ts` with cron `0 * * * *` (hourly), retryLimit 2, expireInSeconds 60
|
||||||
- [ ] 3.3 Verify job appears in `pgboss.schedule` table after planner starts (manual verification after dev stack is running)
|
- [x] 3.3 Verify job appears in `pgboss.schedule` table after planner starts (manual verification after dev stack is running)
|
||||||
- [x] 3.4 Write a test for the expire-sessions handler
|
- [x] 3.4 Write a test for the expire-sessions handler
|
||||||
|
|
||||||
## 4. Journal Worker Setup
|
## 4. Journal Worker Setup
|
||||||
|
|
@ -34,4 +34,4 @@
|
||||||
|
|
||||||
- [x] 6.1 Run `pnpm typecheck` — all packages pass
|
- [x] 6.1 Run `pnpm typecheck` — all packages pass
|
||||||
- [x] 6.2 Run `pnpm test` — new and existing tests pass (63 planner tests, including 2 new expire-sessions tests)
|
- [x] 6.2 Run `pnpm test` — new and existing tests pass (63 planner tests, including 2 new expire-sessions tests)
|
||||||
- [ ] 6.3 Start dev stack with `pnpm dev:full`, verify pg-boss tables are created and expire-sessions schedule is registered (manual verification)
|
- [x] 6.3 Start dev stack with `pnpm dev:full`, verify pg-boss tables are created and expire-sessions schedule is registered (manual verification)
|
||||||
|
|
@ -113,3 +113,18 @@ Creating an activity with `visibility = 'public'` SHALL enqueue a fan-out job th
|
||||||
#### Scenario: No accepted followers means no notifications
|
#### Scenario: No accepted followers means no notifications
|
||||||
- **WHEN** a user with zero accepted followers creates a public activity
|
- **WHEN** a user with zero accepted followers creates a public activity
|
||||||
- **THEN** the fan-out job runs and inserts zero rows
|
- **THEN** the fan-out job runs and inserts zero rows
|
||||||
|
|
||||||
|
### Requirement: Synthetic activity flag
|
||||||
|
The Journal SHALL persist a `synthetic` boolean on every activity so automated / demo content can be distinguished from user-created content.
|
||||||
|
|
||||||
|
#### Scenario: User-created activities default to non-synthetic
|
||||||
|
- **WHEN** an activity is created through any user-facing flow (GPX upload, sync import, Planner handoff)
|
||||||
|
- **THEN** the activity row is persisted with `synthetic = false`
|
||||||
|
|
||||||
|
#### Scenario: Bot inserts flag their rows as synthetic
|
||||||
|
- **WHEN** the demo-activity-bot inserts an activity
|
||||||
|
- **THEN** the row is persisted with `synthetic = true`
|
||||||
|
|
||||||
|
#### Scenario: Synthetic flag is not user-editable
|
||||||
|
- **WHEN** an activity owner edits the activity via any user-facing action
|
||||||
|
- **THEN** the stored `synthetic` value is not changed by the edit
|
||||||
|
|
|
||||||
45
openspec/specs/background-jobs/spec.md
Normal file
45
openspec/specs/background-jobs/spec.md
Normal file
|
|
@ -0,0 +1,45 @@
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Shared background job infrastructure for trails.cool apps, built on pg-boss. Provides a single PostgreSQL-backed queue used by both the planner and journal for one-shot, retried, and cron-scheduled work (e.g., session expiry, periodic maintenance), with worker lifecycle tied to the server process and observability via the existing Grafana stack.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
### Requirement: Job queue initialization
|
||||||
|
The `@trails-cool/jobs` package SHALL initialize a pg-boss instance using the app's `DATABASE_URL` and export a function to start the worker.
|
||||||
|
|
||||||
|
#### Scenario: Worker starts with server
|
||||||
|
- **WHEN** the planner or journal server starts
|
||||||
|
- **THEN** pg-boss connects to PostgreSQL, creates/migrates the `pgboss` schema if needed, and begins polling for jobs
|
||||||
|
|
||||||
|
#### Scenario: Worker stops on shutdown
|
||||||
|
- **WHEN** the server process receives SIGTERM
|
||||||
|
- **THEN** pg-boss completes any in-progress jobs and stops gracefully before the process exits
|
||||||
|
|
||||||
|
### Requirement: Cron job scheduling
|
||||||
|
The system SHALL support registering recurring jobs with cron expressions.
|
||||||
|
|
||||||
|
#### Scenario: Register a cron job
|
||||||
|
- **WHEN** a job is registered with a cron expression (e.g., `0 * * * *` for hourly)
|
||||||
|
- **THEN** pg-boss creates a schedule that enqueues the job at the specified interval
|
||||||
|
|
||||||
|
#### Scenario: Cron job survives restart
|
||||||
|
- **WHEN** the server process restarts
|
||||||
|
- **THEN** existing cron schedules persist and continue firing without re-registration conflicts
|
||||||
|
|
||||||
|
### Requirement: Job retry policy
|
||||||
|
Jobs SHALL support configurable retry policies with exponential backoff.
|
||||||
|
|
||||||
|
#### Scenario: Transient failure retry
|
||||||
|
- **WHEN** a job handler throws an error
|
||||||
|
- **THEN** pg-boss retries the job up to the configured retry limit with exponential backoff
|
||||||
|
|
||||||
|
#### Scenario: Permanent failure
|
||||||
|
- **WHEN** a job exhausts all retries
|
||||||
|
- **THEN** the job moves to the `failed` state and remains queryable for debugging
|
||||||
|
|
||||||
|
### Requirement: Job handler timeout
|
||||||
|
Each job handler SHALL have a configurable execution timeout.
|
||||||
|
|
||||||
|
#### Scenario: Job exceeds timeout
|
||||||
|
- **WHEN** a job handler does not complete within its timeout
|
||||||
|
- **THEN** pg-boss marks the job as failed with a timeout error
|
||||||
119
openspec/specs/demo-activity-bot/spec.md
Normal file
119
openspec/specs/demo-activity-bot/spec.md
Normal file
|
|
@ -0,0 +1,119 @@
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
A scheduled background job in the Journal that generates plausible synthetic routes and activities under a dedicated demo user ("Bruno"), so that fresh deployments and public landing pages have life in the feed without requiring real users. Synthetic content is flagged in the database, capped, and pruned on a configurable retention window.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
### Requirement: Persona configuration
|
||||||
|
The Journal SHALL load a demo persona — username, display name, bio, supported locales, and per-locale content pools — from configuration at worker boot, and SHALL fall back to a built-in default persona if no override is supplied or the supplied override fails validation.
|
||||||
|
|
||||||
|
#### Scenario: No override supplied — built-in default applies
|
||||||
|
- **WHEN** the worker starts with `DEMO_BOT_ENABLED=true` and `DEMO_BOT_PERSONA` is unset
|
||||||
|
- **THEN** the bot uses the built-in default persona (`username=bruno`, playful Berlin-flavoured display name and bio, `locales=["en","de"]`, and the shipped Bruno-voiced name/description pools)
|
||||||
|
- **AND** behaviour is identical to the demo bot before this change
|
||||||
|
|
||||||
|
#### Scenario: Inline JSON override
|
||||||
|
- **WHEN** `DEMO_BOT_PERSONA` is set to a valid inline JSON object with `username`, `displayName`, `bio`, `locales`, and `content.names` / `content.descriptions` for each listed locale
|
||||||
|
- **THEN** the bot uses that persona — `ensureDemoUser` inserts a row with the persona's username/displayName/bio/sentinel email, and subsequent generated routes and activities draw names and descriptions from the persona's per-locale pools
|
||||||
|
|
||||||
|
#### Scenario: File-backed override
|
||||||
|
- **WHEN** `DEMO_BOT_PERSONA` is set to `file:<absolute-path>` and the referenced file contains a valid persona JSON object
|
||||||
|
- **THEN** the worker reads the file once at boot and uses its contents as the persona
|
||||||
|
|
||||||
|
#### Scenario: Invalid JSON or schema violation → fall back
|
||||||
|
- **WHEN** `DEMO_BOT_PERSONA` is set but the value is not valid JSON, or fails the persona schema (bad username pattern, empty or too-short pool, unsupported locale, etc.)
|
||||||
|
- **THEN** the worker logs a warn-level entry describing the first validation failure and uses the built-in default persona
|
||||||
|
- **AND** the bot continues to run
|
||||||
|
|
||||||
|
#### Scenario: File-backed path unreadable → fall back
|
||||||
|
- **WHEN** `DEMO_BOT_PERSONA=file:<path>` but the file cannot be read (missing, permission denied, not a file)
|
||||||
|
- **THEN** the worker logs a warn-level entry and uses the built-in default persona
|
||||||
|
|
||||||
|
### Requirement: Persona username clash detection
|
||||||
|
The Journal SHALL refuse to attach the demo bot to a pre-existing non-demo user account when the supplied persona username collides with a human user already registered on the instance.
|
||||||
|
|
||||||
|
#### Scenario: Configured username belongs to a real user
|
||||||
|
- **WHEN** the worker starts, the persona's username matches an existing `users` row, and that row has no marker identifying it as a prior demo user (i.e. it was registered via the normal signup flow)
|
||||||
|
- **THEN** the worker logs an error-level "demo persona username clash" entry naming the colliding username
|
||||||
|
- **AND** the generation + prune jobs are not scheduled for this process — the bot stays disabled until the operator picks a different username
|
||||||
|
- **AND** the rest of the Journal continues to serve requests normally
|
||||||
|
|
||||||
|
### Requirement: Demo user bootstrap
|
||||||
|
The Journal SHALL ensure a dedicated bot user exists when the demo bot starts, creating it on first run if missing. The user's identity (username, display name, bio, sentinel email local-part) is derived from the active persona — either the operator-supplied persona or the built-in default.
|
||||||
|
|
||||||
|
#### Scenario: Bot user created on first run
|
||||||
|
- **WHEN** the Journal worker starts with `DEMO_BOT_ENABLED=true` and no `users` row matches the persona's username
|
||||||
|
- **THEN** a new `users` row is inserted with that username, the persona's display name, the persona's bio, a sentinel email `<username>@<domain>`, no passkey credentials, and `terms_accepted_at` + `terms_version` populated at the current version
|
||||||
|
- **AND** subsequent worker startups are idempotent — no second row is inserted
|
||||||
|
|
||||||
|
#### Scenario: Demo user has no usable credentials
|
||||||
|
- **WHEN** any request attempts to authenticate as the demo user via passkey or magic-link
|
||||||
|
- **THEN** authentication fails because no passkey is registered and no mailbox receives magic-link mails
|
||||||
|
|
||||||
|
### Requirement: Synthetic content generation job
|
||||||
|
The Journal SHALL run a recurring background job that generates one public route and one linked public activity for the demo user per run, subject to an env flag and a rate cap. The generated name and description are drawn from the active persona's per-locale content pools.
|
||||||
|
|
||||||
|
#### Scenario: Disabled in non-production environments
|
||||||
|
- **WHEN** the `DEMO_BOT_ENABLED` env var is absent or any value other than `"true"`
|
||||||
|
- **THEN** the job body is a no-op: no BRouter calls, no inserts, no errors
|
||||||
|
|
||||||
|
#### Scenario: Enabled generation flow (decide-to-walk fires)
|
||||||
|
- **WHEN** `DEMO_BOT_ENABLED=true`, the local hour is within 07:00–21:00, the per-tick Bernoulli roll fires, and the hard cap has not been reached
|
||||||
|
- **THEN** the job picks a random start and end point within the configured seed region, calls BRouter with the `trekking` profile, persists the returned GPX as a new route with `visibility='public'` and `synthetic=true`, and inserts a linked activity with the same GPX, `visibility='public'`, `synthetic=true`, a plausible `started_at` and `duration`, and a persona-voiced name + description sampled from one of the persona's supported locales
|
||||||
|
- **AND** the route and activity are attributed to the demo user
|
||||||
|
|
||||||
|
#### Scenario: Locale restricted to a single language
|
||||||
|
- **WHEN** the persona's `locales` list is `["en"]`
|
||||||
|
- **THEN** every generated route's name and description come from the persona's English pool — the German pool is never sampled
|
||||||
|
|
||||||
|
#### Scenario: Decide-to-walk does not fire
|
||||||
|
- **WHEN** the local hour is outside 07:00–21:00, or the Bernoulli roll does not fire
|
||||||
|
- **THEN** the job returns without inserting anything
|
||||||
|
|
||||||
|
#### Scenario: BRouter failure is tolerated
|
||||||
|
- **WHEN** the BRouter call returns no route, a rate-limit, or an error
|
||||||
|
- **THEN** the job logs the failure, inserts nothing, and exits without throwing — the next scheduled tick retries
|
||||||
|
|
||||||
|
#### Scenario: Hard cap prevents runaway growth
|
||||||
|
- **WHEN** there are already 40 or more synthetic items created in the last 14 days
|
||||||
|
- **THEN** the job skips generation for that tick
|
||||||
|
|
||||||
|
#### Scenario: Singleton scheduling prevents overlap
|
||||||
|
- **WHEN** a tick fires while the previous run is still executing
|
||||||
|
- **THEN** the new tick is skipped (pg-boss singleton semantics) so the job cannot overlap itself
|
||||||
|
|
||||||
|
### Requirement: Synthetic content retention
|
||||||
|
The Journal SHALL run a recurring job that deletes synthetic routes and activities older than a configurable window.
|
||||||
|
|
||||||
|
#### Scenario: Prune removes old synthetic content
|
||||||
|
- **WHEN** the prune job runs with `DEMO_BOT_RETENTION_DAYS=14`
|
||||||
|
- **THEN** every row in `journal.routes` and `journal.activities` with `synthetic=true` and `created_at < now() - 14 days` is deleted
|
||||||
|
- **AND** route-version rows cascade-delete via existing foreign keys
|
||||||
|
- **AND** rows with `synthetic=false` are never touched
|
||||||
|
|
||||||
|
#### Scenario: Prune is a no-op when nothing is old
|
||||||
|
- **WHEN** the prune job runs and no synthetic rows exceed the retention window
|
||||||
|
- **THEN** no DELETE statements execute and the job returns normally
|
||||||
|
|
||||||
|
#### Scenario: Disabled in non-production environments
|
||||||
|
- **WHEN** `DEMO_BOT_ENABLED` is not `"true"`
|
||||||
|
- **THEN** the prune job body is a no-op
|
||||||
|
|
||||||
|
### Requirement: Initial backfill
|
||||||
|
On first enablement (when no synthetic content exists yet) the Journal SHALL populate the demo profile with a small batch of items so the first visitor does not see a single-item feed.
|
||||||
|
|
||||||
|
#### Scenario: First enablement backfills several items
|
||||||
|
- **WHEN** the bot is enabled for the first time and the count of synthetic routes is 0
|
||||||
|
- **THEN** the generation job produces 3–5 items in sequence during its first run
|
||||||
|
- **AND** subsequent runs produce one item at a time as normal
|
||||||
|
|
||||||
|
### Requirement: Seed region is configurable
|
||||||
|
The Journal SHALL read the seed region (bounding box) from an env var so the deployment can change it without a code change.
|
||||||
|
|
||||||
|
#### Scenario: Env-configured region
|
||||||
|
- **WHEN** `DEMO_BOT_REGION` is set to a JSON object containing a `bbox` array of four numbers `[west, south, east, north]`
|
||||||
|
- **THEN** all generated start and end points fall within that box
|
||||||
|
|
||||||
|
#### Scenario: Sensible default
|
||||||
|
- **WHEN** `DEMO_BOT_REGION` is unset
|
||||||
|
- **THEN** the job uses a documented default region (inner Berlin) so that out-of-the-box runs still produce plausible Bruno-style walks
|
||||||
|
|
@ -128,12 +128,20 @@ Grafana SHALL authenticate users via GitHub OAuth, restricted to the trails-cool
|
||||||
- **WHEN** Grafana is deployed
|
- **WHEN** Grafana is deployed
|
||||||
- **THEN** the login form is disabled and only GitHub OAuth is available
|
- **THEN** the login form is disabled and only GitHub OAuth is available
|
||||||
|
|
||||||
### Requirement: Docker Compose deployment
|
### Requirement: Grafana database access
|
||||||
All services SHALL be deployed via Docker Compose, including Grafana, Prometheus, and Loki for the flagship instance.
|
The `grafana_reader` PostgreSQL role SHALL have SELECT access to the `pgboss` schema for job queue observability.
|
||||||
|
|
||||||
#### Scenario: Monitoring stack starts
|
#### Scenario: Grant access on deploy
|
||||||
- **WHEN** `docker compose up -d` is run
|
- **WHEN** the infrastructure deploy runs
|
||||||
- **THEN** Grafana, Prometheus, Loki, Promtail, postgres-exporter, node-exporter, and cAdvisor containers start alongside the application containers
|
- **THEN** `grafana_reader` is granted `USAGE` on the `pgboss` schema and `SELECT` on all tables in it
|
||||||
|
|
||||||
|
### Requirement: Monitoring stack
|
||||||
|
The Grafana Service Health dashboard SHALL include a job queue health panel.
|
||||||
|
|
||||||
|
#### Scenario: Job queue panel displays metrics
|
||||||
|
- **WHEN** a user views the Service Health dashboard
|
||||||
|
- **THEN** they see a panel showing job queue depth, completed jobs per hour, and failed jobs
|
||||||
|
- **AND** failed jobs are highlighted for investigation
|
||||||
|
|
||||||
### Requirement: Metrics collection
|
### Requirement: Metrics collection
|
||||||
Prometheus SHALL scrape metrics from all application and infrastructure services.
|
Prometheus SHALL scrape metrics from all application and infrastructure services.
|
||||||
|
|
|
||||||
|
|
@ -40,3 +40,20 @@ The Planner SHALL synchronize waypoint edits, route options, and overlay prefere
|
||||||
- **WHEN** a participant reloads the session page
|
- **WHEN** a participant reloads the session page
|
||||||
- **THEN** the notes content is restored from the Yjs document
|
- **THEN** the notes content is restored from the Yjs document
|
||||||
- **AND** the editor displays the existing text immediately
|
- **AND** the editor displays the existing text immediately
|
||||||
|
|
||||||
|
### Requirement: Session expiry
|
||||||
|
Open sessions with no activity for 7 days SHALL be automatically deleted by a scheduled background job.
|
||||||
|
|
||||||
|
#### Scenario: Stale session cleanup
|
||||||
|
- **WHEN** the hourly `expire-sessions` cron job runs
|
||||||
|
- **THEN** all sessions with `last_activity` older than 7 days are deleted from the database
|
||||||
|
- **AND** their Yjs documents are removed from memory
|
||||||
|
|
||||||
|
#### Scenario: Active session preserved
|
||||||
|
- **WHEN** the `expire-sessions` job runs
|
||||||
|
- **THEN** sessions with `last_activity` within the last 7 days are NOT deleted
|
||||||
|
|
||||||
|
#### Scenario: Cleanup is observable
|
||||||
|
- **WHEN** the `expire-sessions` job completes
|
||||||
|
- **THEN** the job output includes the count of expired sessions
|
||||||
|
- **AND** the result is visible in the Grafana job queue dashboard
|
||||||
|
|
|
||||||
|
|
@ -130,3 +130,18 @@ Any listing that exposes routes beyond the owner's own dashboard SHALL only incl
|
||||||
- **WHEN** a logged-in user views their own routes list at `/routes`
|
- **WHEN** a logged-in user views their own routes list at `/routes`
|
||||||
- **THEN** the list includes all of their own routes regardless of visibility
|
- **THEN** the list includes all of their own routes regardless of visibility
|
||||||
|
|
||||||
|
### Requirement: Synthetic route flag
|
||||||
|
The Journal SHALL persist a `synthetic` boolean on every route so automated / demo content can be distinguished from user-created content.
|
||||||
|
|
||||||
|
#### Scenario: User-created routes default to non-synthetic
|
||||||
|
- **WHEN** a route is created through any user-facing flow (New Route, GPX import, Planner handoff)
|
||||||
|
- **THEN** the route row is persisted with `synthetic = false`
|
||||||
|
|
||||||
|
#### Scenario: Bot inserts flag their rows as synthetic
|
||||||
|
- **WHEN** the demo-activity-bot inserts a route
|
||||||
|
- **THEN** the row is persisted with `synthetic = true`
|
||||||
|
|
||||||
|
#### Scenario: Synthetic flag is not user-editable
|
||||||
|
- **WHEN** a route owner edits a route via any user-facing action
|
||||||
|
- **THEN** the stored `synthetic` value is not changed by the edit
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue