trails-cool/trails
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
trails/openspec/specs/observability/spec.md
Ullrich Schäfer acb92b32fa Spec drift catch-up: URLs, settings split, navbar shape 
Outcome of today's `/spec-drift-review`. Concentrated on the
high-and-medium-severity items; low-severity wording left for the
next per-feature change to pick up naturally.

High-severity URL fixes (specs were actively misleading):

- account-management — email-change verification URL was specced as
  `/auth/verify-email-change?token=...`; the actual route is
  `/auth/verify?email-change=1&token=...` (see auth.verify.tsx:8).
  A reader implementing against the old wording would build a link
  that 404s.
- observability — "Both apps SHALL expose a /metrics endpoint" was
  half right: the planner is at /metrics, but the journal exposes
  /api/metrics. The infrastructure spec already had the right URLs;
  the observability spec disagreed with itself. Now both correct,
  with a one-line note explaining the per-app split.

Medium-severity wording drift (Stream E aftermath):

- profile-settings, account-management, connected-services — all
  three said "the settings page SHALL include a [...] section",
  which described the old single-scrollable settings page. Stream E
  (PR #323) split /settings into four sub-pages
  (/settings/{profile,account,security,connections}); rewording each
  spec to point at its specific sub-page. The API endpoints
  (/api/settings/*) and behavior are unchanged.
- authentication-methods — passkey add/delete previously said "via
  the settings page"; now specifically /settings/security.

Code drift fixed inline:

- apps/journal/app/routes/auth.verify.tsx — after a successful
  email change, the redirect was going to `/settings#account` (an
  anchor on the OLD single-scrollable settings page). Stream E
  retired that page; the right destination now is
  `/settings/account`. Without this the user would land on
  /settings/profile (which is what /settings redirects to) instead
  of the page that just changed.

sse-broker spec wording:

- The "no buffering tweaks in the Caddy reverse_proxy block"
  scenario asserted the entry was "a plain `reverse_proxy
  journal:3000`". After PR #329 the journal block has
  `lb_try_duration 30s` / `lb_try_interval 250ms` — neither affects
  streaming, so SSE still works, but the spec's "plain" language
  was no longer literally true. Reworded to forbid only
  buffering-related directives; explicitly call out that
  retry-on-restart directives like lb_try_duration are fine.

Navbar consolidation (journal-landing):

- The shipped navbar's full shape was scattered: notifications said
  "navbar has a bell," explore said "navbar has an Explore entry,"
  but no spec described the avatar dropdown, the primary-nav
  cluster (Feed/Routes/Activities), or the mobile drawer (Stream
  C / PR #324). Added a "Top navbar shape" requirement to
  journal-landing covering all of it — anonymous vs signed-in,
  desktop vs mobile, dropdown contents, drawer behavior. The
  per-feature specs (notifications, explore) still own their own
  badges/entries; this requirement just says what the whole
  cluster looks like.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 21:19:02 +02:00

100 lines
5 KiB
Markdown
Raw Blame History

## Purpose
Health endpoints, Prometheus metrics, structured logging, Grafana dashboards, and alerting for both apps and the flagship instance.
## Requirements
### Requirement: Health endpoints
Both apps SHALL expose a `/health` endpoint returning service and database status.
#### Scenario: Healthy service
- **WHEN** the app is running and the database is reachable
- **THEN** `GET /health` returns `{ "status": "ok", "db": "connected", "version": "<string>" }` with HTTP 200 (version from `SENTRY_RELEASE` env var, defaults to `"dev"`)
#### Scenario: Degraded service
- **WHEN** the app is running but the database is unreachable
- **THEN** `GET /health` returns `{ "status": "degraded", "db": "unreachable", "version": "<string>" }` with HTTP 503
### Requirement: Prometheus metrics
Both apps SHALL expose a Prometheus-formatted metrics endpoint. The Planner exposes it at `/metrics` (a server-side route handler outside the React Router app); the Journal exposes it at `/api/metrics` (a route under the React Router app — the `/api` prefix matches the rest of the app's route handlers and avoids colliding with any user-facing `/metrics` page). Prometheus's scrape config in `infrastructure/prometheus/prometheus.yml` reflects this split.
#### Scenario: Default Node.js metrics
- **WHEN** Prometheus scrapes the Journal's `/api/metrics` (or the Planner's `/metrics`)
- **THEN** it receives event loop lag, heap usage, and GC metrics
#### Scenario: HTTP request metrics
- **WHEN** requests are served
- **THEN** `http_request_duration_seconds` histogram is updated with route, method, and status labels
#### Scenario: Planner-specific metrics
- **WHEN** the Planner is running
- **THEN** `planner_active_sessions` and `planner_connected_clients` gauges reflect current state
#### Scenario: BRouter latency metrics
- **WHEN** the Planner proxies a routing request to BRouter
- **THEN** `brouter_request_duration_seconds` histogram is updated (buckets: 0.1s to 10s)
### Requirement: Structured logging
Both apps SHALL output structured JSON logs in production.
#### Scenario: Request logging
- **WHEN** an HTTP request is served in production
- **THEN** a JSON log line is emitted with method, path, status, duration, and timestamp
#### Scenario: Dev mode pretty printing
- **WHEN** running in development
- **THEN** logs are human-readable (pretty-printed)
### Requirement: Grafana dashboards
The flagship instance SHALL have pre-configured Grafana dashboards.
#### Scenario: Overview dashboard
- **WHEN** an operator opens Grafana
- **THEN** they see request rate, error rate, and latency percentiles
#### Scenario: Infrastructure dashboard
- **WHEN** an operator checks infrastructure health
- **THEN** they see CPU, memory, disk usage, and DB connection pool stats
### Requirement: Alerting
Grafana SHALL alert on critical conditions.
#### Scenario: Disk usage alert
- **WHEN** disk usage exceeds 80% for 15 minutes
- **THEN** an alert fires
#### Scenario: App down alert
- **WHEN** an app returns zero healthy responses for 2 minutes
- **THEN** an alert fires
### Requirement: Log aggregation
Loki SHALL collect and index logs from all Docker containers.
#### Scenario: Search logs
- **WHEN** an operator searches logs in Grafana
- **THEN** they can filter by container name, log level, and time range
### Requirement: Remote BRouter metrics
Prometheus on the flagship host SHALL scrape BRouter-specific metrics from the dedicated BRouter host over the vSwitch. Scraping SHALL be scoped to BRouter containers only and SHALL NOT collect host-level metrics from the dedicated host's other (non-trails.cool) workloads.
#### Scenario: BRouter container metrics collected
- **WHEN** Prometheus scrapes the dedicated BRouter host
- **THEN** metrics from the BRouter container (and, if enabled, the Caddy sidecar) are collected via cAdvisor filtered by container label, or via a BRouter/JMX metrics endpoint exposed on the vSwitch
#### Scenario: No shared-host noise
- **WHEN** Prometheus is configured with the remote BRouter scrape target
- **THEN** no configuration scrapes the dedicated host's `node_exporter` or metrics from containers other than BRouter and its sidecar
#### Scenario: BRouter scrape failure alert
- **WHEN** the BRouter scrape target returns `up == 0` for more than 2 minutes
- **THEN** Grafana fires an alert distinct from flagship-side alerts
### Requirement: Remote BRouter log shipping
Loki on the flagship host SHALL receive BRouter container logs from the dedicated BRouter host via a Promtail or Alloy agent running on that host as the `trails` user. Log shipping SHALL be scoped to BRouter-related containers only.
#### Scenario: BRouter logs visible in Grafana
- **WHEN** the BRouter container writes to stdout or stderr
- **THEN** the log line is available in Grafana Explore via Loki with container and host labels identifying it as coming from the dedicated BRouter host
#### Scenario: Non-BRouter logs not shipped
- **WHEN** a non-BRouter container on the dedicated host writes logs
- **THEN** those logs are not ingested by the flagship Loki instance
Reference in a new issue View git blame Copy permalink