trails-cool/trails
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(journal): bound Prometheus route label cardinality

Browse source 
The httpRequestDuration histogram labeled every request with the raw URL
path (`url.split("?")[0]`), so every distinct path — `/activities/<uuid>`,
`/routes/<uuid>`, probed usernames, crawler junk — became a permanent
label-set. prom-client never evicts label-sets, so RSS grew linearly for
the life of the process and reset only on restart. On the flagship this
showed as ~25 MiB/day of unbounded journal memory growth; a live
`/api/metrics` scrape held 2,655 histogram series across 291 distinct
`route` values (344 KB body), dominated by per-UUID activity paths.

Add `normalizeRoute()` in metrics.server.ts: it collapses dynamic path
segments back to the `:param` templates declared in app/routes.ts (UUID
and numeric segments -> `:id`; `:username`/`:provider` slots keyed by
their preceding static segment), strips query/fragment, and enforces a
hard cap (MAX_ROUTES) that funnels anything beyond the known route table
into a single `/other` bucket as an absolute backstop. Cardinality is now
proportional to the route table, not to traffic.

Logs still record the raw path (`logger.info`) — only the metric label is
bounded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-06-18 15:58:01 +02:00
parent 9266233614
commit cd73093beb
3 changed files with 153 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

4
apps/journal/server.ts
View file

@ -6,7 +6,7 @@ import { createReadStream, statSync } from "node:fs";
import { join, extname, resolve } from "node:path";
import { logger, requestContext } from "./app/lib/logger.server.ts";
import { randomUUID } from "node:crypto";
import { httpRequestDuration, registry } from "./app/lib/metrics.server.ts";
import { httpRequestDuration, normalizeRoute, registry } from "./app/lib/metrics.server.ts";
import { createBoss, startWorker } from "@trails-cool/jobs";
import { getDatabaseUrl } from "@trails-cool/db";
import postgres, { type Sql } from "postgres";
@ -117,7 +117,7 @@ const server = createServer((req, res) => {
const duration = Date.now() - start;
logger.info({ method: req.method, path: url, status: res.statusCode, duration }, "request");
httpRequestDuration.observe(
{ method: req.method ?? "GET", route: url.split("?")[0]!, status: String(res.statusCode) },
{ method: req.method ?? "GET", route: normalizeRoute(url), status: String(res.statusCode) },
duration / 1000,
);
});