ci: stop starving vitest workers to death #126

Merged

ullrich merged 1 commit from ci-vitest-worker-starvation into main

2026-09-07 12:26:43 +00:00

ullrich commented

2026-09-07 09:52:33 +00:00

Owner

Checks fails intermittently with every test passing:

Error: [vitest-pool]: Failed to start forks worker for test files
  .../packages/i18n/src/client.test.ts
Caused by: [vitest-pool-runner]: Timeout waiting for worker to respond
 Test Files  10 passed (10)     Tests  89 passed (89)    Errors  1 error

This is not the cold-import timeout vitest.shared.ts already handles, and testTimeout cannot touch it — the run fails because a worker never finished starting. That deadline is a hardcoded START_TIMEOUT constant inside vitest's pool runner (60s), with no config option and no environment variable to raise it. Confirmed by reading the shipped bundle:

const startPromise = this.withTimeout(this.waitForStart(), START_TIMEOUT);
// START_TIMEOUT = 6e4

So the only available lever is to stop starving the workers.

The starvation is multiplicative, and nothing in the repo made it visible

layer	fan-out
`pnpm test` → `turbo test`	default concurrency 10
packages defining `vitest run`	11
each vitest's fork pool	sized from `os.availableParallelism()` — the host CPU count, because a cgroup cpu limit does not change it → up to (n-1) forks
what the job container is actually allowed	3 CPUs (`cpus: 3, mem_limit: 10g` for the dind service in the runner compose on the BRouter host)

Ten packages × nine forks ≈ 90 node processes on three CPUs. A fork missing a 60s handshake under that load is unsurprising.

The fix

TURBO_CONCURRENCY=3 on the CI step plus maxForks: 2 in vitest.shared.ts bounds the product at ~6 workers instead of ~90. The fork cap applies only when CI is set — locally the cores exist, and capping would just make pnpm test slow.

Verified with CI=1 TURBO_CONCURRENCY=3 pnpm test: 12 tasks successful, every suite green.

Two things this is NOT

Not a proven fix for the whole failure rate. Checks failed 71 of the last 300 runs (23%), and that population also contains genuine failures — openspec 1.12 rejecting five specs on #104, and the TS2724 on #120. This removes one contributor; it does not account for 23%.
Not the borgmatic I/O window. That was the first hypothesis, since the BRouter host's backup saturates I/O in the morning and main's failure was at 08:56 UTC. Bucketing all 300 runs by hour does not support it — failures sit near the 23% baseline across the day rather than clustering.

Follow-up worth considering

The runner resource limits live in host-side compose outside this repo, where nothing here can see or review them. That is the same blind spot that made the playwright image version unfixable in-repo until it was declared in ci.yml.

`Checks` fails intermittently with **every test passing**: ``` Error: [vitest-pool]: Failed to start forks worker for test files .../packages/i18n/src/client.test.ts Caused by: [vitest-pool-runner]: Timeout waiting for worker to respond Test Files 10 passed (10) Tests 89 passed (89) Errors 1 error ``` This is **not** the cold-import timeout `vitest.shared.ts` already handles, and `testTimeout` cannot touch it — the run fails because a worker never finished *starting*. That deadline is a hardcoded `START_TIMEOUT` constant inside vitest's pool runner (60s), with no config option and no environment variable to raise it. Confirmed by reading the shipped bundle: ```js const startPromise = this.withTimeout(this.waitForStart(), START_TIMEOUT); // START_TIMEOUT = 6e4 ``` So the only available lever is to stop starving the workers. ### The starvation is multiplicative, and nothing in the repo made it visible | layer | fan-out | |---|---| | `pnpm test` → `turbo test` | default concurrency **10** | | packages defining `vitest run` | **11** | | each vitest's fork pool | sized from `os.availableParallelism()` — the **host** CPU count, because a cgroup cpu limit does not change it → up to **(n-1)** forks | | what the job container is actually allowed | **3 CPUs** (`cpus: 3, mem_limit: 10g` for the dind service in the runner compose on the BRouter host) | Ten packages × nine forks ≈ **90 node processes on three CPUs**. A fork missing a 60s handshake under that load is unsurprising. ### The fix `TURBO_CONCURRENCY=3` on the CI step plus `maxForks: 2` in `vitest.shared.ts` bounds the product at ~6 workers instead of ~90. The fork cap applies **only when `CI` is set** — locally the cores exist, and capping would just make `pnpm test` slow. Verified with `CI=1 TURBO_CONCURRENCY=3 pnpm test`: 12 tasks successful, every suite green. ### Two things this is NOT - **Not a proven fix for the whole failure rate.** `Checks` failed **71 of the last 300 runs (23%)**, and that population also contains genuine failures — openspec 1.12 rejecting five specs on #104, and the `TS2724` on #120. This removes one contributor; it does not account for 23%. - **Not the borgmatic I/O window.** That was the first hypothesis, since the BRouter host's backup saturates I/O in the morning and main's failure was at 08:56 UTC. Bucketing all 300 runs by hour **does not support it** — failures sit near the 23% baseline across the day rather than clustering. ### Follow-up worth considering The runner resource limits live in host-side compose outside this repo, where nothing here can see or review them. That is the same blind spot that made the playwright image version unfixable in-repo until it was declared in `ci.yml`.

ullrich added 1 commit

2026-09-07 09:52:33 +00:00

ci: stop starving vitest workers to death

CI / Security Scan (pull_request) Successful in 50s

Details

CI / Dockerfile Package Check (pull_request) Successful in 35s

Details

CI / Checks (pull_request) Successful in 2m35s

Details

CI / Visual Tests (pull_request) Successful in 2m14s

Details

CI / Journal Image Smoke Test (pull_request) Successful in 9m5s

Details

CI / E2E Tests (pull_request) Successful in 4m21s

Details

Cancel superseded CI / Cancel in-flight CI (pull_request) Successful in 13s

Details

cc6f2ebdb6

`Checks` fails intermittently with every test passing:

  Error: [vitest-pool]: Failed to start forks worker for test files
    .../packages/i18n/src/client.test.ts
  Caused by: [vitest-pool-runner]: Timeout waiting for worker to respond
   Test Files  10 passed (10)     Tests  89 passed (89)    Errors  1 error

This is NOT the cold-import timeout that vitest.shared.ts already handles, and
testTimeout cannot touch it: the run fails because a worker never finished
*starting*. That deadline is a hardcoded START_TIMEOUT constant inside vitest's
pool runner — 60s, with no config option and no environment variable to raise
it (confirmed by reading the shipped bundle). So the only lever is to stop
starving the workers.

The starvation is multiplicative, and nothing in the repo made it visible:

  pnpm test           -> turbo test, default concurrency 10
  11 packages         -> each defines a `vitest run` task
  each vitest         -> sizes its fork pool from os.availableParallelism(),
                         which reports the HOST cpu count because a cgroup cpu
                         limit does not change it, so up to (n-1) forks
  the job container   -> is allowed 3 CPUs (`cpus: 3, mem_limit: 10g` for the
                         dind service in /home/trails/forgejo-runners/
                         docker-compose.yml on the BRouter host)

Ten packages by nine forks is ~90 node processes contending for three CPUs. A
fork missing a 60s handshake under that load is unsurprising.

TURBO_CONCURRENCY=3 on the CI step plus maxForks=2 in vitest.shared.ts bounds
the product at ~6 workers instead of ~90. The fork cap is applied only when CI
is set — locally the cores exist and capping would just make `pnpm test` slow.

Two things this is NOT:

- Not a proven fix for the whole failure rate. `Checks` failed 71 of the last
  300 runs (23%), and that population also contains genuine failures: openspec
  1.12 rejecting five specs on #104, and the TS2724 on #120. This removes one
  contributor.
- Not the borgmatic I/O window. That was the first hypothesis, since the BRouter
  host's backup saturates I/O in the morning and main's failure was at 08:56
  UTC. Bucketing all 300 runs by hour does not support it — failures sit near
  the 23% baseline across the day rather than clustering.

Verified locally with `CI=1 TURBO_CONCURRENCY=3 pnpm test`: 12 tasks
successful, every suite green.

Follow-up worth considering: the runner resource limits live in host-side
compose outside this repo, where nothing here can see or review them. That is
the same blind spot that made the playwright image version unfixable in-repo
until it was declared in ci.yml.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W9K6hHEZx5aqyWZKHNrxsD

ullrich merged commit e5b04e3827 into main

2026-09-07 12:26:43 +00:00

ullrich deleted branch ci-vitest-worker-starvation

2026-09-07 12:26:44 +00:00

ullrich referenced this pull request from a commit

2026-09-07 12:26:44 +00:00

Merge pull request 'ci: stop starving vitest workers to death' (#126) from ci-vitest-worker-starvation into main

ullrich referenced this pull request from a commit

2026-09-07 12:43:07 +00:00

Complete the vitest 5 upgrade and port maxForks to maxWorkers

ullrich referenced this pull request

2026-09-07 12:43:24 +00:00