ci(e2e): validate the BRouter segment by size, not by "file is non-empty" #57

Merged

ullrich merged 1 commit from ci/brouter-segment-integrity into main

2026-07-30 21:27:20 +00:00

ullrich commented

2026-07-30 14:03:31 +00:00

Owner

Root-causes the recurring BRouter did not serve a route within 180s E2E failure — the one that has been muddying the whole Renovate queue.

It was never a timeout, and never contention

I asserted both of those before reading the evidence. The container logs captured at failure show BRouter answering every poll in ~600ms for the full 180 s:

13:38:00.351Z  ms=618 -> GET /brouter?lonlats=13.4,52.5|13.5,52.5&profile=trekking
13:38:02.961Z  ms=602 -> GET /brouter?...
…40 more, all sub-second…

It was up and healthy. It simply could not produce a route, because the cached segment was corrupt. Each failing run logged "Berlin segment already cached" immediately beforehand.

Two lines made that permanent

if [ -s /dst/E10_N50.rd5 ]          # non-empty, nothing more
wget -q … -O /dst/E10_N50.rd5       # writes straight to the cache path

A transfer truncated mid-flight is the corruption, and every later run on that slot accepts it as a valid cache. That is why it hit #46 and both #54 runs rather than being intermittent.

Fixes

Compare the cached size to the upstream Content-Length (124,034,647 bytes today) and refetch on mismatch
Download to .E10_N50.part, rename only once the size checks out — a failed download can no longer become the cache
A 100 MB floor as a second guard for when HEAD yields no Content-Length. Without it a failed HEAD would mean refetching 118 MB every run while still being unable to validate the result; the floor still rejects an obviously truncated file
Print the response body on failure. This is the diagnostic that was missing — the old handler printed only the container log, which showed healthy 600 ms replies and therefore read as a slow start. no route found vs connection refused vs a Java stack trace mean completely different things
Drop the segment volume on failure so a bad cache self-heals next run instead of failing forever on that slot
Fix the progress message: it printed ${i}0s while sleeping 2 s per iteration, so "ready after 300s" actually meant 60 s

Verified

bash -n on both step scripts and sh -n on the inner alpine script, plus the cache-validity logic exercised directly:

cached	upstream size	decision
124,034,647	124,034,647	REUSE
51,200	124,034,647	REFETCH
124,034,647	unknown	REUSE (above floor)
51,200	unknown	REFETCH
124,034,647	125,000,000	REFETCH

Under the old [ -s ] check, the 51 KB row was REUSE — that is the bug in one line.

Note

The volume is currently absent on both DinD slots (pruned since), so the next e2e run refetches from scratch and should pass regardless of this PR. This makes sure it does not silently return.

Root-causes the recurring `BRouter did not serve a route within 180s` E2E failure — the one that has been muddying the whole Renovate queue. ## It was never a timeout, and never contention I asserted both of those before reading the evidence. The container logs captured at failure show BRouter answering **every poll in ~600ms** for the full 180 s: ``` 13:38:00.351Z ms=618 -> GET /brouter?lonlats=13.4,52.5|13.5,52.5&profile=trekking 13:38:02.961Z ms=602 -> GET /brouter?... …40 more, all sub-second… ``` It was up and healthy. It simply could not produce a route, because the cached segment was corrupt. Each failing run logged **"Berlin segment already cached"** immediately beforehand. ## Two lines made that permanent ```sh if [ -s /dst/E10_N50.rd5 ] # non-empty, nothing more wget -q … -O /dst/E10_N50.rd5 # writes straight to the cache path ``` A transfer truncated mid-flight *is* the corruption, and every later run on that slot accepts it as a valid cache. That is why it hit #46 and **both** #54 runs rather than being intermittent. ## Fixes - Compare the cached size to the upstream `Content-Length` (124,034,647 bytes today) and refetch on mismatch - Download to `.E10_N50.part`, rename only once the size checks out — a failed download can no longer become the cache - A **100 MB floor** as a second guard for when `HEAD` yields no `Content-Length`. Without it a failed HEAD would mean refetching 118 MB every run *while still being unable to validate the result*; the floor still rejects an obviously truncated file - **Print the response body on failure.** This is the diagnostic that was missing — the old handler printed only the container log, which showed healthy 600 ms replies and therefore read as a slow start. `no route found` vs `connection refused` vs a Java stack trace mean completely different things - Drop the segment volume on failure so a bad cache self-heals next run instead of failing forever on that slot - Fix the progress message: it printed `${i}0s` while sleeping 2 s per iteration, so "ready after 300s" actually meant 60 s ## Verified `bash -n` on both step scripts and `sh -n` on the inner alpine script, plus the cache-validity logic exercised directly: | cached | upstream size | decision | |---|---|---| | 124,034,647 | 124,034,647 | REUSE | | 51,200 | 124,034,647 | **REFETCH** | | 124,034,647 | unknown | REUSE (above floor) | | 51,200 | unknown | **REFETCH** | | 124,034,647 | 125,000,000 | REFETCH | Under the old `[ -s ]` check, the 51 KB row was REUSE — that is the bug in one line. ## Note The volume is currently absent on both DinD slots (pruned since), so the next e2e run refetches from scratch and should pass regardless of this PR. This makes sure it does not silently return.

ullrich added 1 commit

2026-07-30 14:03:31 +00:00

ci(e2e): validate the BRouter segment by size, not by "file is non-empty"

CI / Dockerfile Package Check (pull_request) Successful in 24s

Details

CI / Security Scan (pull_request) Successful in 48s

Details

CI / Visual Tests (pull_request) Successful in 3m53s

Details

CI / Checks (pull_request) Successful in 4m33s

Details

CI / Journal Image Smoke Test (pull_request) Successful in 10m31s

Details

CI / E2E Tests (pull_request) Successful in 12m56s

Details

49c950992f

Root-causes the recurring "BRouter did not serve a route within 180s" E2E
failure. It was never a timeout and never contention — both of which I asserted
before reading the evidence.

The container logs captured at failure show BRouter answering EVERY poll in
~600ms for the full 180s. It was up and healthy; it just could not produce a
route, because the cached segment was corrupt. The failing runs each logged
"Berlin segment already cached" immediately beforehand.

Two lines made that state permanent:

  if [ -s /dst/E10_N50.rd5 ]   # non-empty, nothing more
  wget -q ... -O /dst/E10_N50.rd5   # writes straight to the cache path

So a transfer truncated mid-flight IS the corruption, and every later run on that
slot accepts it as a valid cache. Hit #46 and both #54 runs.

Fixes:

- Compare the cached size against the upstream Content-Length (124,034,647 bytes
  today) and refetch on mismatch.
- Download to `.E10_N50.part` and rename only after the size checks out, so a
  failed download can never become the cache.
- A 100MB floor as a second guard, used when the HEAD request yields no
  Content-Length. Without it a failed HEAD would mean refetching 118MB every run
  while still being unable to validate the result — the floor still rejects an
  obviously truncated file.
- On warmup failure, print the actual response body. That is the diagnostic that
  was missing: the old handler printed only the container log, which showed
  healthy 600ms replies and therefore read as a slow start. "no route found",
  "connection refused" and a Java stack trace mean completely different things.
- Also drop the segment volume on failure, so a bad cache self-heals on the next
  run instead of failing forever on that slot.
- Fix the progress message: it printed "${i}0s" while sleeping 2s per iteration,
  so "ready after 300s" actually meant 60s.

Verified: `bash -n` on both step scripts and `sh -n` on the inner alpine script,
plus the cache-validity logic exercised directly —

  complete cache          -> REUSE
  truncated cache         -> REFETCH
  no Content-Length, good -> REUSE (above floor)
  no Content-Length, trunc-> REFETCH
  upstream resized        -> REFETCH

against the old behaviour, where a 51KB file was REUSE.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X8hNxgYp777FRqYtVmQNaU

ullrich merged commit a443a07653 into main

2026-07-30 21:27:20 +00:00

ullrich deleted branch ci/brouter-segment-integrity

2026-07-30 21:27:20 +00:00

ullrich referenced this pull request from a commit

2026-07-30 21:27:21 +00:00

Merge pull request 'ci(e2e): validate the BRouter segment by size, not by "file is non-empty"' (#57) from ci/brouter-segment-integrity into main