trails-cool/trails
Watch
1
1
Fork
You've already forked trails
 0
Code Issues 2 Pull requests Projects Releases Packages Activity Actions 9

SENTRY_AUTH_TOKEN returns 401 — sourcemap upload silently broken #5

New issue
Closed
opened 2026-07-26 07:13:59 +00:00 by ullrich · 2 comments
ullrich commented 2026-07-26 07:13:59 +00:00
Owner
Copy link

Root cause found 2026-09-11 — see PR #143. The "Likely fix" section below is wrong.
The token was never the problem: it returns 200 against both sentry.io and
de.sentry.io. The CD workflows passed it with secrets: (literal values)
where secret-files: (paths) was needed, so the build received the string
"/tmp/sentry_token" as the credential. Scopes and data region were both
investigated and eliminated — see the comments.

This issue stays open for its follow-up: a failed upload must fail the build.

Production builds log:

sentry reported an error: Invalid token (http status: 401)
[sentry-vite-plugin] Error: ... Command sourcemaps upload -p journal --release <sha> ... failed

This does not fail the build. sentry-vite-plugin reports the error and the build continues, so images ship fine — but sourcemaps are never uploaded and production stack traces stay unsymbolicated. It went unnoticed for exactly that reason.

The token was rotated on 2026-07-25 and the new value is confirmed committed (f575eae replaced the line in infrastructure/secrets.app.env) and used by the production build — it still 401s. So the token itself is wrong, not the plumbing.

Likely fix

sentry-cli does two things here: releases new and sourcemaps upload. Both need release write.

  • Prefer an Organization Auth Token (Sentry → Settings → Auth Tokens), which is the type intended for CI
  • On a legacy user token, it needs project:releases (plus org:read)

Update with sops infrastructure/secrets.app.env. Nothing to change on the Forgejo side — CD decrypts it with AGE_SECRET_KEY and passes it as a BuildKit secret (which is also why it never lands in the image).

Follow-up worth doing

Make the build fail when the Sentry upload fails, so this cannot rot silently again.

> **Root cause found 2026-09-11 — see PR #143. The "Likely fix" section below is wrong.** > The token was never the problem: it returns 200 against both `sentry.io` and > `de.sentry.io`. The CD workflows passed it with `secrets:` (literal values) > where `secret-files:` (paths) was needed, so the build received the string > "/tmp/sentry_token" as the credential. Scopes and data region were both > investigated and eliminated — see the comments. > > This issue stays open for its follow-up: a failed upload must fail the build. --- Production builds log: ``` sentry reported an error: Invalid token (http status: 401) [sentry-vite-plugin] Error: ... Command sourcemaps upload -p journal --release <sha> ... failed ``` **This does not fail the build.** `sentry-vite-plugin` reports the error and the build continues, so images ship fine — but sourcemaps are never uploaded and production stack traces stay unsymbolicated. It went unnoticed for exactly that reason. The token was rotated on 2026-07-25 and the new value is confirmed committed (`f575eae` replaced the line in `infrastructure/secrets.app.env`) and used by the production build — it still 401s. So the token itself is wrong, not the plumbing. ## Likely fix `sentry-cli` does two things here: `releases new` and `sourcemaps upload`. Both need release write. - Prefer an **Organization Auth Token** (Sentry → Settings → Auth Tokens), which is the type intended for CI - On a legacy user token, it needs `project:releases` (plus `org:read`) Update with `sops infrastructure/secrets.app.env`. Nothing to change on the Forgejo side — CD decrypts it with `AGE_SECRET_KEY` and passes it as a BuildKit secret (which is also why it never lands in the image). ## Follow-up worth doing Make the build **fail** when the Sentry upload fails, so this cannot rot silently again.
ullrich commented 2026-09-11 12:00:54 +00:00
Author
Owner
Copy link

Status on 2026-09-11: still broken, and two causes are now eliminated

The most recent production deploy (cd-apps run 1459, commit 7968b49) shows the same error for the journal and for the planner:

sentry-cli releases new 7968b49546e806bfc46fbafe0505c0ff2aaadacd
  sentry reported an error: Invalid token (http status: 401)

The token has not changed since this issue was written. The ciphertext of SENTRY_AUTH_TOKEN in infrastructure/secrets.app.env last changed in f575eae (2026-07-25). It is the same at main today.

Eliminated: the scope theory in the description above

Insufficient scopes give 403 Forbidden. This is 401 Invalid token. The two are different failures. The description names scopes as the likely fix, so please do not start there.

Eliminated: the data region

The org is in the EU region. Every DSN in the repository has an o4509530546634752.ingest.de.sentry.io host. But sentry-cli sends all API calls to https://sentry.io unless you configure a different URL, and no SENTRY_URL was set anywhere in the repository. This looked decisive.

PR #140 set url: "https://de.sentry.io" in apps/journal/vite.config.ts and apps/planner/vite.config.ts. It merged as 7968b49. Run 1459 rebuilt the #24 build layer (the layer was not cached, so the new configuration applied), and the error did not change. The EU endpoint also rejects the token.

That line is correct and it stays, but it was not the cause.

Eliminated: ambiguous extraction in the workflow

cd-apps.yml reads the token with an unanchored match:

grep SENTRY_AUTH_TOKEN /tmp/secrets.env | cut -d= -f2- | tr -d '\n' > /tmp/sentry_token

Two matching keys would concatenate into one invalid value. Only one key in secrets.app.env contains that substring, so the extraction gives one value.

What remains

The token value itself is wrong. It is revoked, or mistyped, or it has quotation marks that cut -d= -f2- copies into the authorization header. All three give the same 401.

Recommended fix

Rotate the token instead of diagnosing it further. Make an Organization Auth Token (Sentry → Settings → Auth Tokens) and write it with sops infrastructure/secrets.app.env. This token type carries its own region URL and the necessary permissions, so it removes all remaining causes together. Make sure the value has no quotation marks around it.

To identify the cause before you rotate, decrypt the file and read the prefix: sntrys_ is an organization token, sntryu_ is a legacy user token, and any other prefix means the value is damaged.

The follow-up is more important than before

A failed upload still does not fail the build. sentry-vite-plugin reports the error and the build continues, and run 1459 finished green with two 401s in its log. This is why the problem stayed hidden for six weeks, and it is why it can happen again after the rotation. Please keep this issue open for that change.

## Status on 2026-09-11: still broken, and two causes are now eliminated The most recent production deploy (`cd-apps` run 1459, commit `7968b49`) shows the same error for the journal and for the planner: ``` sentry-cli releases new 7968b49546e806bfc46fbafe0505c0ff2aaadacd sentry reported an error: Invalid token (http status: 401) ``` The token has not changed since this issue was written. The ciphertext of `SENTRY_AUTH_TOKEN` in `infrastructure/secrets.app.env` last changed in `f575eae` (2026-07-25). It is the same at `main` today. ### Eliminated: the scope theory in the description above Insufficient scopes give **403 Forbidden**. This is **401 Invalid token**. The two are different failures. The description names scopes as the likely fix, so please do not start there. ### Eliminated: the data region The org is in the EU region. Every DSN in the repository has an `o4509530546634752.ingest.de.sentry.io` host. But `sentry-cli` sends all API calls to `https://sentry.io` unless you configure a different URL, and no `SENTRY_URL` was set anywhere in the repository. This looked decisive. PR #140 set `url: "https://de.sentry.io"` in `apps/journal/vite.config.ts` and `apps/planner/vite.config.ts`. It merged as `7968b49`. Run 1459 rebuilt the `#24` build layer (the layer was not cached, so the new configuration applied), and the error did not change. The EU endpoint also rejects the token. That line is correct and it stays, but it was not the cause. ### Eliminated: ambiguous extraction in the workflow `cd-apps.yml` reads the token with an unanchored match: ```bash grep SENTRY_AUTH_TOKEN /tmp/secrets.env | cut -d= -f2- | tr -d '\n' > /tmp/sentry_token ``` Two matching keys would concatenate into one invalid value. Only one key in `secrets.app.env` contains that substring, so the extraction gives one value. ### What remains The token value itself is wrong. It is revoked, or mistyped, or it has quotation marks that `cut -d= -f2-` copies into the authorization header. All three give the same 401. ### Recommended fix Rotate the token instead of diagnosing it further. Make an **Organization Auth Token** (Sentry → Settings → Auth Tokens) and write it with `sops infrastructure/secrets.app.env`. This token type carries its own region URL and the necessary permissions, so it removes all remaining causes together. Make sure the value has no quotation marks around it. To identify the cause before you rotate, decrypt the file and read the prefix: `sntrys_` is an organization token, `sntryu_` is a legacy user token, and any other prefix means the value is damaged. ### The follow-up is more important than before A failed upload still does not fail the build. `sentry-vite-plugin` reports the error and the build continues, and run 1459 finished green with two 401s in its log. This is why the problem stayed hidden for six weeks, and it is why it can happen again after the rotation. Please keep this issue open for that change.
ullrich commented 2026-09-11 12:10:17 +00:00
Author
Owner
Copy link

Root cause found: the token never reached the build

The token is correct. My two earlier theories were both wrong, and so was the original description. PR #143 has the fix.

docker/build-push-action has two different inputs:

  • secrets: takes literal values
  • secret-files: takes paths

Both CD workflows used the first one with a path:

secrets: |
  SENTRY_AUTH_TOKEN=/tmp/sentry_token

So the build received the 17-character string /tmp/sentry_token as the token itself. The Dockerfile read it out of /run/secrets/, sentry-cli sent it as the credential, and Sentry answered Invalid token (http status: 401). The answer was correct. The decrypted token never entered the build.

The build log shows it:

--secret id=SENTRY_AUTH_TOKEN,src=/tmp/docker-actions-toolkit-Y6sWb8/tmp-189-HUtub87R06pM

That src is a temp file that the action made for the literal value. With secret-files:, src points at /tmp/sentry_token itself.

The measurement that broke the deadlock

The stored token answers 200 on both endpoints:

sentry.io:    200
de.sentry.io: 200

Its value is 71 characters (sntryu_ + 64 hex), with no quotation marks. So the credential is good, the file is good, and the grep/cut extraction is good. Only the transport was left.

Full record of eliminated theories

Theory Verdict
Scopes too narrow (this issue's description) Scopes give 403, not 401
EU data region not configured #140 pinned de.sentry.io; the 401 did not change
Two keys concatenated by an unanchored grep Only one key matches in secrets.app.env
Token revoked, mistyped, or quoted 200 on both hosts; 71 characters; no quotes
secrets: used where secret-files: was needed Confirmed — PR #143

What #140 got wrong

That PR pinned url: "https://de.sentry.io" and said the region was the cause. It was not. The 401 did not move because the token was absent, whatever URL was used. #143 keeps the pin, because our org is in the EU and the pin costs nothing, but it rewrites the comment to say the pin is defensive.

This issue stays open

A failed upload still does not fail the build. Run 1459 finished green with two 401s in its log. Until that changes, the next broken upload will also go unseen. That follow-up is the remaining work here.

## Root cause found: the token never reached the build The token is correct. My two earlier theories were both wrong, and so was the original description. PR #143 has the fix. `docker/build-push-action` has two different inputs: - `secrets:` takes literal **values** - `secret-files:` takes **paths** Both CD workflows used the first one with a path: ```yaml secrets: | SENTRY_AUTH_TOKEN=/tmp/sentry_token ``` So the build received the 17-character string `/tmp/sentry_token` as the token itself. The Dockerfile read it out of `/run/secrets/`, `sentry-cli` sent it as the credential, and Sentry answered `Invalid token (http status: 401)`. The answer was correct. The decrypted token never entered the build. The build log shows it: ``` --secret id=SENTRY_AUTH_TOKEN,src=/tmp/docker-actions-toolkit-Y6sWb8/tmp-189-HUtub87R06pM ``` That `src` is a temp file that the action made for the literal value. With `secret-files:`, `src` points at `/tmp/sentry_token` itself. ### The measurement that broke the deadlock The stored token answers **200** on both endpoints: ``` sentry.io: 200 de.sentry.io: 200 ``` Its value is 71 characters (`sntryu_` + 64 hex), with no quotation marks. So the credential is good, the file is good, and the `grep`/`cut` extraction is good. Only the transport was left. ### Full record of eliminated theories | Theory | Verdict | |---|---| | Scopes too narrow (this issue's description) | Scopes give 403, not 401 | | EU data region not configured | #140 pinned `de.sentry.io`; the 401 did not change | | Two keys concatenated by an unanchored `grep` | Only one key matches in `secrets.app.env` | | Token revoked, mistyped, or quoted | 200 on both hosts; 71 characters; no quotes | | **`secrets:` used where `secret-files:` was needed** | **Confirmed — PR #143** | ### What #140 got wrong That PR pinned `url: "https://de.sentry.io"` and said the region was the cause. It was not. The 401 did not move because the token was absent, whatever URL was used. #143 keeps the pin, because our org is in the EU and the pin costs nothing, but it rewrites the comment to say the pin is defensive. ### This issue stays open A failed upload still does not fail the build. Run 1459 finished green with two 401s in its log. Until that changes, the next broken upload will also go unseen. That follow-up is the remaining work here.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
vitest-5-complete-set
caddy-mta-sts
chore/development-deps
fix/restore-openspec-1.10
fix/sentry-upload-loud-and-documented
No results found.
Labels
Clear labels   
dependencies
Dependency updates raised by Renovate (renovate.json5 labels)

  
major
Major version bump — always needs human review, never auto-merged

  
security-pin
Touches a pnpm.overrides CVE remediation pin, not an ordinary dependency

No labels
dependencies
major
security-pin
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
trails-cool/trails#5
No description provided.