promote: main → release (jinemi registry namespace, Caddy trusted_proxies, CI login fix) #159
No reviewers
Labels
No labels
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: Jinemi/thermograph#159
Loading…
Reference in a new issue
No description provided.
Delete branch "main"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Catches prod up with 26 commits. No application source changes, no migrations — the only file under
backend/orfrontend/that is not documentation or test-compose isbackend/scripts/smoke.sh, whose retired image path was corrected.What prod actually gains
jinemi/thermograph/*. Prod currently resolvesemi/through auser_redirectleft from renamingadmin_emi— a name that has to be freed. Until this lands, prod is the last environment holding that redirect open.trusted_proxies. The LB is the second Caddy hop and was overwritingX-Forwarded-Protowithhttp, so every OAuth redirect URI the backend built came out non-HTTPS. Discord tolerates that; Google rejects it outright, so Google sign-in cannot work in prod without this. The equivalent is already live on beta.${{ }}that silently dropped the registry-login step, which made every push anonymous.dev.jinemi.comwork, openbao parity gate and secrets-rendering changes.Pre-flight
thermograph/backend:sha-af21d8e47776andthermograph/frontend:sha-df409f88b3fd.deploy.ymldoes not wait onbuild-push, so this was checked rather than assumed.emi/thermograph/*:sha-deb039ee249f,thermograph.orgreturning 200 and/api/versionpayload_ver: p2. That tag exists in both namespaces, so a roll back is available either way.deploy-stack.shappliesdeploy/stack/lb/Caddyfile, so the LB change goes live with this deploy.Repos moved to the Jinemi org; the container packages did not follow, since Forgejo does not transfer packages with a repo. The deploy path still resolved `emi/thermograph/*` — a user_redirect to admin_emi — while build-push.yml derives its push path from ${github.repository}, now jinemi/thermograph. The next backend or frontend build would have published somewhere no deploy looks. Point the image paths at jinemi/thermograph/* (lowercase: OCI references admit no uppercase, which is why build-push.yml already pipes through tr), the clone URLs at Jinemi/thermograph, and the registry logins at admin_emi — the account that actually owns the tokens, rather than the redirect. The live tags and both ci-runner tags were copied into the Jinemi namespace first, so the switch has something to pull. thermograph-infra, thermograph-observability and the retired */app packages stay under admin_emi; they did not move.`docker login --password-stdin` strips exactly ONE trailing newline. `echo "$T"` adds one. So a secret pasted WITH a trailing newline -- which is what copying from a terminal, or an editor that terminates files with one, produces -- arrives at the registry as "<token>\n" and gets back: Error response from daemon: Get "https://.../v2/": denied: with no further detail. That is indistinguishable from a revoked or wrong token. On 2026-08-01 it stopped every build and deploy in the estate and cost an afternoon of diagnosis on a credential that was in fact valid: the same token, tested by hand, returned 200 on /v2/ and was issued a pull,push-scoped registry token for jinemi/thermograph/backend. So: strip leading/trailing whitespace and CR/LF before the pipe, and never `echo` a credential into stdin. The token now travels through the ENVIRONMENT rather than being interpolated into the script text. `${{ }}` is substituted before bash parses the line, so a value containing a quote or a newline changes the shape of the command itself, not merely its arguments. Two diagnostics, deliberately asymmetric: * empty/unset -> HARD FAILURE naming where to set it. There is no case where proceeding helps. * wrong shape -> WARNING only (length, and whether it is outside [0-9a-f]), then attempt the login anyway. A hard assertion on token format would block every build the day Forgejo changes that format, which is a worse failure than the one being prevented. The warning is enough to turn the registry's opaque "denied:" into a diagnosis. Neither diagnostic prints the value; only its length and character class. Note the username is not a factor: tested against the live registry, all of admin_emi, emi and jinemi authenticate identically with a valid token and are each issued a push-scoped token. Forgejo's container registry authenticates on the token, not the username.Forgejo evaluates ${{ }} expressions inside a step's `run:` script, comments included. An EMPTY expression is a parse error, and the runner's response is to drop the step -- no log line, no error, no failed status. The login step simply never ran, so every subsequent `docker push` went out anonymous. The registry then answers `unauthorized: reqPackageAccess` (or, depending on the client path, `no basic auth credentials`), which reads exactly like a revoked token or a missing scope. It is neither. Both are downstream of a comment. Isolated on one branch, one variable, back to back: * empty expression present -> both legs fail, no `Login Succeeded` in the log * empty expression removed -> both legs pass, `Login Succeeded` present, sha-df409f88b3fd published for backend and frontend Nothing was wrong with the credential. The token, its scope and whether it sat at repo or organization level were all ruled out first: pushes to jinemi/thermograph/* succeed by hand from vps1 and from the runner host, with matching and mismatched usernames, and the run log shows REGISTRY_TOKEN arriving in the job environment. Do not write a bare ${{ }} in a run block, in a comment or otherwise.Audited the five CLAUDE.md files and all twenty-one README.md files against the tree, machine-checking every in-repo path they name and verifying the testable claims against the live hosts. The one that matters is in the root file: dev was documented as reachable on the mesh at 10.10.0.2:8137. It is not, and never was from anywhere but vps1 — infra/docker-compose.yml binds the port to 127.0.0.1, and the address answers from neither vps2 nor vps1 itself. Anyone following it gets a connection refused with nothing to explain it. The rest are stale paths, several from the reunification: * assetlinks.json moved under frontend/static/ in the subtree merge; the TWA README kept the pre-merge path in both places it names it. Following it would put the file where nothing serves it and Android app-link verification would fail silently. * push.py and notify.py now live in backend/notifications/. * INFRA.md and deploy/stack/README have never existed in this repo, in any branch. * the Caddyfile is at deploy/stack/lb/Caddyfile. * three bare relative paths that resolve for a reader but not from the directory the file sits in: units.js is the frontend's, deploy.sh is infra's, entrypoint.sh is the backend's. Also records why mesh clients must pin the ROOT_URL host and not only the image host: the registry's bearer-token realm follows ROOT_URL, so pinning git.thermograph.org alone still sends the token request out the public route, where the /v2/* matcher returns 403 and docker falls back to anonymous. That surfaces as `unauthorized: reqPackageAccess`, indistinguishable from a bad credential. Verified true and left alone: the four-domain layout, both .claude runbooks, the absence of any domain-level .forgejo directory, the pinned compose project name, the deploy contract, prod's eight stack services, beta's five prefixed ones with no db of its own, dev's five, and every documented make target.