Both runners exist and are up; these are the two defects found while verifying
that.
The desktop's systemd --user unit had Restart=on-failure. That is the same
distinction that took Forgejo down for 27 hours on 2026-07-29 (docker-stack.yml
records it for the db service): a daemon that exits 0 is not "finished
successfully", and on-failure cannot tell that from a clean shutdown. The
failure mode is silent — unit inactive (dead), runner offline in the UI, and
every protected-branch merge blocked on a check nothing will produce.
The StartLimit directives that bound that retry loop were in [Service], where
systemd accepts them without complaint and ignores them; the live unit was
running the 10s default rather than the intended 300s. Moved to [Unit], which
is where they are read.
runner-vps2/README told you to copy config.yaml next to docker-compose.yml, but
the compose file mounts ./data:/data and loads --config /data/config.yaml, so a
config there is invisible to the container — the daemon starts on defaults with
no --add-host, and every registry push then fails as if the credential were
wrong. vps2 had a stray copy at the documented path proving the instruction had
been followed.
Audited the five CLAUDE.md files and all twenty-one README.md files against the
tree, machine-checking every in-repo path they name and verifying the testable
claims against the live hosts.
The one that matters is in the root file: dev was documented as reachable on
the mesh at 10.10.0.2:8137. It is not, and never was from anywhere but vps1 —
infra/docker-compose.yml binds the port to 127.0.0.1, and the address answers
from neither vps2 nor vps1 itself. Anyone following it gets a connection
refused with nothing to explain it.
The rest are stale paths, several from the reunification:
* assetlinks.json moved under frontend/static/ in the subtree merge; the TWA
README kept the pre-merge path in both places it names it. Following it
would put the file where nothing serves it and Android app-link
verification would fail silently.
* push.py and notify.py now live in backend/notifications/.
* INFRA.md and deploy/stack/README have never existed in this repo, in any
branch.
* the Caddyfile is at deploy/stack/lb/Caddyfile.
* three bare relative paths that resolve for a reader but not from the
directory the file sits in: units.js is the frontend's, deploy.sh is
infra's, entrypoint.sh is the backend's.
Also records why mesh clients must pin the ROOT_URL host and not only the image
host: the registry's bearer-token realm follows ROOT_URL, so pinning
git.thermograph.org alone still sends the token request out the public route,
where the /v2/* matcher returns 403 and docker falls back to anonymous. That
surfaces as `unauthorized: reqPackageAccess`, indistinguishable from a bad
credential.
Verified true and left alone: the four-domain layout, both .claude runbooks,
the absence of any domain-level .forgejo directory, the pinned compose project
name, the deploy contract, prod's eight stack services, beta's five prefixed
ones with no db of its own, dev's five, and every documented make target.
This README said, in bold, that a runner must never go on prod or beta. A
runner has been registered and running on vps2 since 2026-08-01. Both
statements cannot stand: an instruction the estate visibly ignores teaches
readers to ignore the next one too.
The original objection is kept rather than deleted, because it is correct on
its merits -- docker_host: automount hands job containers the host's Docker
socket, which on vps2 is root over both the prod and beta stacks. What changed
is that the alternative proved worse. The desktop was the ONLY registered
runner in the estate; when it dropped on 2026-07-31 nothing merged, nothing
deployed and the nightly backup did not fire for 21 hours. On a repo where
every branch is protected and every change is a PR, one absent runner freezes
everything, and the backup hangs off the same path.
So the section now records the trade and the bounds actually applied --
capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty, no
thermograph network -- and states plainly that this is defence against
accident, not against a hostile workflow author. Forgejo's database already
stored VPS2_SSH_KEY, which is root on that box; what changed is that the
material is now reachable by a job rather than only at rest.
It also keeps the original advice for the case it was actually written for:
if you are adding CAPACITY rather than REDUNDANCY, raise capacity or use a box
that hosts nothing.
Forgejo derives every absolute URL from its single ROOT_URL, so this is what
moves clone URLs, the Google OAuth callback, webhook payload URLs and mail
links onto the new domain. Both redirect URIs are registered on the Google
OAuth client and were verified against Google before the flip — login is
SSO-only, so a mismatch locks everyone out of the UI including out of undoing
it. README documents the probe so the check is repeatable rather than assumed.
git.thermograph.org stays on the same Caddy site block and stays load-bearing:
the registry host baked into image names, the mesh /etc/hosts pins that make
the /v2/* matcher see a mesh source IP, and the runners' registered instance
URL all still use it. Renaming the registry is a separate migration; the docs
now say so where someone would otherwise retire the name as dead weight.
The stack has no auto-deploy, so this takes effect on a manual
`docker stack deploy` from the manager.
Repos moved to the Jinemi org; the container packages did not follow, since
Forgejo does not transfer packages with a repo. The deploy path still resolved
`emi/thermograph/*` — a user_redirect to admin_emi — while build-push.yml
derives its push path from ${github.repository}, now jinemi/thermograph. The
next backend or frontend build would have published somewhere no deploy looks.
Point the image paths at jinemi/thermograph/* (lowercase: OCI references admit
no uppercase, which is why build-push.yml already pipes through tr), the clone
URLs at Jinemi/thermograph, and the registry logins at admin_emi — the account
that actually owns the tokens, rather than the redirect.
The live tags and both ci-runner tags were copied into the Jinemi namespace
first, so the switch has something to pull. thermograph-infra,
thermograph-observability and the retired */app packages stay under admin_emi;
they did not move.
Both hostnames go on a single Caddy site block. That is load-bearing rather
than cosmetic: the /v2/* mesh-only matcher that keeps the OCI registry API
off the public internet (hazard #15) is scoped per block, so a separate
block for the new name would serve the same Forgejo with the registry open.
git.thermograph.org stays canonical — Forgejo has one ROOT_URL and builds
every absolute URL from it, so clone URLs, the OAuth callback and post-login
redirects keep naming the .org host. Flipping FORGEJO_DOMAIN additionally
requires the Google OAuth redirect URI, the registry host baked into image
names and runner labels, and the mesh /etc/hosts pins; README documents
those prerequisites.
Config verified with `caddy validate` and by asserting against the adapted
JSON that the /v2 403 guard matches both hostnames.
The estate had exactly one registered runner, on the desktop. It went offline
at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required
check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both
application databases and for Forgejo -- did not fire. Forgejo queued that
scheduled run rather than dropping it, so it completed on reconnect and nothing
was lost. A longer outage would have meant real gaps.
Three files claimed an "always-on Swarm-hosted runner" existed and that the
estate therefore no longer depended on the desktop. It did not exist: an early
revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was
removed. That claim is why a single point of failure sat unnoticed. Corrected
in docker-stack.yml, forgejo/README.md and register-lan-runner.sh.
The new runner is a plain restart:always container, not a Swarm service: a
Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would
be gone exactly when the cluster is what is broken. It runs from
/opt/forgejo-runner rather than in place, because the checkout is reset on
every prod deploy and one `git clean -fdx` there would destroy the
registration.
vps2 runs prod, so the socket mount is bounded rather than assumed benign:
capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no
job can bind-mount /etc/thermograph.env, and no thermograph network joined.
This defends against accident, not against a hostile workflow author -- stated
plainly in the compose header rather than implied.
The desktop runner stays registered as extra capacity. Nothing may assume it
is up.
forgejo_db has been at 0/1 replicas since 2026-07-29 01:24 UTC. Postgres hit
an invalid data-directory lock file ("could not open file postmaster.pid ...
performing immediate shutdown because data directory lock file is invalid")
and exited 0. With restart_policy.condition=on-failure, Swarm read the zero
status as successful completion, marked the task Complete, and never
rescheduled it.
The forgejo service itself stayed Up and kept serving its homepage, so the
outage presented as every repository page, the whole API and all CI returning
500 with "dial tcp: lookup db on 127.0.0.11:53: no such host" — including the
auth path, which is why API calls reported "user does not exist [uid: 0]"
rather than a database error.
on-failure cannot distinguish "finished successfully" from "shut itself down
and should be restarted", and Postgres exits 0 on several such paths, so it is
the wrong policy for an always-on stateful service.
This is the durable fix; it does not restart the currently stopped task.
