001e6b1365
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 5s
shell-lint / shellcheck (pull_request) Successful in 8s
PR build (required check) / changes (pull_request) Successful in 13s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 7s
secrets-guard / encrypted (push) Successful in 7s
shell-lint / shellcheck (push) Successful in 7s
The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up. |
||
|---|---|---|
| .. | ||
| .gitignore | ||
| config.yaml | ||
| docker-compose.yml | ||
| README.md | ||
The always-on Actions runner (vps2)
The runner CI actually depends on. The desktop runner
(../register-lan-runner.sh) stays registered as extra capacity, but nothing in
the estate may assume it is up.
Labels: docker only. thermograph-lan is deliberately not registered here
— it was the host-native LAN-deploy label, dev moved to vps1 and is deployed
over SSH like beta and prod, and no workflow requests it any more.
Why this exists
On 2026-07-31 16:31Z the desktop — then the estate's only registered runner
— went offline. For the next 21 hours nothing merged (every protected branch
requires a check that only a runner can produce), nothing deployed, and the
03:00Z ops-cron did not run. Forgejo held that scheduled run in waiting
rather than dropping it, so all three jobs completed on reconnect and no backup
was actually lost. A longer outage would not have been so kind.
docker-stack.yml and this directory's parent README both described an
"always-on Swarm-hosted runner" during that entire window. There wasn't one; an
early revision had it as a Docker-in-Docker sidecar and it was removed. That is
why the single point of failure went unnoticed for weeks, and it is the reason
the root CLAUDE.md treats a stale doc as a correctness bug rather than a
documentation one.
Prerequisites
- vps2's docker daemon is logged in to
git.thermograph.org(it is —/root/.docker/config.json), so the daemon can pull the job image. git.thermograph.orgresolves to the mesh address from vps2 for registry traffic. Job containers get this via--add-hostinconfig.yaml; the host daemon already has it.- The job image exists:
git.thermograph.org/emi/thermograph/ci-runner:v2. Built from../ci-runner/Dockerfile; see that file for when to rebuild.
Where it is deployed, and why not from the checkout
/opt/forgejo-runner/ on vps2, holding a copy of docker-compose.yml and
config.yaml from this directory, plus a data/ the runner owns.
Deliberately not run in place from /opt/thermograph/…/runner-vps2/. That
checkout is git reset --hard'd on every prod deploy; untracked files do
survive that (it is the same property deploy/.image-tags.env relies on), but a
single git clean -fdx during troubleshooting would delete data/.runner and
de-register the runner. Putting the thing that repairs the estate inside the
tree the estate rewrites is the same mistake as scheduling it on the Swarm it
deploys.
So this directory is the source of truth; vps2 gets a copy:
install -d -m 0755 /opt/forgejo-runner
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \
/opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
/opt/forgejo-runner/
Re-copy after changing either file here, then docker compose up -d. There is
no automation for that step and there should not be: a workflow that redeploys
the runner runs on the runner.
One-time registration
The registration token is short-lived and single-purpose. Fetch it from the Forgejo UI (Site Administration → Actions → Runners → Create new Runner) or the API, and keep it out of shell history — it is a credential, however briefly.
cd /opt/forgejo-runner
mkdir -p data
cp config.yaml data/config.yaml
read -rs REG_TOKEN && export REG_TOKEN # paste; not echoed, not in history
docker run --rm -it \
-v "$PWD/data:/data" -w /data --user root \
code.forgejo.org/forgejo/runner:6.3.1 \
forgejo-runner register --no-interactive \
--instance https://git.thermograph.org \
--token "$REG_TOKEN" \
--name thermograph-vps2 \
--labels 'docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v2'
unset REG_TOKEN
That writes data/.runner, which holds the long-lived runner token. It is
0600 and must stay out of git — data/ is gitignored for exactly this reason.
Re-running register with a fresh registration token replaces it; the old
registration then shows offline in the UI and should be deleted there.
Run it
docker compose up -d
docker compose logs -f --tail=50 runner # expect "runner: ... successfully activated"
Confirm Forgejo agrees, rather than trusting the log:
# on vps1, where Forgejo's database lives
fdb=$(docker ps -qf name=forgejo_db | head -1)
docker exec "$fdb" sh -c 'psql -U $POSTGRES_USER -d $POSTGRES_DB -c \
"select name, to_timestamp(last_online) as last_online, agent_labels from action_runner order by id;"'
Two rows, both recent. last_online is the only honest liveness signal — a
runner process that is up but cannot reach Forgejo looks healthy locally and is
useless.
Upgrading
Pin changes go in docker-compose.yml (image:) and are applied with
docker compose up -d. The runner token survives an image change; it lives in
data/.runner, not in the image.
Changing the job image (the docker://…ci-runner:vN label) is a
re-registration or a hand-edit of data/.runner's labels array — the same
caveat ../ci-runner/Dockerfile records for the desktop runner. Keep the two
runners on the same job image or a workflow's behaviour starts depending on
which runner happened to pick it up.
Troubleshooting
Jobs queue forever with both runners online. The workflow asked for a label
nobody registered. runs-on: docker is the only label this estate uses.
docker pull fails inside a job, works on the host. The --add-host line
in config.yaml is missing or wrong. vps1's Caddy serves /v2/* to the mesh
only, and the job container has no other way to learn the mesh address.
Jobs fail with permission denied on the socket. user: root was dropped
from the compose file.
The runner is up but last_online is stale. It cannot reach
https://git.thermograph.org — check egress from vps2 and that Forgejo on vps1
is serving.