001e6b1365
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 5s
shell-lint / shellcheck (pull_request) Successful in 8s
PR build (required check) / changes (pull_request) Successful in 13s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 7s
secrets-guard / encrypted (push) Successful in 7s
shell-lint / shellcheck (push) Successful in 7s
The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up.
140 lines
5.8 KiB
Markdown
140 lines
5.8 KiB
Markdown
# The always-on Actions runner (vps2)
|
|
|
|
The runner CI actually depends on. The desktop runner
|
|
(`../register-lan-runner.sh`) stays registered as extra capacity, but nothing in
|
|
the estate may assume it is up.
|
|
|
|
Labels: `docker` only. `thermograph-lan` is deliberately **not** registered here
|
|
— it was the host-native LAN-deploy label, dev moved to vps1 and is deployed
|
|
over SSH like beta and prod, and no workflow requests it any more.
|
|
|
|
## Why this exists
|
|
|
|
On 2026-07-31 16:31Z the desktop — then the estate's **only** registered runner
|
|
— went offline. For the next 21 hours nothing merged (every protected branch
|
|
requires a check that only a runner can produce), nothing deployed, and the
|
|
03:00Z `ops-cron` did not run. Forgejo held that scheduled run in `waiting`
|
|
rather than dropping it, so all three jobs completed on reconnect and no backup
|
|
was actually lost. A longer outage would not have been so kind.
|
|
|
|
`docker-stack.yml` and this directory's parent README both described an
|
|
"always-on Swarm-hosted runner" during that entire window. There wasn't one; an
|
|
early revision had it as a Docker-in-Docker sidecar and it was removed. That is
|
|
why the single point of failure went unnoticed for weeks, and it is the reason
|
|
the root `CLAUDE.md` treats a stale doc as a correctness bug rather than a
|
|
documentation one.
|
|
|
|
## Prerequisites
|
|
|
|
- vps2's docker daemon is logged in to `git.thermograph.org` (it is —
|
|
`/root/.docker/config.json`), so the daemon can pull the job image.
|
|
- `git.thermograph.org` resolves to the **mesh** address from vps2 for registry
|
|
traffic. Job containers get this via `--add-host` in `config.yaml`; the host
|
|
daemon already has it.
|
|
- The job image exists: `git.thermograph.org/emi/thermograph/ci-runner:v2`.
|
|
Built from `../ci-runner/Dockerfile`; see that file for when to rebuild.
|
|
|
|
## Where it is deployed, and why not from the checkout
|
|
|
|
**`/opt/forgejo-runner/` on vps2**, holding a copy of `docker-compose.yml` and
|
|
`config.yaml` from this directory, plus a `data/` the runner owns.
|
|
|
|
Deliberately *not* run in place from `/opt/thermograph/…/runner-vps2/`. That
|
|
checkout is `git reset --hard`'d on every prod deploy; untracked files do
|
|
survive that (it is the same property `deploy/.image-tags.env` relies on), but a
|
|
single `git clean -fdx` during troubleshooting would delete `data/.runner` and
|
|
de-register the runner. Putting the thing that repairs the estate inside the
|
|
tree the estate rewrites is the same mistake as scheduling it on the Swarm it
|
|
deploys.
|
|
|
|
So this directory is the source of truth; vps2 gets a copy:
|
|
|
|
```bash
|
|
install -d -m 0755 /opt/forgejo-runner
|
|
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \
|
|
/opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
|
|
/opt/forgejo-runner/
|
|
```
|
|
|
|
Re-copy after changing either file here, then `docker compose up -d`. There is
|
|
no automation for that step and there should not be: a workflow that redeploys
|
|
the runner runs *on* the runner.
|
|
|
|
## One-time registration
|
|
|
|
The registration token is short-lived and single-purpose. Fetch it from the
|
|
Forgejo UI (**Site Administration → Actions → Runners → Create new Runner**) or
|
|
the API, and keep it out of shell history — it is a credential, however
|
|
briefly.
|
|
|
|
```bash
|
|
cd /opt/forgejo-runner
|
|
mkdir -p data
|
|
cp config.yaml data/config.yaml
|
|
|
|
read -rs REG_TOKEN && export REG_TOKEN # paste; not echoed, not in history
|
|
|
|
docker run --rm -it \
|
|
-v "$PWD/data:/data" -w /data --user root \
|
|
code.forgejo.org/forgejo/runner:6.3.1 \
|
|
forgejo-runner register --no-interactive \
|
|
--instance https://git.thermograph.org \
|
|
--token "$REG_TOKEN" \
|
|
--name thermograph-vps2 \
|
|
--labels 'docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v2'
|
|
|
|
unset REG_TOKEN
|
|
```
|
|
|
|
That writes `data/.runner`, which holds the **long-lived** runner token. It is
|
|
`0600` and must stay out of git — `data/` is gitignored for exactly this reason.
|
|
Re-running `register` with a fresh registration token replaces it; the old
|
|
registration then shows offline in the UI and should be deleted there.
|
|
|
|
## Run it
|
|
|
|
```bash
|
|
docker compose up -d
|
|
docker compose logs -f --tail=50 runner # expect "runner: ... successfully activated"
|
|
```
|
|
|
|
Confirm Forgejo agrees, rather than trusting the log:
|
|
|
|
```bash
|
|
# on vps1, where Forgejo's database lives
|
|
fdb=$(docker ps -qf name=forgejo_db | head -1)
|
|
docker exec "$fdb" sh -c 'psql -U $POSTGRES_USER -d $POSTGRES_DB -c \
|
|
"select name, to_timestamp(last_online) as last_online, agent_labels from action_runner order by id;"'
|
|
```
|
|
|
|
Two rows, both recent. `last_online` is the only honest liveness signal — a
|
|
runner process that is up but cannot reach Forgejo looks healthy locally and is
|
|
useless.
|
|
|
|
## Upgrading
|
|
|
|
Pin changes go in `docker-compose.yml` (`image:`) and are applied with
|
|
`docker compose up -d`. The runner token survives an image change; it lives in
|
|
`data/.runner`, not in the image.
|
|
|
|
Changing the **job** image (the `docker://…ci-runner:vN` label) is a
|
|
re-registration or a hand-edit of `data/.runner`'s `labels` array — the same
|
|
caveat `../ci-runner/Dockerfile` records for the desktop runner. Keep the two
|
|
runners on the same job image or a workflow's behaviour starts depending on
|
|
which runner happened to pick it up.
|
|
|
|
## Troubleshooting
|
|
|
|
**Jobs queue forever with both runners online.** The workflow asked for a label
|
|
nobody registered. `runs-on: docker` is the only label this estate uses.
|
|
|
|
**`docker pull` fails inside a job, works on the host.** The `--add-host` line
|
|
in `config.yaml` is missing or wrong. vps1's Caddy serves `/v2/*` to the mesh
|
|
only, and the job container has no other way to learn the mesh address.
|
|
|
|
**Jobs fail with permission denied on the socket.** `user: root` was dropped
|
|
from the compose file.
|
|
|
|
**The runner is up but `last_online` is stale.** It cannot reach
|
|
`https://git.thermograph.org` — check egress from vps2 and that Forgejo on vps1
|
|
is serving.
|