thermograph/infra/deploy/forgejo/runner-vps2/README.md

The always-on Actions runner (vps2)

The runner CI actually depends on. The desktop runner (../register-lan-runner.sh) stays registered as extra capacity, but nothing in the estate may assume it is up.

Labels: docker only. thermograph-lan is deliberately not registered here — it was the host-native LAN-deploy label, dev moved to vps1 and is deployed over SSH like beta and prod, and no workflow requests it any more.

Why this exists

On 2026-07-31 16:31Z the desktop — then the estate's only registered runner — went offline. For the next 21 hours nothing merged (every protected branch requires a check that only a runner can produce), nothing deployed, and the 03:00Z ops-cron did not run. Forgejo held that scheduled run in waiting rather than dropping it, so all three jobs completed on reconnect and no backup was actually lost. A longer outage would not have been so kind.

docker-stack.yml and this directory's parent README both described an "always-on Swarm-hosted runner" during that entire window. There wasn't one; an early revision had it as a Docker-in-Docker sidecar and it was removed. That is why the single point of failure went unnoticed for weeks, and it is the reason the root CLAUDE.md treats a stale doc as a correctness bug rather than a documentation one.

Prerequisites

• vps2's docker daemon is logged in to git.thermograph.org (it is — /root/.docker/config.json), so the daemon can pull the job image.
• git.thermograph.org resolves to the mesh address from vps2 for registry traffic. Job containers get this via --add-host in config.yaml; the host daemon already has it.
• The job image exists: git.thermograph.org/jinemi/thermograph/ci-runner:v2. Built from ../ci-runner/Dockerfile; see that file for when to rebuild.

Where it is deployed, and why not from the checkout

/opt/forgejo-runner/ on vps2, holding a copy of docker-compose.yml and config.yaml from this directory, plus a data/ the runner owns.

Deliberately not run in place from /opt/thermograph/…/runner-vps2/. That checkout is git reset --hard'd on every prod deploy; untracked files do survive that (it is the same property deploy/.image-tags.env relies on), but a single git clean -fdx during troubleshooting would delete data/.runner and de-register the runner. Putting the thing that repairs the estate inside the tree the estate rewrites is the same mistake as scheduling it on the Swarm it deploys.

So this directory is the source of truth; vps2 gets a copy:

The two files do not go to the same place, and getting it wrong fails in a confusing way. docker-compose.yml mounts ./data:/data and starts the daemon with --config /data/config.yaml, so /data is data/ — a config.yaml sitting beside the compose file is invisible to the container. The daemon then either refuses to start or silently runs on defaults (no --add-host, so every registry push fails as if the credential were wrong).

install -d -m 0755 /opt/forgejo-runner/data
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \
   /opt/forgejo-runner/
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
   /opt/forgejo-runner/data/

data/ is also where register writes .runner, so the runner's whole state — config plus registration — lives in the one directory the compose file mounts.

Re-copy after changing either file here, then docker compose up -d. There is no automation for that step and there should not be: a workflow that redeploys the runner runs on the runner.

One-time registration

The registration token is short-lived and single-purpose. Fetch it from the Forgejo UI (Site Administration → Actions → Runners → Create new Runner) or the API, and keep it out of shell history — it is a credential, however briefly.

cd /opt/forgejo-runner
mkdir -p data
cp config.yaml data/config.yaml

read -rs REG_TOKEN && export REG_TOKEN     # paste; not echoed, not in history

docker run --rm -it \
  -v "$PWD/data:/data" -w /data --user root \
  code.forgejo.org/forgejo/runner:6.3.1 \
  forgejo-runner register --no-interactive \
    --instance https://git.thermograph.org \
    --token "$REG_TOKEN" \
    --name thermograph-vps2 \
    --labels 'docker:docker://git.thermograph.org/jinemi/thermograph/ci-runner:v2'

unset REG_TOKEN

That writes data/.runner, which holds the long-lived runner token. It is 0600 and must stay out of git — data/ is gitignored for exactly this reason. Re-running register with a fresh registration token replaces it; the old registration then shows offline in the UI and should be deleted there.

Run it

docker compose up -d
docker compose logs -f --tail=50 runner     # expect "runner: ... successfully activated"

Confirm Forgejo agrees, rather than trusting the log:

# on vps1, where Forgejo's database lives
fdb=$(docker ps -qf name=forgejo_db | head -1)
docker exec "$fdb" sh -c 'psql -U $POSTGRES_USER -d $POSTGRES_DB -c \
  "select name, to_timestamp(last_online) as last_online, agent_labels from action_runner order by id;"'

Two rows, both recent. last_online is the only honest liveness signal — a runner process that is up but cannot reach Forgejo looks healthy locally and is useless.

Upgrading

Pin changes go in docker-compose.yml (image:) and are applied with docker compose up -d. The runner token survives an image change; it lives in data/.runner, not in the image.

Changing the job image (the docker://…ci-runner:vN label) is a re-registration or a hand-edit of data/.runner's labels array — the same caveat ../ci-runner/Dockerfile records for the desktop runner. Keep the two runners on the same job image or a workflow's behaviour starts depending on which runner happened to pick it up.

Troubleshooting

Jobs queue forever with both runners online. The workflow asked for a label nobody registered. runs-on: docker is the only label this estate uses.

docker pull fails inside a job, works on the host. The --add-host line in config.yaml is missing or wrong. vps1's Caddy serves /v2/* to the mesh only, and the job container has no other way to learn the mesh address.

Jobs fail with permission denied on the socket. user: root was dropped from the compose file.

The runner is up but last_online is stale. It cannot reach https://git.thermograph.org — check egress from vps2 and that Forgejo on vps1 is serving.