thermograph/infra/deploy/forgejo/runner-vps2/README.md

# The always-on Actions runner (vps2)

The runner CI actually depends on. The desktop runner
(`../register-lan-runner.sh`) stays registered as extra capacity, but nothing in
the estate may assume it is up.

Labels: `docker` only. `thermograph-lan` is deliberately **not** registered here
— it was the host-native LAN-deploy label, dev moved to vps1 and is deployed
over SSH like beta and prod, and no workflow requests it any more.

## Why this exists

On 2026-07-31 16:31Z the desktop — then the estate's **only** registered runner
— went offline. For the next 21 hours nothing merged (every protected branch
requires a check that only a runner can produce), nothing deployed, and the
03:00Z `ops-cron` did not run. Forgejo held that scheduled run in `waiting`
rather than dropping it, so all three jobs completed on reconnect and no backup
was actually lost. A longer outage would not have been so kind.

`docker-stack.yml` and this directory's parent README both described an
"always-on Swarm-hosted runner" during that entire window. There wasn't one; an
early revision had it as a Docker-in-Docker sidecar and it was removed. That is
why the single point of failure went unnoticed for weeks, and it is the reason
the root `CLAUDE.md` treats a stale doc as a correctness bug rather than a
documentation one.

## Prerequisites

- vps2's docker daemon is logged in to `dev.jinemi.com`
  (`/root/.docker/config.json`), so the daemon can pull the job image. A login
  against the old `git.thermograph.org` does not carry over — docker keys
  credentials by registry host, so this needs its own `docker login`.
- `dev.jinemi.com` resolves to the **mesh** address from vps2 for registry
  traffic. Job containers get this via `--add-host` in `config.yaml`; the host
  daemon needs the same pin in `/etc/hosts`.
- The job image exists: `dev.jinemi.com/jinemi/thermograph/ci-runner:v2`.
  Built from `../ci-runner/Dockerfile`; see that file for when to rebuild.

## Where it is deployed, and why not from the checkout

**`/opt/forgejo-runner/` on vps2**, holding a copy of `docker-compose.yml` and
`config.yaml` from this directory, plus a `data/` the runner owns.

Deliberately *not* run in place from `/opt/thermograph/…/runner-vps2/`. That
checkout is `git reset --hard`'d on every prod deploy; untracked files do
survive that (it is the same property `deploy/.image-tags.env` relies on), but a
single `git clean -fdx` during troubleshooting would delete `data/.runner` and
de-register the runner. Putting the thing that repairs the estate inside the
tree the estate rewrites is the same mistake as scheduling it on the Swarm it
deploys.

So this directory is the source of truth; vps2 gets a copy:

The two files do **not** go to the same place, and getting it wrong fails in a
confusing way. `docker-compose.yml` mounts `./data:/data` and starts the daemon
with `--config /data/config.yaml`, so `/data` *is* `data/` — a `config.yaml`
sitting beside the compose file is invisible to the container. The daemon then
either refuses to start or silently runs on defaults (no `--add-host`, so every
registry push fails as if the credential were wrong).

```bash
install -d -m 0755 /opt/forgejo-runner/data
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \
   /opt/forgejo-runner/
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
   /opt/forgejo-runner/data/
```

`data/` is also where `register` writes `.runner`, so the runner's whole state —
config plus registration — lives in the one directory the compose file mounts.

Re-copy after changing either file here, then `docker compose up -d`. There is
no automation for that step and there should not be: a workflow that redeploys
the runner runs *on* the runner.

## One-time registration

The registration token is short-lived and single-purpose. Fetch it from the
Forgejo UI (**Site Administration → Actions → Runners → Create new Runner**) or
the API, and keep it out of shell history — it is a credential, however
briefly.

```bash
cd /opt/forgejo-runner
mkdir -p data
cp config.yaml data/config.yaml

read -rs REG_TOKEN && export REG_TOKEN     # paste; not echoed, not in history

docker run --rm -it \
  -v "$PWD/data:/data" -w /data --user root \
  code.forgejo.org/forgejo/runner:6.3.1 \
  forgejo-runner register --no-interactive \
    --instance https://dev.jinemi.com \
    --token "$REG_TOKEN" \
    --name thermograph-vps2 \
    --labels 'docker:docker://dev.jinemi.com/jinemi/thermograph/ci-runner:v2'

unset REG_TOKEN
```

The runner registered before the domain migration holds
`https://git.thermograph.org` as its instance URL and keeps working while Caddy
serves that name (it does — see `../caddy-git.conf`). Re-registering with the
command above is how it moves to the canonical name; it is not required, and it
costs a fresh registration token and a runner restart.

That writes `data/.runner`, which holds the **long-lived** runner token. It is
`0600` and must stay out of git — `data/` is gitignored for exactly this reason.
Re-running `register` with a fresh registration token replaces it; the old
registration then shows offline in the UI and should be deleted there.

## Run it

```bash
docker compose up -d
docker compose logs -f --tail=50 runner     # expect "runner: ... successfully activated"
```

Confirm Forgejo agrees, rather than trusting the log:

```bash
# on vps1, where Forgejo's database lives
fdb=$(docker ps -qf name=forgejo_db | head -1)
docker exec "$fdb" sh -c 'psql -U $POSTGRES_USER -d $POSTGRES_DB -c \
  "select name, to_timestamp(last_online) as last_online, agent_labels from action_runner order by id;"'
```

Two rows, both recent. `last_online` is the only honest liveness signal — a
runner process that is up but cannot reach Forgejo looks healthy locally and is
useless.

## Upgrading

Pin changes go in `docker-compose.yml` (`image:`) and are applied with
`docker compose up -d`. The runner token survives an image change; it lives in
`data/.runner`, not in the image.

Changing the **job** image (the `docker://…ci-runner:vN` label) is a
re-registration or a hand-edit of `data/.runner`'s `labels` array — the same
caveat `../ci-runner/Dockerfile` records for the desktop runner. Keep the two
runners on the same job image or a workflow's behaviour starts depending on
which runner happened to pick it up.

## Troubleshooting

**Jobs queue forever with both runners online.** The workflow asked for a label
nobody registered. `runs-on: docker` is the only label this estate uses.

**`docker pull` fails inside a job, works on the host.** The `--add-host` line
in `config.yaml` is missing or wrong. vps1's Caddy serves `/v2/*` to the mesh
only, and the job container has no other way to learn the mesh address.

**Jobs fail with permission denied on the socket.** `user: root` was dropped
from the compose file.

**The runner is up but `last_online` is stale.** It cannot reach
the instance URL it registered with (`https://dev.jinemi.com`, or
`https://git.thermograph.org` for a runner registered before the migration) —
check egress from vps2 and that Forgejo on vps1 is serving that name.