f25466ca76
All checks were successful
PR build (required check) / changes (pull_request) Successful in 9s
shell-lint / shellcheck (pull_request) Successful in 9s
secrets-guard / encrypted (pull_request) Successful in 11s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
Both runners exist and are up; these are the two defects found while verifying that. The desktop's systemd --user unit had Restart=on-failure. That is the same distinction that took Forgejo down for 27 hours on 2026-07-29 (docker-stack.yml records it for the db service): a daemon that exits 0 is not "finished successfully", and on-failure cannot tell that from a clean shutdown. The failure mode is silent — unit inactive (dead), runner offline in the UI, and every protected-branch merge blocked on a check nothing will produce. The StartLimit directives that bound that retry loop were in [Service], where systemd accepts them without complaint and ignores them; the live unit was running the 10s default rather than the intended 300s. Moved to [Unit], which is where they are read. runner-vps2/README told you to copy config.yaml next to docker-compose.yml, but the compose file mounts ./data:/data and loads --config /data/config.yaml, so a config there is invisible to the container — the daemon starts on defaults with no --add-host, and every registry push then fails as if the credential were wrong. vps2 had a stray copy at the documented path proving the instruction had been followed.
151 lines
6.4 KiB
Markdown
151 lines
6.4 KiB
Markdown
# The always-on Actions runner (vps2)
|
|
|
|
The runner CI actually depends on. The desktop runner
|
|
(`../register-lan-runner.sh`) stays registered as extra capacity, but nothing in
|
|
the estate may assume it is up.
|
|
|
|
Labels: `docker` only. `thermograph-lan` is deliberately **not** registered here
|
|
— it was the host-native LAN-deploy label, dev moved to vps1 and is deployed
|
|
over SSH like beta and prod, and no workflow requests it any more.
|
|
|
|
## Why this exists
|
|
|
|
On 2026-07-31 16:31Z the desktop — then the estate's **only** registered runner
|
|
— went offline. For the next 21 hours nothing merged (every protected branch
|
|
requires a check that only a runner can produce), nothing deployed, and the
|
|
03:00Z `ops-cron` did not run. Forgejo held that scheduled run in `waiting`
|
|
rather than dropping it, so all three jobs completed on reconnect and no backup
|
|
was actually lost. A longer outage would not have been so kind.
|
|
|
|
`docker-stack.yml` and this directory's parent README both described an
|
|
"always-on Swarm-hosted runner" during that entire window. There wasn't one; an
|
|
early revision had it as a Docker-in-Docker sidecar and it was removed. That is
|
|
why the single point of failure went unnoticed for weeks, and it is the reason
|
|
the root `CLAUDE.md` treats a stale doc as a correctness bug rather than a
|
|
documentation one.
|
|
|
|
## Prerequisites
|
|
|
|
- vps2's docker daemon is logged in to `git.thermograph.org` (it is —
|
|
`/root/.docker/config.json`), so the daemon can pull the job image.
|
|
- `git.thermograph.org` resolves to the **mesh** address from vps2 for registry
|
|
traffic. Job containers get this via `--add-host` in `config.yaml`; the host
|
|
daemon already has it.
|
|
- The job image exists: `git.thermograph.org/jinemi/thermograph/ci-runner:v2`.
|
|
Built from `../ci-runner/Dockerfile`; see that file for when to rebuild.
|
|
|
|
## Where it is deployed, and why not from the checkout
|
|
|
|
**`/opt/forgejo-runner/` on vps2**, holding a copy of `docker-compose.yml` and
|
|
`config.yaml` from this directory, plus a `data/` the runner owns.
|
|
|
|
Deliberately *not* run in place from `/opt/thermograph/…/runner-vps2/`. That
|
|
checkout is `git reset --hard`'d on every prod deploy; untracked files do
|
|
survive that (it is the same property `deploy/.image-tags.env` relies on), but a
|
|
single `git clean -fdx` during troubleshooting would delete `data/.runner` and
|
|
de-register the runner. Putting the thing that repairs the estate inside the
|
|
tree the estate rewrites is the same mistake as scheduling it on the Swarm it
|
|
deploys.
|
|
|
|
So this directory is the source of truth; vps2 gets a copy:
|
|
|
|
The two files do **not** go to the same place, and getting it wrong fails in a
|
|
confusing way. `docker-compose.yml` mounts `./data:/data` and starts the daemon
|
|
with `--config /data/config.yaml`, so `/data` *is* `data/` — a `config.yaml`
|
|
sitting beside the compose file is invisible to the container. The daemon then
|
|
either refuses to start or silently runs on defaults (no `--add-host`, so every
|
|
registry push fails as if the credential were wrong).
|
|
|
|
```bash
|
|
install -d -m 0755 /opt/forgejo-runner/data
|
|
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \
|
|
/opt/forgejo-runner/
|
|
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
|
|
/opt/forgejo-runner/data/
|
|
```
|
|
|
|
`data/` is also where `register` writes `.runner`, so the runner's whole state —
|
|
config plus registration — lives in the one directory the compose file mounts.
|
|
|
|
Re-copy after changing either file here, then `docker compose up -d`. There is
|
|
no automation for that step and there should not be: a workflow that redeploys
|
|
the runner runs *on* the runner.
|
|
|
|
## One-time registration
|
|
|
|
The registration token is short-lived and single-purpose. Fetch it from the
|
|
Forgejo UI (**Site Administration → Actions → Runners → Create new Runner**) or
|
|
the API, and keep it out of shell history — it is a credential, however
|
|
briefly.
|
|
|
|
```bash
|
|
cd /opt/forgejo-runner
|
|
mkdir -p data
|
|
cp config.yaml data/config.yaml
|
|
|
|
read -rs REG_TOKEN && export REG_TOKEN # paste; not echoed, not in history
|
|
|
|
docker run --rm -it \
|
|
-v "$PWD/data:/data" -w /data --user root \
|
|
code.forgejo.org/forgejo/runner:6.3.1 \
|
|
forgejo-runner register --no-interactive \
|
|
--instance https://git.thermograph.org \
|
|
--token "$REG_TOKEN" \
|
|
--name thermograph-vps2 \
|
|
--labels 'docker:docker://git.thermograph.org/jinemi/thermograph/ci-runner:v2'
|
|
|
|
unset REG_TOKEN
|
|
```
|
|
|
|
That writes `data/.runner`, which holds the **long-lived** runner token. It is
|
|
`0600` and must stay out of git — `data/` is gitignored for exactly this reason.
|
|
Re-running `register` with a fresh registration token replaces it; the old
|
|
registration then shows offline in the UI and should be deleted there.
|
|
|
|
## Run it
|
|
|
|
```bash
|
|
docker compose up -d
|
|
docker compose logs -f --tail=50 runner # expect "runner: ... successfully activated"
|
|
```
|
|
|
|
Confirm Forgejo agrees, rather than trusting the log:
|
|
|
|
```bash
|
|
# on vps1, where Forgejo's database lives
|
|
fdb=$(docker ps -qf name=forgejo_db | head -1)
|
|
docker exec "$fdb" sh -c 'psql -U $POSTGRES_USER -d $POSTGRES_DB -c \
|
|
"select name, to_timestamp(last_online) as last_online, agent_labels from action_runner order by id;"'
|
|
```
|
|
|
|
Two rows, both recent. `last_online` is the only honest liveness signal — a
|
|
runner process that is up but cannot reach Forgejo looks healthy locally and is
|
|
useless.
|
|
|
|
## Upgrading
|
|
|
|
Pin changes go in `docker-compose.yml` (`image:`) and are applied with
|
|
`docker compose up -d`. The runner token survives an image change; it lives in
|
|
`data/.runner`, not in the image.
|
|
|
|
Changing the **job** image (the `docker://…ci-runner:vN` label) is a
|
|
re-registration or a hand-edit of `data/.runner`'s `labels` array — the same
|
|
caveat `../ci-runner/Dockerfile` records for the desktop runner. Keep the two
|
|
runners on the same job image or a workflow's behaviour starts depending on
|
|
which runner happened to pick it up.
|
|
|
|
## Troubleshooting
|
|
|
|
**Jobs queue forever with both runners online.** The workflow asked for a label
|
|
nobody registered. `runs-on: docker` is the only label this estate uses.
|
|
|
|
**`docker pull` fails inside a job, works on the host.** The `--add-host` line
|
|
in `config.yaml` is missing or wrong. vps1's Caddy serves `/v2/*` to the mesh
|
|
only, and the job container has no other way to learn the mesh address.
|
|
|
|
**Jobs fail with permission denied on the socket.** `user: root` was dropped
|
|
from the compose file.
|
|
|
|
**The runner is up but `last_online` is stale.** It cannot reach
|
|
`https://git.thermograph.org` — check egress from vps2 and that Forgejo on vps1
|
|
is serving.
|