df409f88b3
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 4s
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / validate-observability (pull_request) Successful in 20s
PR build (required check) / build-frontend (pull_request) Successful in 37s
PR build (required check) / build-backend (pull_request) Successful in 51s
PR build (required check) / gate (pull_request) Successful in 1s
Repos moved to the Jinemi org; the container packages did not follow, since
Forgejo does not transfer packages with a repo. The deploy path still resolved
`emi/thermograph/*` — a user_redirect to admin_emi — while build-push.yml
derives its push path from ${github.repository}, now jinemi/thermograph. The
next backend or frontend build would have published somewhere no deploy looks.
Point the image paths at jinemi/thermograph/* (lowercase: OCI references admit
no uppercase, which is why build-push.yml already pipes through tr), the clone
URLs at Jinemi/thermograph, and the registry logins at admin_emi — the account
that actually owns the tokens, rather than the redirect.
The live tags and both ci-runner tags were copied into the Jinemi namespace
first, so the switch has something to pull. thermograph-infra,
thermograph-observability and the retired */app packages stay under admin_emi;
they did not move.
140 lines
5.8 KiB
Markdown
140 lines
5.8 KiB
Markdown
# The always-on Actions runner (vps2)
|
|
|
|
The runner CI actually depends on. The desktop runner
|
|
(`../register-lan-runner.sh`) stays registered as extra capacity, but nothing in
|
|
the estate may assume it is up.
|
|
|
|
Labels: `docker` only. `thermograph-lan` is deliberately **not** registered here
|
|
— it was the host-native LAN-deploy label, dev moved to vps1 and is deployed
|
|
over SSH like beta and prod, and no workflow requests it any more.
|
|
|
|
## Why this exists
|
|
|
|
On 2026-07-31 16:31Z the desktop — then the estate's **only** registered runner
|
|
— went offline. For the next 21 hours nothing merged (every protected branch
|
|
requires a check that only a runner can produce), nothing deployed, and the
|
|
03:00Z `ops-cron` did not run. Forgejo held that scheduled run in `waiting`
|
|
rather than dropping it, so all three jobs completed on reconnect and no backup
|
|
was actually lost. A longer outage would not have been so kind.
|
|
|
|
`docker-stack.yml` and this directory's parent README both described an
|
|
"always-on Swarm-hosted runner" during that entire window. There wasn't one; an
|
|
early revision had it as a Docker-in-Docker sidecar and it was removed. That is
|
|
why the single point of failure went unnoticed for weeks, and it is the reason
|
|
the root `CLAUDE.md` treats a stale doc as a correctness bug rather than a
|
|
documentation one.
|
|
|
|
## Prerequisites
|
|
|
|
- vps2's docker daemon is logged in to `git.thermograph.org` (it is —
|
|
`/root/.docker/config.json`), so the daemon can pull the job image.
|
|
- `git.thermograph.org` resolves to the **mesh** address from vps2 for registry
|
|
traffic. Job containers get this via `--add-host` in `config.yaml`; the host
|
|
daemon already has it.
|
|
- The job image exists: `git.thermograph.org/jinemi/thermograph/ci-runner:v2`.
|
|
Built from `../ci-runner/Dockerfile`; see that file for when to rebuild.
|
|
|
|
## Where it is deployed, and why not from the checkout
|
|
|
|
**`/opt/forgejo-runner/` on vps2**, holding a copy of `docker-compose.yml` and
|
|
`config.yaml` from this directory, plus a `data/` the runner owns.
|
|
|
|
Deliberately *not* run in place from `/opt/thermograph/…/runner-vps2/`. That
|
|
checkout is `git reset --hard`'d on every prod deploy; untracked files do
|
|
survive that (it is the same property `deploy/.image-tags.env` relies on), but a
|
|
single `git clean -fdx` during troubleshooting would delete `data/.runner` and
|
|
de-register the runner. Putting the thing that repairs the estate inside the
|
|
tree the estate rewrites is the same mistake as scheduling it on the Swarm it
|
|
deploys.
|
|
|
|
So this directory is the source of truth; vps2 gets a copy:
|
|
|
|
```bash
|
|
install -d -m 0755 /opt/forgejo-runner
|
|
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \
|
|
/opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
|
|
/opt/forgejo-runner/
|
|
```
|
|
|
|
Re-copy after changing either file here, then `docker compose up -d`. There is
|
|
no automation for that step and there should not be: a workflow that redeploys
|
|
the runner runs *on* the runner.
|
|
|
|
## One-time registration
|
|
|
|
The registration token is short-lived and single-purpose. Fetch it from the
|
|
Forgejo UI (**Site Administration → Actions → Runners → Create new Runner**) or
|
|
the API, and keep it out of shell history — it is a credential, however
|
|
briefly.
|
|
|
|
```bash
|
|
cd /opt/forgejo-runner
|
|
mkdir -p data
|
|
cp config.yaml data/config.yaml
|
|
|
|
read -rs REG_TOKEN && export REG_TOKEN # paste; not echoed, not in history
|
|
|
|
docker run --rm -it \
|
|
-v "$PWD/data:/data" -w /data --user root \
|
|
code.forgejo.org/forgejo/runner:6.3.1 \
|
|
forgejo-runner register --no-interactive \
|
|
--instance https://git.thermograph.org \
|
|
--token "$REG_TOKEN" \
|
|
--name thermograph-vps2 \
|
|
--labels 'docker:docker://git.thermograph.org/jinemi/thermograph/ci-runner:v2'
|
|
|
|
unset REG_TOKEN
|
|
```
|
|
|
|
That writes `data/.runner`, which holds the **long-lived** runner token. It is
|
|
`0600` and must stay out of git — `data/` is gitignored for exactly this reason.
|
|
Re-running `register` with a fresh registration token replaces it; the old
|
|
registration then shows offline in the UI and should be deleted there.
|
|
|
|
## Run it
|
|
|
|
```bash
|
|
docker compose up -d
|
|
docker compose logs -f --tail=50 runner # expect "runner: ... successfully activated"
|
|
```
|
|
|
|
Confirm Forgejo agrees, rather than trusting the log:
|
|
|
|
```bash
|
|
# on vps1, where Forgejo's database lives
|
|
fdb=$(docker ps -qf name=forgejo_db | head -1)
|
|
docker exec "$fdb" sh -c 'psql -U $POSTGRES_USER -d $POSTGRES_DB -c \
|
|
"select name, to_timestamp(last_online) as last_online, agent_labels from action_runner order by id;"'
|
|
```
|
|
|
|
Two rows, both recent. `last_online` is the only honest liveness signal — a
|
|
runner process that is up but cannot reach Forgejo looks healthy locally and is
|
|
useless.
|
|
|
|
## Upgrading
|
|
|
|
Pin changes go in `docker-compose.yml` (`image:`) and are applied with
|
|
`docker compose up -d`. The runner token survives an image change; it lives in
|
|
`data/.runner`, not in the image.
|
|
|
|
Changing the **job** image (the `docker://…ci-runner:vN` label) is a
|
|
re-registration or a hand-edit of `data/.runner`'s `labels` array — the same
|
|
caveat `../ci-runner/Dockerfile` records for the desktop runner. Keep the two
|
|
runners on the same job image or a workflow's behaviour starts depending on
|
|
which runner happened to pick it up.
|
|
|
|
## Troubleshooting
|
|
|
|
**Jobs queue forever with both runners online.** The workflow asked for a label
|
|
nobody registered. `runs-on: docker` is the only label this estate uses.
|
|
|
|
**`docker pull` fails inside a job, works on the host.** The `--add-host` line
|
|
in `config.yaml` is missing or wrong. vps1's Caddy serves `/v2/*` to the mesh
|
|
only, and the job container has no other way to learn the mesh address.
|
|
|
|
**Jobs fail with permission denied on the socket.** `user: root` was dropped
|
|
from the compose file.
|
|
|
|
**The runner is up but `last_online` is stale.** It cannot reach
|
|
`https://git.thermograph.org` — check egress from vps2 and that Forgejo on vps1
|
|
is serving.
|