53e2eb7c84
All checks were successful
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-centralis (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 9s
secrets-guard / encrypted (push) Successful in 5s
Validate observability stack / validate (push) Successful in 12s
shell-lint / shellcheck (push) Successful in 8s
Build + push images (Forgejo registry) / build-push (frontend) (push) Successful in 26s
Build + push images (Forgejo registry) / build-push (backend) (push) Successful in 1m10s
Deploy / deploy (backend) (push) Successful in 1m41s
Deploy / deploy (frontend) (push) Successful in 1m48s
secrets-guard / encrypted (pull_request) Successful in 8s
shell-lint / shellcheck (pull_request) Successful in 9s
PR build (required check) / changes (pull_request) Successful in 20s
PR build (required check) / validate-observability (pull_request) Successful in 15s
PR build (required check) / build-frontend (pull_request) Successful in 17s
PR build (required check) / build-backend (pull_request) Successful in 2m26s
PR build (required check) / gate (pull_request) Successful in 1s
Completes the Forgejo domain migration. ROOT_URL moved to dev.jinemi.com earlier; the registry half was deliberately deferred. Every image name, REGISTRY_HOST default, runner label and --add-host pin now names dev.jinemi.com, so the registry host and the bearer-token realm agree again. Both names address the same Forgejo, so no image needs re-pushing and a rollback to a tag pushed under the old prefix still resolves. git.thermograph.org therefore stays served off the same Caddy site block -- one block, so the /v2/* mesh-only matcher keeps covering both names -- for pre-migration tags and for runners holding it as their registered instance URL. Mesh clients now pin both names in /etc/hosts: the new one as registry host and token realm, the old one for pre-migration tags. runner-vps2/config.yaml carries both --add-host entries for the same reason. Also renames Centralis' endpoint to mcp.jinemi.com in the two places this repo names it; Centralis itself is provisioned outside this repo. Host-side steps this cannot do (documented in deploy/forgejo/README.md, "Host-side steps"): the Forgejo Actions variable REGISTRY_HOST, docker login against the new host, and the /etc/hosts pins.
160 lines
7 KiB
Markdown
160 lines
7 KiB
Markdown
# The always-on Actions runner (vps2)
|
|
|
|
The runner CI actually depends on. The desktop runner
|
|
(`../register-lan-runner.sh`) stays registered as extra capacity, but nothing in
|
|
the estate may assume it is up.
|
|
|
|
Labels: `docker` only. `thermograph-lan` is deliberately **not** registered here
|
|
— it was the host-native LAN-deploy label, dev moved to vps1 and is deployed
|
|
over SSH like beta and prod, and no workflow requests it any more.
|
|
|
|
## Why this exists
|
|
|
|
On 2026-07-31 16:31Z the desktop — then the estate's **only** registered runner
|
|
— went offline. For the next 21 hours nothing merged (every protected branch
|
|
requires a check that only a runner can produce), nothing deployed, and the
|
|
03:00Z `ops-cron` did not run. Forgejo held that scheduled run in `waiting`
|
|
rather than dropping it, so all three jobs completed on reconnect and no backup
|
|
was actually lost. A longer outage would not have been so kind.
|
|
|
|
`docker-stack.yml` and this directory's parent README both described an
|
|
"always-on Swarm-hosted runner" during that entire window. There wasn't one; an
|
|
early revision had it as a Docker-in-Docker sidecar and it was removed. That is
|
|
why the single point of failure went unnoticed for weeks, and it is the reason
|
|
the root `CLAUDE.md` treats a stale doc as a correctness bug rather than a
|
|
documentation one.
|
|
|
|
## Prerequisites
|
|
|
|
- vps2's docker daemon is logged in to `dev.jinemi.com`
|
|
(`/root/.docker/config.json`), so the daemon can pull the job image. A login
|
|
against the old `git.thermograph.org` does not carry over — docker keys
|
|
credentials by registry host, so this needs its own `docker login`.
|
|
- `dev.jinemi.com` resolves to the **mesh** address from vps2 for registry
|
|
traffic. Job containers get this via `--add-host` in `config.yaml`; the host
|
|
daemon needs the same pin in `/etc/hosts`.
|
|
- The job image exists: `dev.jinemi.com/jinemi/thermograph/ci-runner:v2`.
|
|
Built from `../ci-runner/Dockerfile`; see that file for when to rebuild.
|
|
|
|
## Where it is deployed, and why not from the checkout
|
|
|
|
**`/opt/forgejo-runner/` on vps2**, holding a copy of `docker-compose.yml` and
|
|
`config.yaml` from this directory, plus a `data/` the runner owns.
|
|
|
|
Deliberately *not* run in place from `/opt/thermograph/…/runner-vps2/`. That
|
|
checkout is `git reset --hard`'d on every prod deploy; untracked files do
|
|
survive that (it is the same property `deploy/.image-tags.env` relies on), but a
|
|
single `git clean -fdx` during troubleshooting would delete `data/.runner` and
|
|
de-register the runner. Putting the thing that repairs the estate inside the
|
|
tree the estate rewrites is the same mistake as scheduling it on the Swarm it
|
|
deploys.
|
|
|
|
So this directory is the source of truth; vps2 gets a copy:
|
|
|
|
The two files do **not** go to the same place, and getting it wrong fails in a
|
|
confusing way. `docker-compose.yml` mounts `./data:/data` and starts the daemon
|
|
with `--config /data/config.yaml`, so `/data` *is* `data/` — a `config.yaml`
|
|
sitting beside the compose file is invisible to the container. The daemon then
|
|
either refuses to start or silently runs on defaults (no `--add-host`, so every
|
|
registry push fails as if the credential were wrong).
|
|
|
|
```bash
|
|
install -d -m 0755 /opt/forgejo-runner/data
|
|
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \
|
|
/opt/forgejo-runner/
|
|
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
|
|
/opt/forgejo-runner/data/
|
|
```
|
|
|
|
`data/` is also where `register` writes `.runner`, so the runner's whole state —
|
|
config plus registration — lives in the one directory the compose file mounts.
|
|
|
|
Re-copy after changing either file here, then `docker compose up -d`. There is
|
|
no automation for that step and there should not be: a workflow that redeploys
|
|
the runner runs *on* the runner.
|
|
|
|
## One-time registration
|
|
|
|
The registration token is short-lived and single-purpose. Fetch it from the
|
|
Forgejo UI (**Site Administration → Actions → Runners → Create new Runner**) or
|
|
the API, and keep it out of shell history — it is a credential, however
|
|
briefly.
|
|
|
|
```bash
|
|
cd /opt/forgejo-runner
|
|
mkdir -p data
|
|
cp config.yaml data/config.yaml
|
|
|
|
read -rs REG_TOKEN && export REG_TOKEN # paste; not echoed, not in history
|
|
|
|
docker run --rm -it \
|
|
-v "$PWD/data:/data" -w /data --user root \
|
|
code.forgejo.org/forgejo/runner:6.3.1 \
|
|
forgejo-runner register --no-interactive \
|
|
--instance https://dev.jinemi.com \
|
|
--token "$REG_TOKEN" \
|
|
--name thermograph-vps2 \
|
|
--labels 'docker:docker://dev.jinemi.com/jinemi/thermograph/ci-runner:v2'
|
|
|
|
unset REG_TOKEN
|
|
```
|
|
|
|
The runner registered before the domain migration holds
|
|
`https://git.thermograph.org` as its instance URL and keeps working while Caddy
|
|
serves that name (it does — see `../caddy-git.conf`). Re-registering with the
|
|
command above is how it moves to the canonical name; it is not required, and it
|
|
costs a fresh registration token and a runner restart.
|
|
|
|
That writes `data/.runner`, which holds the **long-lived** runner token. It is
|
|
`0600` and must stay out of git — `data/` is gitignored for exactly this reason.
|
|
Re-running `register` with a fresh registration token replaces it; the old
|
|
registration then shows offline in the UI and should be deleted there.
|
|
|
|
## Run it
|
|
|
|
```bash
|
|
docker compose up -d
|
|
docker compose logs -f --tail=50 runner # expect "runner: ... successfully activated"
|
|
```
|
|
|
|
Confirm Forgejo agrees, rather than trusting the log:
|
|
|
|
```bash
|
|
# on vps1, where Forgejo's database lives
|
|
fdb=$(docker ps -qf name=forgejo_db | head -1)
|
|
docker exec "$fdb" sh -c 'psql -U $POSTGRES_USER -d $POSTGRES_DB -c \
|
|
"select name, to_timestamp(last_online) as last_online, agent_labels from action_runner order by id;"'
|
|
```
|
|
|
|
Two rows, both recent. `last_online` is the only honest liveness signal — a
|
|
runner process that is up but cannot reach Forgejo looks healthy locally and is
|
|
useless.
|
|
|
|
## Upgrading
|
|
|
|
Pin changes go in `docker-compose.yml` (`image:`) and are applied with
|
|
`docker compose up -d`. The runner token survives an image change; it lives in
|
|
`data/.runner`, not in the image.
|
|
|
|
Changing the **job** image (the `docker://…ci-runner:vN` label) is a
|
|
re-registration or a hand-edit of `data/.runner`'s `labels` array — the same
|
|
caveat `../ci-runner/Dockerfile` records for the desktop runner. Keep the two
|
|
runners on the same job image or a workflow's behaviour starts depending on
|
|
which runner happened to pick it up.
|
|
|
|
## Troubleshooting
|
|
|
|
**Jobs queue forever with both runners online.** The workflow asked for a label
|
|
nobody registered. `runs-on: docker` is the only label this estate uses.
|
|
|
|
**`docker pull` fails inside a job, works on the host.** The `--add-host` line
|
|
in `config.yaml` is missing or wrong. vps1's Caddy serves `/v2/*` to the mesh
|
|
only, and the job container has no other way to learn the mesh address.
|
|
|
|
**Jobs fail with permission denied on the socket.** `user: root` was dropped
|
|
from the compose file.
|
|
|
|
**The runner is up but `last_online` is stale.** It cannot reach
|
|
the instance URL it registered with (`https://dev.jinemi.com`, or
|
|
`https://git.thermograph.org` for a runner registered before the migration) —
|
|
check egress from vps2 and that Forgejo on vps1 is serving that name.
|