ci: add an always-on Actions runner on vps2

The estate had exactly one registered runner, on the desktop. It went offline
at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required
check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both
application databases and for Forgejo -- did not fire. Forgejo queued that
scheduled run rather than dropping it, so it completed on reconnect and nothing
was lost. A longer outage would have meant real gaps.

Three files claimed an "always-on Swarm-hosted runner" existed and that the
estate therefore no longer depended on the desktop. It did not exist: an early
revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was
removed. That claim is why a single point of failure sat unnoticed. Corrected
in docker-stack.yml, forgejo/README.md and register-lan-runner.sh.

The new runner is a plain restart:always container, not a Swarm service: a
Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would
be gone exactly when the cluster is what is broken. It runs from
/opt/forgejo-runner rather than in place, because the checkout is reset on
every prod deploy and one `git clean -fdx` there would destroy the
registration.

vps2 runs prod, so the socket mount is bounded rather than assumed benign:
capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no
job can bind-mount /etc/thermograph.env, and no thermograph network joined.
This defends against accident, not against a hostile workflow author -- stated
plainly in the compose header rather than implied.

The desktop runner stays registered as extra capacity. Nothing may assume it
is up.

.forgejo/workflows/ops-cron.yml

@@ -8,8 +8,15 @@ name: Ops cron (backup + IndexNow)
 #
 # The jobs SSH into a host and run inside the already-running stack (docker
 # exec), the same way deploy.sh runs its own post-deploy IndexNow ping -- no new
-# network exposure, no separate dependency install. Runs on the `docker` label
+# network exposure, no separate dependency install. Runs on the `docker` label,
-# (the always-on Swarm-hosted runner deploy/forgejo/ stood up).
+# served by the vps2 runner (infra/deploy/forgejo/runner-vps2/) and, when it is
+# up, the desktop.
+#
+# THIS WORKFLOW IS THE ESTATE'S ONLY BACKUP, so its dependency on a runner is
+# load-bearing. On 2026-07-31 the desktop was the only registered runner and
+# went offline for 21 hours; this schedule did not fire at 03:00Z. Forgejo held
+# the run in `waiting` and it completed on reconnect, so nothing was lost that
+# time. The vps2 runner exists so the next outage is not a coin flip.
 #
 # WHICH BOX EACH JOB TALKS TO, and why the secret names say so:
 #

infra/deploy/forgejo/README.md

@@ -7,12 +7,20 @@ deploy`s that happen to run on the manager node, vps2 — see
 (`75.119.132.91`) via the `role=forge` label from
 `deploy/swarm/label-forge-node.sh`.
 
-The Actions **runner** is deliberately *not* part of this stack — see
+The Actions **runner** is deliberately *not* part of this stack: a
-`DEPLOY-DEV.md` and the note in `register-lan-runner.sh`'s own header for
+Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would
-where it runs today. That's the canonical placement per
+be unavailable exactly when the cluster is what needs repairing. An earlier
-`thermograph-docs/runbooks/implementation-handoff.md` Track B step 5; an
+revision did run it as a Swarm-scheduled Docker-in-Docker sidecar pinned to the
-earlier revision of this stack ran the runner as a Swarm-scheduled
+Forgejo node; that is gone.
-Docker-in-Docker sidecar pinned to the Forgejo node, which is gone now.
+
+Runners live in two places, and only one of them counts:
+
+- **`runner-vps2/`** — the always-on runner, a plain `restart: always` container
+  on vps2. This is the one CI depends on.
+- **`register-lan-runner.sh`** — the desktop's systemd runner. Extra capacity.
+  **Nothing may assume it is up.** It was the only registered runner in the
+  estate until 2026-08-01, and its 21-hour outage on 2026-07-31 froze every
+  merge and deploy and stopped the nightly backup from firing.
 
 ## Prerequisites
 

infra/deploy/forgejo/docker-stack.yml

@@ -3,11 +3,16 @@
 # from the Terraform-managed docker-compose.yml that runs the Thermograph app
 # itself: this stack's only job is Forgejo and its container registry.
 #
-# The Actions RUNNER is deliberately NOT a service in this stack. Per
+# The Actions RUNNER is deliberately NOT a service in this stack, and that is a
-# thermograph-docs/runbooks/implementation-handoff.md (Track B step 5), the canonical
+# reasoned choice rather than an omission: a Swarm-scheduled runner cannot
-# design registers the runner on the desktop node as a plain systemd service
+# redeploy the Swarm that schedules it, so CI would be gone exactly when the
-# (deploy/forgejo/register-lan-runner.sh) — the same place the pre-Forgejo
+# cluster is the thing that is broken.
-# GitHub self-hosted runner already lived, not a Swarm-scheduled container.
+#
+# The runner that CI depends on runs on vps2 as a plain restart:always container
+# — deploy/forgejo/runner-vps2/. The desktop's systemd runner
+# (deploy/forgejo/register-lan-runner.sh) stays registered as extra capacity.
+# Nothing may assume the desktop is up: it was the ONLY runner until 2026-08-01,
+# and its 21-hour outage on 2026-07-31 froze every merge, deploy and backup.
 #
 # No Traefik here. Forgejo is pinned to vps1 (node.labels.role == forge)
 # because that's what was chosen, and vps1 is ALSO the emigriffith.dev portfolio

infra/deploy/forgejo/register-lan-runner.sh

@@ -18,11 +18,16 @@
 # breaks by keeping it registered — no workflow requests it — but do not build
 # anything new on it.
 #
-# TODO(cutover): whether this machine keeps serving the `docker` label at all is
+# This machine keeps serving the `docker` label as EXTRA capacity, and that is
-# an open call. It is no longer the only runner (the Swarm-hosted one is
+# now a settled call rather than an open one. The claim it used to make here —
-# always-on), and the box's new job is AI model hosting plus flex Swarm-worker
+# that a Swarm-hosted runner was always-on, so the estate no longer depended on
-# capacity. Keeping it is fine — it is real CI capacity — but the estate no
+# this box — was false: no such runner existed. This was the only registered
-# longer depends on it, so decide deliberately rather than by inertia.
+# runner in the estate until 2026-08-01, and when it went offline on 2026-07-31
+# every merge, deploy and backup stopped for 21 hours.
+#
+# The runner the estate depends on is deploy/forgejo/runner-vps2/. Keeping this
+# one is worthwhile (a second runner means a queue drains instead of stalling),
+# but nothing may be built on the assumption that this box is up.
 #
 #   bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
 #

infra/deploy/forgejo/runner-vps2/.gitignore

@@ -0,0 +1,3 @@
+# The runner registration (data/.runner) holds a long-lived runner token, and
+# the daemon writes caches and job workspaces beside it. None of it belongs in git.
+data/

infra/deploy/forgejo/runner-vps2/README.md

@@ -0,0 +1,140 @@
+# The always-on Actions runner (vps2)
+
+The runner CI actually depends on. The desktop runner
+(`../register-lan-runner.sh`) stays registered as extra capacity, but nothing in
+the estate may assume it is up.
+
+Labels: `docker` only. `thermograph-lan` is deliberately **not** registered here
+— it was the host-native LAN-deploy label, dev moved to vps1 and is deployed
+over SSH like beta and prod, and no workflow requests it any more.
+
+## Why this exists
+
+On 2026-07-31 16:31Z the desktop — then the estate's **only** registered runner
+— went offline. For the next 21 hours nothing merged (every protected branch
+requires a check that only a runner can produce), nothing deployed, and the
+03:00Z `ops-cron` did not run. Forgejo held that scheduled run in `waiting`
+rather than dropping it, so all three jobs completed on reconnect and no backup
+was actually lost. A longer outage would not have been so kind.
+
+`docker-stack.yml` and this directory's parent README both described an
+"always-on Swarm-hosted runner" during that entire window. There wasn't one; an
+early revision had it as a Docker-in-Docker sidecar and it was removed. That is
+why the single point of failure went unnoticed for weeks, and it is the reason
+the root `CLAUDE.md` treats a stale doc as a correctness bug rather than a
+documentation one.
+
+## Prerequisites
+
+- vps2's docker daemon is logged in to `git.thermograph.org` (it is —
+  `/root/.docker/config.json`), so the daemon can pull the job image.
+- `git.thermograph.org` resolves to the **mesh** address from vps2 for registry
+  traffic. Job containers get this via `--add-host` in `config.yaml`; the host
+  daemon already has it.
+- The job image exists: `git.thermograph.org/emi/thermograph/ci-runner:v2`.
+  Built from `../ci-runner/Dockerfile`; see that file for when to rebuild.
+
+## Where it is deployed, and why not from the checkout
+
+**`/opt/forgejo-runner/` on vps2**, holding a copy of `docker-compose.yml` and
+`config.yaml` from this directory, plus a `data/` the runner owns.
+
+Deliberately *not* run in place from `/opt/thermograph/…/runner-vps2/`. That
+checkout is `git reset --hard`'d on every prod deploy; untracked files do
+survive that (it is the same property `deploy/.image-tags.env` relies on), but a
+single `git clean -fdx` during troubleshooting would delete `data/.runner` and
+de-register the runner. Putting the thing that repairs the estate inside the
+tree the estate rewrites is the same mistake as scheduling it on the Swarm it
+deploys.
+
+So this directory is the source of truth; vps2 gets a copy:
+
+```bash
+install -d -m 0755 /opt/forgejo-runner
+cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \
+   /opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
+   /opt/forgejo-runner/
+```
+
+Re-copy after changing either file here, then `docker compose up -d`. There is
+no automation for that step and there should not be: a workflow that redeploys
+the runner runs *on* the runner.
+
+## One-time registration
+
+The registration token is short-lived and single-purpose. Fetch it from the
+Forgejo UI (**Site Administration → Actions → Runners → Create new Runner**) or
+the API, and keep it out of shell history — it is a credential, however
+briefly.
+
+```bash
+cd /opt/forgejo-runner
+mkdir -p data
+cp config.yaml data/config.yaml
+
+read -rs REG_TOKEN && export REG_TOKEN     # paste; not echoed, not in history
+
+docker run --rm -it \
+  -v "$PWD/data:/data" -w /data --user root \
+  code.forgejo.org/forgejo/runner:6.3.1 \
+  forgejo-runner register --no-interactive \
+    --instance https://git.thermograph.org \
+    --token "$REG_TOKEN" \
+    --name thermograph-vps2 \
+    --labels 'docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v2'
+
+unset REG_TOKEN
+```
+
+That writes `data/.runner`, which holds the **long-lived** runner token. It is
+`0600` and must stay out of git — `data/` is gitignored for exactly this reason.
+Re-running `register` with a fresh registration token replaces it; the old
+registration then shows offline in the UI and should be deleted there.
+
+## Run it
+
+```bash
+docker compose up -d
+docker compose logs -f --tail=50 runner     # expect "runner: ... successfully activated"
+```
+
+Confirm Forgejo agrees, rather than trusting the log:
+
+```bash
+# on vps1, where Forgejo's database lives
+fdb=$(docker ps -qf name=forgejo_db | head -1)
+docker exec "$fdb" sh -c 'psql -U $POSTGRES_USER -d $POSTGRES_DB -c \
+  "select name, to_timestamp(last_online) as last_online, agent_labels from action_runner order by id;"'
+```
+
+Two rows, both recent. `last_online` is the only honest liveness signal — a
+runner process that is up but cannot reach Forgejo looks healthy locally and is
+useless.
+
+## Upgrading
+
+Pin changes go in `docker-compose.yml` (`image:`) and are applied with
+`docker compose up -d`. The runner token survives an image change; it lives in
+`data/.runner`, not in the image.
+
+Changing the **job** image (the `docker://…ci-runner:vN` label) is a
+re-registration or a hand-edit of `data/.runner`'s `labels` array — the same
+caveat `../ci-runner/Dockerfile` records for the desktop runner. Keep the two
+runners on the same job image or a workflow's behaviour starts depending on
+which runner happened to pick it up.
+
+## Troubleshooting
+
+**Jobs queue forever with both runners online.** The workflow asked for a label
+nobody registered. `runs-on: docker` is the only label this estate uses.
+
+**`docker pull` fails inside a job, works on the host.** The `--add-host` line
+in `config.yaml` is missing or wrong. vps1's Caddy serves `/v2/*` to the mesh
+only, and the job container has no other way to learn the mesh address.
+
+**Jobs fail with permission denied on the socket.** `user: root` was dropped
+from the compose file.
+
+**The runner is up but `last_online` is stale.** It cannot reach
+`https://git.thermograph.org` — check egress from vps2 and that Forgejo on vps1
+is serving.

infra/deploy/forgejo/runner-vps2/config.yaml

@@ -0,0 +1,53 @@
+# Forgejo Actions runner config for the vps2 runner. Mounted at /data/config.yaml
+# by runner-vps2/docker-compose.yml.
+#
+# Values that differ from `forgejo-runner generate-config` defaults are the only
+# ones written out below; everything else is left at its default deliberately, so
+# a future runner upgrade inherits new defaults instead of a frozen copy of the
+# old ones.
+
+log:
+  level: info
+  job_level: info
+
+runner:
+  file: .runner
+
+  # ONE job at a time. The desktop runner uses capacity 8, which is right for a
+  # box whose whole job is CI. This runner shares a host with prod, so the
+  # binding constraint is not throughput, it is that a CI burst must never be
+  # able to contend with thermograph_web for CPU. One job at a time also makes
+  # the resource ceiling below exact rather than a per-job estimate multiplied
+  # by an unknown.
+  capacity: 1
+
+  timeout: 3h
+  shutdown_timeout: 3h
+
+container:
+  # 1) --add-host: git.thermograph.org resolves publicly to vps1, but vps1's
+  #    Caddy rejects the /v2/* registry API from off-mesh. Job containers that
+  #    docker-pull or docker-push therefore need the MESH address, and DNS will
+  #    not give it to them. Same line the desktop runner carries; the value is
+  #    the same from vps2 because both boxes are on 10.10.0.0/24.
+  #
+  # 2) --cpus/--memory: vps2 runs prod. Without a ceiling a runaway job (a
+  #    `docker build` that fans out, a test that leaks) competes with the
+  #    production app for the same 6 cores. These bound the JOB container, which
+  #    is a sibling of the runner rather than its child, so the runner's own
+  #    compose limits do not cover it -- this is the only lever that does.
+  options: --add-host=git.thermograph.org:10.10.0.2 --cpus=2 --memory=4g
+
+  # Jobs may mount NOTHING from the host. The workflows here do not need to:
+  # they check out into the job container's own workspace and reach the estate
+  # over SSH. An empty list is the difference between "a job can read
+  # /etc/thermograph.env" and "a job cannot", on the box where that file is
+  # prod's.
+  valid_volumes: []
+
+  privileged: false
+
+  # Mounts the host docker socket into the job container. build-push.yml runs
+  # `docker build`, so it needs a daemon; this points it at vps2's. See the
+  # README for what that does and does not mean for trust.
+  docker_host: "automount"

infra/deploy/forgejo/runner-vps2/docker-compose.yml

@@ -0,0 +1,83 @@
+# The always-on Forgejo Actions runner, on vps2.
+#
+#   cd /opt/forgejo-runner        # a COPY of this file, not the checkout
+#   docker compose up -d
+#
+# It runs from /opt/forgejo-runner rather than in place, because the checkout is
+# git reset --hard'd on every prod deploy and one `git clean -fdx` there would
+# delete data/.runner. See README.md.
+#
+# Registration is a one-off and is NOT in this file, because it takes a
+# short-lived token a human fetches. See README.md.
+#
+# ============================================================================
+# WHY A SECOND RUNNER EXISTS AT ALL
+# ============================================================================
+# Until 2026-08-01 the estate had exactly one registered runner, on the desktop.
+# It went offline at 2026-07-31 16:31Z and everything stopped for 21 hours: no
+# merge could satisfy a required check, no deploy could run, and the 03:00Z
+# ops-cron -- THE backup for both application databases and for Forgejo -- did
+# not fire. Forgejo queued that run rather than dropping it, so it completed on
+# reconnect, but a longer outage would have meant real backup gaps.
+#
+# So this runner is not extra capacity. It is the one that has to be up.
+#
+# ============================================================================
+# WHY A PLAIN CONTAINER AND NOT A SWARM SERVICE
+# ============================================================================
+# Same argument as infra/openbao/config/openbao.hcl makes for OpenBao: the thing
+# that repairs the estate must not depend on the estate. A Swarm-scheduled
+# runner cannot redeploy the Swarm it is scheduled by, and if the cluster is the
+# thing that is broken, CI is gone exactly when it is needed. `restart: always`
+# on the local daemon has one dependency: the local daemon.
+#
+# An earlier revision of deploy/forgejo/docker-stack.yml did run a runner as a
+# Swarm-scheduled Docker-in-Docker sidecar. It was removed, and the docs kept
+# claiming an "always-on Swarm-hosted runner" for weeks afterwards -- which is
+# how the single point of failure went unnoticed.
+#
+# ============================================================================
+# WHAT MOUNTING THE DOCKER SOCKET ON THE PROD HOST MEANS
+# ============================================================================
+# Be plain about it: /var/run/docker.sock is root-equivalent on vps2, and vps2
+# runs prod. A workflow that can start a container can mount / and read
+# /etc/thermograph.env. This runner does not create that exposure from nothing
+# -- Forgejo's own database already stores VPS2_SSH_KEY, which is root on this
+# box -- but it does move the exposure from "a secret at rest" to "a secret a
+# job can reach", and that is a real difference.
+#
+# What bounds it, in order of how much each is worth:
+#   1. Only this repo's workflows run here, and only the owner can push to it.
+#   2. config.yaml sets valid_volumes: [] -- jobs cannot bind-mount host paths.
+#   3. capacity: 1 and --cpus/--memory keep a job from contending with prod.
+#   4. This compose project joins no thermograph network, so a job container
+#      reaches prod's services no more easily than any other host process.
+#
+# It is defence against accident, not against a hostile workflow author. On a
+# two-person estate where both people can already SSH to this box as root, that
+# is the honest boundary.
+
+name: forgejo-runner
+
+services:
+  runner:
+    image: code.forgejo.org/forgejo/runner:6.3.1
+    restart: always
+    # The socket is root:docker on the host; the image's default user is not in
+    # that group. Running as root is what the socket mount requires.
+    user: root
+    working_dir: /data
+    command: forgejo-runner daemon --config /data/config.yaml
+    volumes:
+      - ./data:/data
+      - /var/run/docker.sock:/var/run/docker.sock
+    # Bounds the DAEMON. Job containers are siblings started through the socket,
+    # not children, so they are unaffected by these -- container.options in
+    # config.yaml is what bounds those.
+    cpus: 1.0
+    mem_limit: 1g
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"