diff --git a/.forgejo/workflows/ops-cron.yml b/.forgejo/workflows/ops-cron.yml index 112d0be..3412794 100644 --- a/.forgejo/workflows/ops-cron.yml +++ b/.forgejo/workflows/ops-cron.yml @@ -8,8 +8,15 @@ name: Ops cron (backup + IndexNow) # # The jobs SSH into a host and run inside the already-running stack (docker # exec), the same way deploy.sh runs its own post-deploy IndexNow ping -- no new -# network exposure, no separate dependency install. Runs on the `docker` label -# (the always-on Swarm-hosted runner deploy/forgejo/ stood up). +# network exposure, no separate dependency install. Runs on the `docker` label, +# served by the vps2 runner (infra/deploy/forgejo/runner-vps2/) and, when it is +# up, the desktop. +# +# THIS WORKFLOW IS THE ESTATE'S ONLY BACKUP, so its dependency on a runner is +# load-bearing. On 2026-07-31 the desktop was the only registered runner and +# went offline for 21 hours; this schedule did not fire at 03:00Z. Forgejo held +# the run in `waiting` and it completed on reconnect, so nothing was lost that +# time. The vps2 runner exists so the next outage is not a coin flip. # # WHICH BOX EACH JOB TALKS TO, and why the secret names say so: # diff --git a/infra/deploy/forgejo/README.md b/infra/deploy/forgejo/README.md index f15711f..9e295c1 100644 --- a/infra/deploy/forgejo/README.md +++ b/infra/deploy/forgejo/README.md @@ -7,12 +7,20 @@ deploy`s that happen to run on the manager node, vps2 — see (`75.119.132.91`) via the `role=forge` label from `deploy/swarm/label-forge-node.sh`. -The Actions **runner** is deliberately *not* part of this stack — see -`DEPLOY-DEV.md` and the note in `register-lan-runner.sh`'s own header for -where it runs today. That's the canonical placement per -`thermograph-docs/runbooks/implementation-handoff.md` Track B step 5; an -earlier revision of this stack ran the runner as a Swarm-scheduled -Docker-in-Docker sidecar pinned to the Forgejo node, which is gone now. +The Actions **runner** is deliberately *not* part of this stack: a +Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would +be unavailable exactly when the cluster is what needs repairing. An earlier +revision did run it as a Swarm-scheduled Docker-in-Docker sidecar pinned to the +Forgejo node; that is gone. + +Runners live in two places, and only one of them counts: + +- **`runner-vps2/`** — the always-on runner, a plain `restart: always` container + on vps2. This is the one CI depends on. +- **`register-lan-runner.sh`** — the desktop's systemd runner. Extra capacity. + **Nothing may assume it is up.** It was the only registered runner in the + estate until 2026-08-01, and its 21-hour outage on 2026-07-31 froze every + merge and deploy and stopped the nightly backup from firing. ## Prerequisites diff --git a/infra/deploy/forgejo/docker-stack.yml b/infra/deploy/forgejo/docker-stack.yml index b939025..7d1603f 100644 --- a/infra/deploy/forgejo/docker-stack.yml +++ b/infra/deploy/forgejo/docker-stack.yml @@ -3,11 +3,16 @@ # from the Terraform-managed docker-compose.yml that runs the Thermograph app # itself: this stack's only job is Forgejo and its container registry. # -# The Actions RUNNER is deliberately NOT a service in this stack. Per -# thermograph-docs/runbooks/implementation-handoff.md (Track B step 5), the canonical -# design registers the runner on the desktop node as a plain systemd service -# (deploy/forgejo/register-lan-runner.sh) — the same place the pre-Forgejo -# GitHub self-hosted runner already lived, not a Swarm-scheduled container. +# The Actions RUNNER is deliberately NOT a service in this stack, and that is a +# reasoned choice rather than an omission: a Swarm-scheduled runner cannot +# redeploy the Swarm that schedules it, so CI would be gone exactly when the +# cluster is the thing that is broken. +# +# The runner that CI depends on runs on vps2 as a plain restart:always container +# — deploy/forgejo/runner-vps2/. The desktop's systemd runner +# (deploy/forgejo/register-lan-runner.sh) stays registered as extra capacity. +# Nothing may assume the desktop is up: it was the ONLY runner until 2026-08-01, +# and its 21-hour outage on 2026-07-31 froze every merge, deploy and backup. # # No Traefik here. Forgejo is pinned to vps1 (node.labels.role == forge) # because that's what was chosen, and vps1 is ALSO the emigriffith.dev portfolio diff --git a/infra/deploy/forgejo/register-lan-runner.sh b/infra/deploy/forgejo/register-lan-runner.sh index c204798..1785c52 100755 --- a/infra/deploy/forgejo/register-lan-runner.sh +++ b/infra/deploy/forgejo/register-lan-runner.sh @@ -18,11 +18,16 @@ # breaks by keeping it registered — no workflow requests it — but do not build # anything new on it. # -# TODO(cutover): whether this machine keeps serving the `docker` label at all is -# an open call. It is no longer the only runner (the Swarm-hosted one is -# always-on), and the box's new job is AI model hosting plus flex Swarm-worker -# capacity. Keeping it is fine — it is real CI capacity — but the estate no -# longer depends on it, so decide deliberately rather than by inertia. +# This machine keeps serving the `docker` label as EXTRA capacity, and that is +# now a settled call rather than an open one. The claim it used to make here — +# that a Swarm-hosted runner was always-on, so the estate no longer depended on +# this box — was false: no such runner existed. This was the only registered +# runner in the estate until 2026-08-01, and when it went offline on 2026-07-31 +# every merge, deploy and backup stopped for 21 hours. +# +# The runner the estate depends on is deploy/forgejo/runner-vps2/. Keeping this +# one is worthwhile (a second runner means a queue drains instead of stalling), +# but nothing may be built on the assumption that this box is up. # # bash deploy/forgejo/register-lan-runner.sh # diff --git a/infra/deploy/forgejo/runner-vps2/.gitignore b/infra/deploy/forgejo/runner-vps2/.gitignore new file mode 100644 index 0000000..ebfdccb --- /dev/null +++ b/infra/deploy/forgejo/runner-vps2/.gitignore @@ -0,0 +1,3 @@ +# The runner registration (data/.runner) holds a long-lived runner token, and +# the daemon writes caches and job workspaces beside it. None of it belongs in git. +data/ diff --git a/infra/deploy/forgejo/runner-vps2/README.md b/infra/deploy/forgejo/runner-vps2/README.md new file mode 100644 index 0000000..a5f746a --- /dev/null +++ b/infra/deploy/forgejo/runner-vps2/README.md @@ -0,0 +1,140 @@ +# The always-on Actions runner (vps2) + +The runner CI actually depends on. The desktop runner +(`../register-lan-runner.sh`) stays registered as extra capacity, but nothing in +the estate may assume it is up. + +Labels: `docker` only. `thermograph-lan` is deliberately **not** registered here +— it was the host-native LAN-deploy label, dev moved to vps1 and is deployed +over SSH like beta and prod, and no workflow requests it any more. + +## Why this exists + +On 2026-07-31 16:31Z the desktop — then the estate's **only** registered runner +— went offline. For the next 21 hours nothing merged (every protected branch +requires a check that only a runner can produce), nothing deployed, and the +03:00Z `ops-cron` did not run. Forgejo held that scheduled run in `waiting` +rather than dropping it, so all three jobs completed on reconnect and no backup +was actually lost. A longer outage would not have been so kind. + +`docker-stack.yml` and this directory's parent README both described an +"always-on Swarm-hosted runner" during that entire window. There wasn't one; an +early revision had it as a Docker-in-Docker sidecar and it was removed. That is +why the single point of failure went unnoticed for weeks, and it is the reason +the root `CLAUDE.md` treats a stale doc as a correctness bug rather than a +documentation one. + +## Prerequisites + +- vps2's docker daemon is logged in to `git.thermograph.org` (it is — + `/root/.docker/config.json`), so the daemon can pull the job image. +- `git.thermograph.org` resolves to the **mesh** address from vps2 for registry + traffic. Job containers get this via `--add-host` in `config.yaml`; the host + daemon already has it. +- The job image exists: `git.thermograph.org/emi/thermograph/ci-runner:v2`. + Built from `../ci-runner/Dockerfile`; see that file for when to rebuild. + +## Where it is deployed, and why not from the checkout + +**`/opt/forgejo-runner/` on vps2**, holding a copy of `docker-compose.yml` and +`config.yaml` from this directory, plus a `data/` the runner owns. + +Deliberately *not* run in place from `/opt/thermograph/…/runner-vps2/`. That +checkout is `git reset --hard`'d on every prod deploy; untracked files do +survive that (it is the same property `deploy/.image-tags.env` relies on), but a +single `git clean -fdx` during troubleshooting would delete `data/.runner` and +de-register the runner. Putting the thing that repairs the estate inside the +tree the estate rewrites is the same mistake as scheduling it on the Swarm it +deploys. + +So this directory is the source of truth; vps2 gets a copy: + +```bash +install -d -m 0755 /opt/forgejo-runner +cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \ + /opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \ + /opt/forgejo-runner/ +``` + +Re-copy after changing either file here, then `docker compose up -d`. There is +no automation for that step and there should not be: a workflow that redeploys +the runner runs *on* the runner. + +## One-time registration + +The registration token is short-lived and single-purpose. Fetch it from the +Forgejo UI (**Site Administration → Actions → Runners → Create new Runner**) or +the API, and keep it out of shell history — it is a credential, however +briefly. + +```bash +cd /opt/forgejo-runner +mkdir -p data +cp config.yaml data/config.yaml + +read -rs REG_TOKEN && export REG_TOKEN # paste; not echoed, not in history + +docker run --rm -it \ + -v "$PWD/data:/data" -w /data --user root \ + code.forgejo.org/forgejo/runner:6.3.1 \ + forgejo-runner register --no-interactive \ + --instance https://git.thermograph.org \ + --token "$REG_TOKEN" \ + --name thermograph-vps2 \ + --labels 'docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v2' + +unset REG_TOKEN +``` + +That writes `data/.runner`, which holds the **long-lived** runner token. It is +`0600` and must stay out of git — `data/` is gitignored for exactly this reason. +Re-running `register` with a fresh registration token replaces it; the old +registration then shows offline in the UI and should be deleted there. + +## Run it + +```bash +docker compose up -d +docker compose logs -f --tail=50 runner # expect "runner: ... successfully activated" +``` + +Confirm Forgejo agrees, rather than trusting the log: + +```bash +# on vps1, where Forgejo's database lives +fdb=$(docker ps -qf name=forgejo_db | head -1) +docker exec "$fdb" sh -c 'psql -U $POSTGRES_USER -d $POSTGRES_DB -c \ + "select name, to_timestamp(last_online) as last_online, agent_labels from action_runner order by id;"' +``` + +Two rows, both recent. `last_online` is the only honest liveness signal — a +runner process that is up but cannot reach Forgejo looks healthy locally and is +useless. + +## Upgrading + +Pin changes go in `docker-compose.yml` (`image:`) and are applied with +`docker compose up -d`. The runner token survives an image change; it lives in +`data/.runner`, not in the image. + +Changing the **job** image (the `docker://…ci-runner:vN` label) is a +re-registration or a hand-edit of `data/.runner`'s `labels` array — the same +caveat `../ci-runner/Dockerfile` records for the desktop runner. Keep the two +runners on the same job image or a workflow's behaviour starts depending on +which runner happened to pick it up. + +## Troubleshooting + +**Jobs queue forever with both runners online.** The workflow asked for a label +nobody registered. `runs-on: docker` is the only label this estate uses. + +**`docker pull` fails inside a job, works on the host.** The `--add-host` line +in `config.yaml` is missing or wrong. vps1's Caddy serves `/v2/*` to the mesh +only, and the job container has no other way to learn the mesh address. + +**Jobs fail with permission denied on the socket.** `user: root` was dropped +from the compose file. + +**The runner is up but `last_online` is stale.** It cannot reach +`https://git.thermograph.org` — check egress from vps2 and that Forgejo on vps1 +is serving. diff --git a/infra/deploy/forgejo/runner-vps2/config.yaml b/infra/deploy/forgejo/runner-vps2/config.yaml new file mode 100644 index 0000000..9862e4c --- /dev/null +++ b/infra/deploy/forgejo/runner-vps2/config.yaml @@ -0,0 +1,53 @@ +# Forgejo Actions runner config for the vps2 runner. Mounted at /data/config.yaml +# by runner-vps2/docker-compose.yml. +# +# Values that differ from `forgejo-runner generate-config` defaults are the only +# ones written out below; everything else is left at its default deliberately, so +# a future runner upgrade inherits new defaults instead of a frozen copy of the +# old ones. + +log: + level: info + job_level: info + +runner: + file: .runner + + # ONE job at a time. The desktop runner uses capacity 8, which is right for a + # box whose whole job is CI. This runner shares a host with prod, so the + # binding constraint is not throughput, it is that a CI burst must never be + # able to contend with thermograph_web for CPU. One job at a time also makes + # the resource ceiling below exact rather than a per-job estimate multiplied + # by an unknown. + capacity: 1 + + timeout: 3h + shutdown_timeout: 3h + +container: + # 1) --add-host: git.thermograph.org resolves publicly to vps1, but vps1's + # Caddy rejects the /v2/* registry API from off-mesh. Job containers that + # docker-pull or docker-push therefore need the MESH address, and DNS will + # not give it to them. Same line the desktop runner carries; the value is + # the same from vps2 because both boxes are on 10.10.0.0/24. + # + # 2) --cpus/--memory: vps2 runs prod. Without a ceiling a runaway job (a + # `docker build` that fans out, a test that leaks) competes with the + # production app for the same 6 cores. These bound the JOB container, which + # is a sibling of the runner rather than its child, so the runner's own + # compose limits do not cover it -- this is the only lever that does. + options: --add-host=git.thermograph.org:10.10.0.2 --cpus=2 --memory=4g + + # Jobs may mount NOTHING from the host. The workflows here do not need to: + # they check out into the job container's own workspace and reach the estate + # over SSH. An empty list is the difference between "a job can read + # /etc/thermograph.env" and "a job cannot", on the box where that file is + # prod's. + valid_volumes: [] + + privileged: false + + # Mounts the host docker socket into the job container. build-push.yml runs + # `docker build`, so it needs a daemon; this points it at vps2's. See the + # README for what that does and does not mean for trust. + docker_host: "automount" diff --git a/infra/deploy/forgejo/runner-vps2/docker-compose.yml b/infra/deploy/forgejo/runner-vps2/docker-compose.yml new file mode 100644 index 0000000..a85934d --- /dev/null +++ b/infra/deploy/forgejo/runner-vps2/docker-compose.yml @@ -0,0 +1,83 @@ +# The always-on Forgejo Actions runner, on vps2. +# +# cd /opt/forgejo-runner # a COPY of this file, not the checkout +# docker compose up -d +# +# It runs from /opt/forgejo-runner rather than in place, because the checkout is +# git reset --hard'd on every prod deploy and one `git clean -fdx` there would +# delete data/.runner. See README.md. +# +# Registration is a one-off and is NOT in this file, because it takes a +# short-lived token a human fetches. See README.md. +# +# ============================================================================ +# WHY A SECOND RUNNER EXISTS AT ALL +# ============================================================================ +# Until 2026-08-01 the estate had exactly one registered runner, on the desktop. +# It went offline at 2026-07-31 16:31Z and everything stopped for 21 hours: no +# merge could satisfy a required check, no deploy could run, and the 03:00Z +# ops-cron -- THE backup for both application databases and for Forgejo -- did +# not fire. Forgejo queued that run rather than dropping it, so it completed on +# reconnect, but a longer outage would have meant real backup gaps. +# +# So this runner is not extra capacity. It is the one that has to be up. +# +# ============================================================================ +# WHY A PLAIN CONTAINER AND NOT A SWARM SERVICE +# ============================================================================ +# Same argument as infra/openbao/config/openbao.hcl makes for OpenBao: the thing +# that repairs the estate must not depend on the estate. A Swarm-scheduled +# runner cannot redeploy the Swarm it is scheduled by, and if the cluster is the +# thing that is broken, CI is gone exactly when it is needed. `restart: always` +# on the local daemon has one dependency: the local daemon. +# +# An earlier revision of deploy/forgejo/docker-stack.yml did run a runner as a +# Swarm-scheduled Docker-in-Docker sidecar. It was removed, and the docs kept +# claiming an "always-on Swarm-hosted runner" for weeks afterwards -- which is +# how the single point of failure went unnoticed. +# +# ============================================================================ +# WHAT MOUNTING THE DOCKER SOCKET ON THE PROD HOST MEANS +# ============================================================================ +# Be plain about it: /var/run/docker.sock is root-equivalent on vps2, and vps2 +# runs prod. A workflow that can start a container can mount / and read +# /etc/thermograph.env. This runner does not create that exposure from nothing +# -- Forgejo's own database already stores VPS2_SSH_KEY, which is root on this +# box -- but it does move the exposure from "a secret at rest" to "a secret a +# job can reach", and that is a real difference. +# +# What bounds it, in order of how much each is worth: +# 1. Only this repo's workflows run here, and only the owner can push to it. +# 2. config.yaml sets valid_volumes: [] -- jobs cannot bind-mount host paths. +# 3. capacity: 1 and --cpus/--memory keep a job from contending with prod. +# 4. This compose project joins no thermograph network, so a job container +# reaches prod's services no more easily than any other host process. +# +# It is defence against accident, not against a hostile workflow author. On a +# two-person estate where both people can already SSH to this box as root, that +# is the honest boundary. + +name: forgejo-runner + +services: + runner: + image: code.forgejo.org/forgejo/runner:6.3.1 + restart: always + # The socket is root:docker on the host; the image's default user is not in + # that group. Running as root is what the socket mount requires. + user: root + working_dir: /data + command: forgejo-runner daemon --config /data/config.yaml + volumes: + - ./data:/data + - /var/run/docker.sock:/var/run/docker.sock + # Bounds the DAEMON. Job containers are siblings started through the socket, + # not children, so they are unaffected by these -- container.options in + # config.yaml is what bounds those. + cpus: 1.0 + mem_limit: 1g + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3"