ci: add an always-on Actions runner on vps2

admin_emi commented

2026-08-01 14:54:52 +00:00

Owner

The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron — the only backup for both application databases and for Forgejo — did not fire. Forgejo held that scheduled run in waiting rather than dropping it, so all three jobs completed on reconnect and no backup was actually lost. A longer outage would not have been so kind.

Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. No such runner existed — an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That stale claim is why the single point of failure went unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh.

Design notes, both deliberate:

Plain container, not a Swarm service. A Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be unavailable exactly when the cluster is what needs repairing. Same argument openbao.hcl makes for OpenBao.
Runs from /opt/forgejo-runner, not in place. The checkout is git reset --hard on every prod deploy — untracked files survive that, but one git clean -fdx during troubleshooting would delete data/.runner and de-register the runner.

vps2 runs prod, so the docker-socket mount is bounded rather than assumed benign: capacity 1, --cpus=2 --memory=4g on job containers, valid_volumes: [] so no job can bind-mount /etc/thermograph.env, and the compose project joins no thermograph network. The compose header states plainly that this is defence against accident, not against a hostile workflow author — Forgejo already stores VPS2_SSH_KEY, which is root on this box.

The desktop runner stays registered as extra capacity. Nothing may assume it is up.

The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron — the only backup for both application databases and for Forgejo — did not fire. Forgejo held that scheduled run in `waiting` rather than dropping it, so all three jobs completed on reconnect and no backup was actually lost. A longer outage would not have been so kind. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. No such runner existed — an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That stale claim is why the single point of failure went unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. Design notes, both deliberate: - **Plain container, not a Swarm service.** A Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be unavailable exactly when the cluster is what needs repairing. Same argument openbao.hcl makes for OpenBao. - **Runs from /opt/forgejo-runner, not in place.** The checkout is `git reset --hard` on every prod deploy — untracked files survive that, but one `git clean -fdx` during troubleshooting would delete data/.runner and de-register the runner. vps2 runs prod, so the docker-socket mount is bounded rather than assumed benign: capacity 1, `--cpus=2 --memory=4g` on job containers, `valid_volumes: []` so no job can bind-mount /etc/thermograph.env, and the compose project joins no thermograph network. The compose header states plainly that this is defence against accident, not against a hostile workflow author — Forgejo already stores VPS2_SSH_KEY, which is root on this box. The desktop runner stays registered as extra capacity. Nothing may assume it is up.

admin_emi added 1 commit 2026-08-01 14:54:53 +00:00

ci: add an always-on Actions runner on vps2

secrets-guard / encrypted (pull_request) Successful in 5s

Details

shell-lint / shellcheck (pull_request) Successful in 8s

Details

PR build (required check) / changes (pull_request) Successful in 13s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

Sync infra to hosts / sync-beta (push) Has been skipped

Details

Sync infra to hosts / sync-prod (push) Has been skipped

Details

Sync infra to hosts / sync-dev (push) Successful in 7s

Details

secrets-guard / encrypted (push) Successful in 7s

Details

shell-lint / shellcheck (push) Successful in 7s

Details

001e6b1365

The estate had exactly one registered runner, on the desktop. It went offline
at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required
check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both
application databases and for Forgejo -- did not fire. Forgejo queued that
scheduled run rather than dropping it, so it completed on reconnect and nothing
was lost. A longer outage would have meant real gaps.

Three files claimed an "always-on Swarm-hosted runner" existed and that the
estate therefore no longer depended on the desktop. It did not exist: an early
revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was
removed. That claim is why a single point of failure sat unnoticed. Corrected
in docker-stack.yml, forgejo/README.md and register-lan-runner.sh.

The new runner is a plain restart:always container, not a Swarm service: a
Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would
be gone exactly when the cluster is what is broken. It runs from
/opt/forgejo-runner rather than in place, because the checkout is reset on
every prod deploy and one `git clean -fdx` there would destroy the
registration.

vps2 runs prod, so the socket mount is bounded rather than assumed benign:
capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no
job can bind-mount /etc/thermograph.env, and no thermograph network joined.
This defends against accident, not against a hostile workflow author -- stated
plainly in the compose header rather than implied.

The desktop runner stays registered as extra capacity. Nothing may assume it
is up.

admin_emi merged commit 001e6b1365 into dev

2026-08-01 15:35:56 +00:00

admin_emi referenced this pull request

2026-08-01 16:59:34 +00:00

forgejo: record why the runner is on vps2, rather than leaving a contradiction #153

ci: add an always-on Actions runner on vps2 #141