001e6b1365
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 5s
shell-lint / shellcheck (pull_request) Successful in 8s
PR build (required check) / changes (pull_request) Successful in 13s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 7s
secrets-guard / encrypted (push) Successful in 7s
shell-lint / shellcheck (push) Successful in 7s
The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up.
83 lines
4 KiB
YAML
83 lines
4 KiB
YAML
# The always-on Forgejo Actions runner, on vps2.
|
|
#
|
|
# cd /opt/forgejo-runner # a COPY of this file, not the checkout
|
|
# docker compose up -d
|
|
#
|
|
# It runs from /opt/forgejo-runner rather than in place, because the checkout is
|
|
# git reset --hard'd on every prod deploy and one `git clean -fdx` there would
|
|
# delete data/.runner. See README.md.
|
|
#
|
|
# Registration is a one-off and is NOT in this file, because it takes a
|
|
# short-lived token a human fetches. See README.md.
|
|
#
|
|
# ============================================================================
|
|
# WHY A SECOND RUNNER EXISTS AT ALL
|
|
# ============================================================================
|
|
# Until 2026-08-01 the estate had exactly one registered runner, on the desktop.
|
|
# It went offline at 2026-07-31 16:31Z and everything stopped for 21 hours: no
|
|
# merge could satisfy a required check, no deploy could run, and the 03:00Z
|
|
# ops-cron -- THE backup for both application databases and for Forgejo -- did
|
|
# not fire. Forgejo queued that run rather than dropping it, so it completed on
|
|
# reconnect, but a longer outage would have meant real backup gaps.
|
|
#
|
|
# So this runner is not extra capacity. It is the one that has to be up.
|
|
#
|
|
# ============================================================================
|
|
# WHY A PLAIN CONTAINER AND NOT A SWARM SERVICE
|
|
# ============================================================================
|
|
# Same argument as infra/openbao/config/openbao.hcl makes for OpenBao: the thing
|
|
# that repairs the estate must not depend on the estate. A Swarm-scheduled
|
|
# runner cannot redeploy the Swarm it is scheduled by, and if the cluster is the
|
|
# thing that is broken, CI is gone exactly when it is needed. `restart: always`
|
|
# on the local daemon has one dependency: the local daemon.
|
|
#
|
|
# An earlier revision of deploy/forgejo/docker-stack.yml did run a runner as a
|
|
# Swarm-scheduled Docker-in-Docker sidecar. It was removed, and the docs kept
|
|
# claiming an "always-on Swarm-hosted runner" for weeks afterwards -- which is
|
|
# how the single point of failure went unnoticed.
|
|
#
|
|
# ============================================================================
|
|
# WHAT MOUNTING THE DOCKER SOCKET ON THE PROD HOST MEANS
|
|
# ============================================================================
|
|
# Be plain about it: /var/run/docker.sock is root-equivalent on vps2, and vps2
|
|
# runs prod. A workflow that can start a container can mount / and read
|
|
# /etc/thermograph.env. This runner does not create that exposure from nothing
|
|
# -- Forgejo's own database already stores VPS2_SSH_KEY, which is root on this
|
|
# box -- but it does move the exposure from "a secret at rest" to "a secret a
|
|
# job can reach", and that is a real difference.
|
|
#
|
|
# What bounds it, in order of how much each is worth:
|
|
# 1. Only this repo's workflows run here, and only the owner can push to it.
|
|
# 2. config.yaml sets valid_volumes: [] -- jobs cannot bind-mount host paths.
|
|
# 3. capacity: 1 and --cpus/--memory keep a job from contending with prod.
|
|
# 4. This compose project joins no thermograph network, so a job container
|
|
# reaches prod's services no more easily than any other host process.
|
|
#
|
|
# It is defence against accident, not against a hostile workflow author. On a
|
|
# two-person estate where both people can already SSH to this box as root, that
|
|
# is the honest boundary.
|
|
|
|
name: forgejo-runner
|
|
|
|
services:
|
|
runner:
|
|
image: code.forgejo.org/forgejo/runner:6.3.1
|
|
restart: always
|
|
# The socket is root:docker on the host; the image's default user is not in
|
|
# that group. Running as root is what the socket mount requires.
|
|
user: root
|
|
working_dir: /data
|
|
command: forgejo-runner daemon --config /data/config.yaml
|
|
volumes:
|
|
- ./data:/data
|
|
- /var/run/docker.sock:/var/run/docker.sock
|
|
# Bounds the DAEMON. Job containers are siblings started through the socket,
|
|
# not children, so they are unaffected by these -- container.options in
|
|
# config.yaml is what bounds those.
|
|
cpus: 1.0
|
|
mem_limit: 1g
|
|
logging:
|
|
driver: json-file
|
|
options:
|
|
max-size: "10m"
|
|
max-file: "3"
|