ci: add an always-on Actions runner on vps2
The estate had exactly one registered runner, on the desktop. It went offline
at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required
check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both
application databases and for Forgejo -- did not fire. Forgejo queued that
scheduled run rather than dropping it, so it completed on reconnect and nothing
was lost. A longer outage would have meant real gaps.
Three files claimed an "always-on Swarm-hosted runner" existed and that the
estate therefore no longer depended on the desktop. It did not exist: an early
revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was
removed. That claim is why a single point of failure sat unnoticed. Corrected
in docker-stack.yml, forgejo/README.md and register-lan-runner.sh.
The new runner is a plain restart:always container, not a Swarm service: a
Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would
be gone exactly when the cluster is what is broken. It runs from
/opt/forgejo-runner rather than in place, because the checkout is reset on
every prod deploy and one `git clean -fdx` there would destroy the
registration.
vps2 runs prod, so the socket mount is bounded rather than assumed benign:
capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no
job can bind-mount /etc/thermograph.env, and no thermograph network joined.
This defends against accident, not against a hostile workflow author -- stated
plainly in the compose header rather than implied.
The desktop runner stays registered as extra capacity. Nothing may assume it
is up.
2026-08-01 14:54:28 +00:00
|
|
|
# Forgejo Actions runner config for the vps2 runner. Mounted at /data/config.yaml
|
|
|
|
|
# by runner-vps2/docker-compose.yml.
|
|
|
|
|
#
|
|
|
|
|
# Values that differ from `forgejo-runner generate-config` defaults are the only
|
|
|
|
|
# ones written out below; everything else is left at its default deliberately, so
|
|
|
|
|
# a future runner upgrade inherits new defaults instead of a frozen copy of the
|
|
|
|
|
# old ones.
|
|
|
|
|
|
|
|
|
|
log:
|
|
|
|
|
level: info
|
|
|
|
|
job_level: info
|
|
|
|
|
|
|
|
|
|
runner:
|
|
|
|
|
file: .runner
|
|
|
|
|
|
|
|
|
|
# ONE job at a time. The desktop runner uses capacity 8, which is right for a
|
|
|
|
|
# box whose whole job is CI. This runner shares a host with prod, so the
|
|
|
|
|
# binding constraint is not throughput, it is that a CI burst must never be
|
|
|
|
|
# able to contend with thermograph_web for CPU. One job at a time also makes
|
|
|
|
|
# the resource ceiling below exact rather than a per-job estimate multiplied
|
|
|
|
|
# by an unknown.
|
|
|
|
|
capacity: 1
|
|
|
|
|
|
|
|
|
|
timeout: 3h
|
|
|
|
|
shutdown_timeout: 3h
|
|
|
|
|
|
|
|
|
|
container:
|
registry: move the registry host to dev.jinemi.com, MCP to mcp.jinemi.com
Completes the Forgejo domain migration. ROOT_URL moved to dev.jinemi.com
earlier; the registry half was deliberately deferred. Every image name,
REGISTRY_HOST default, runner label and --add-host pin now names
dev.jinemi.com, so the registry host and the bearer-token realm agree again.
Both names address the same Forgejo, so no image needs re-pushing and a
rollback to a tag pushed under the old prefix still resolves.
git.thermograph.org therefore stays served off the same Caddy site block --
one block, so the /v2/* mesh-only matcher keeps covering both names -- for
pre-migration tags and for runners holding it as their registered instance
URL.
Mesh clients now pin both names in /etc/hosts: the new one as registry host
and token realm, the old one for pre-migration tags. runner-vps2/config.yaml
carries both --add-host entries for the same reason.
Also renames Centralis' endpoint to mcp.jinemi.com in the two places this
repo names it; Centralis itself is provisioned outside this repo.
Host-side steps this cannot do (documented in deploy/forgejo/README.md,
"Host-side steps"): the Forgejo Actions variable REGISTRY_HOST, docker login
against the new host, and the /etc/hosts pins.
2026-08-01 23:06:22 +00:00
|
|
|
# 1) --add-host: both Forgejo names resolve publicly to vps1, but vps1's
|
ci: add an always-on Actions runner on vps2
The estate had exactly one registered runner, on the desktop. It went offline
at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required
check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both
application databases and for Forgejo -- did not fire. Forgejo queued that
scheduled run rather than dropping it, so it completed on reconnect and nothing
was lost. A longer outage would have meant real gaps.
Three files claimed an "always-on Swarm-hosted runner" existed and that the
estate therefore no longer depended on the desktop. It did not exist: an early
revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was
removed. That claim is why a single point of failure sat unnoticed. Corrected
in docker-stack.yml, forgejo/README.md and register-lan-runner.sh.
The new runner is a plain restart:always container, not a Swarm service: a
Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would
be gone exactly when the cluster is what is broken. It runs from
/opt/forgejo-runner rather than in place, because the checkout is reset on
every prod deploy and one `git clean -fdx` there would destroy the
registration.
vps2 runs prod, so the socket mount is bounded rather than assumed benign:
capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no
job can bind-mount /etc/thermograph.env, and no thermograph network joined.
This defends against accident, not against a hostile workflow author -- stated
plainly in the compose header rather than implied.
The desktop runner stays registered as extra capacity. Nothing may assume it
is up.
2026-08-01 14:54:28 +00:00
|
|
|
# Caddy rejects the /v2/* registry API from off-mesh. Job containers that
|
|
|
|
|
# docker-pull or docker-push therefore need the MESH address, and DNS will
|
|
|
|
|
# not give it to them. Same line the desktop runner carries; the value is
|
|
|
|
|
# the same from vps2 because both boxes are on 10.10.0.0/24.
|
|
|
|
|
#
|
registry: move the registry host to dev.jinemi.com, MCP to mcp.jinemi.com
Completes the Forgejo domain migration. ROOT_URL moved to dev.jinemi.com
earlier; the registry half was deliberately deferred. Every image name,
REGISTRY_HOST default, runner label and --add-host pin now names
dev.jinemi.com, so the registry host and the bearer-token realm agree again.
Both names address the same Forgejo, so no image needs re-pushing and a
rollback to a tag pushed under the old prefix still resolves.
git.thermograph.org therefore stays served off the same Caddy site block --
one block, so the /v2/* mesh-only matcher keeps covering both names -- for
pre-migration tags and for runners holding it as their registered instance
URL.
Mesh clients now pin both names in /etc/hosts: the new one as registry host
and token realm, the old one for pre-migration tags. runner-vps2/config.yaml
carries both --add-host entries for the same reason.
Also renames Centralis' endpoint to mcp.jinemi.com in the two places this
repo names it; Centralis itself is provisioned outside this repo.
Host-side steps this cannot do (documented in deploy/forgejo/README.md,
"Host-side steps"): the Forgejo Actions variable REGISTRY_HOST, docker login
against the new host, and the /etc/hosts pins.
2026-08-01 23:06:22 +00:00
|
|
|
# BOTH names are pinned, and that is not belt-and-braces. dev.jinemi.com is
|
|
|
|
|
# the registry host in image names AND Forgejo's ROOT_URL, from which the
|
|
|
|
|
# registry's bearer-token realm is derived -- so a pull needs it twice over.
|
|
|
|
|
# git.thermograph.org stays pinned because already-pushed tags carry it as
|
|
|
|
|
# their prefix; unpinned, a rollback to such a tag takes the public route
|
|
|
|
|
# and gets a 403 that reads like a credential failure. See
|
|
|
|
|
# ../README.md, "Migrating the domain".
|
|
|
|
|
#
|
ci: add an always-on Actions runner on vps2
The estate had exactly one registered runner, on the desktop. It went offline
at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required
check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both
application databases and for Forgejo -- did not fire. Forgejo queued that
scheduled run rather than dropping it, so it completed on reconnect and nothing
was lost. A longer outage would have meant real gaps.
Three files claimed an "always-on Swarm-hosted runner" existed and that the
estate therefore no longer depended on the desktop. It did not exist: an early
revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was
removed. That claim is why a single point of failure sat unnoticed. Corrected
in docker-stack.yml, forgejo/README.md and register-lan-runner.sh.
The new runner is a plain restart:always container, not a Swarm service: a
Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would
be gone exactly when the cluster is what is broken. It runs from
/opt/forgejo-runner rather than in place, because the checkout is reset on
every prod deploy and one `git clean -fdx` there would destroy the
registration.
vps2 runs prod, so the socket mount is bounded rather than assumed benign:
capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no
job can bind-mount /etc/thermograph.env, and no thermograph network joined.
This defends against accident, not against a hostile workflow author -- stated
plainly in the compose header rather than implied.
The desktop runner stays registered as extra capacity. Nothing may assume it
is up.
2026-08-01 14:54:28 +00:00
|
|
|
# 2) --cpus/--memory: vps2 runs prod. Without a ceiling a runaway job (a
|
|
|
|
|
# `docker build` that fans out, a test that leaks) competes with the
|
|
|
|
|
# production app for the same 6 cores. These bound the JOB container, which
|
|
|
|
|
# is a sibling of the runner rather than its child, so the runner's own
|
|
|
|
|
# compose limits do not cover it -- this is the only lever that does.
|
registry: move the registry host to dev.jinemi.com, MCP to mcp.jinemi.com
Completes the Forgejo domain migration. ROOT_URL moved to dev.jinemi.com
earlier; the registry half was deliberately deferred. Every image name,
REGISTRY_HOST default, runner label and --add-host pin now names
dev.jinemi.com, so the registry host and the bearer-token realm agree again.
Both names address the same Forgejo, so no image needs re-pushing and a
rollback to a tag pushed under the old prefix still resolves.
git.thermograph.org therefore stays served off the same Caddy site block --
one block, so the /v2/* mesh-only matcher keeps covering both names -- for
pre-migration tags and for runners holding it as their registered instance
URL.
Mesh clients now pin both names in /etc/hosts: the new one as registry host
and token realm, the old one for pre-migration tags. runner-vps2/config.yaml
carries both --add-host entries for the same reason.
Also renames Centralis' endpoint to mcp.jinemi.com in the two places this
repo names it; Centralis itself is provisioned outside this repo.
Host-side steps this cannot do (documented in deploy/forgejo/README.md,
"Host-side steps"): the Forgejo Actions variable REGISTRY_HOST, docker login
against the new host, and the /etc/hosts pins.
2026-08-01 23:06:22 +00:00
|
|
|
options: --add-host=dev.jinemi.com:10.10.0.2 --add-host=git.thermograph.org:10.10.0.2 --cpus=2 --memory=4g
|
ci: add an always-on Actions runner on vps2
The estate had exactly one registered runner, on the desktop. It went offline
at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required
check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both
application databases and for Forgejo -- did not fire. Forgejo queued that
scheduled run rather than dropping it, so it completed on reconnect and nothing
was lost. A longer outage would have meant real gaps.
Three files claimed an "always-on Swarm-hosted runner" existed and that the
estate therefore no longer depended on the desktop. It did not exist: an early
revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was
removed. That claim is why a single point of failure sat unnoticed. Corrected
in docker-stack.yml, forgejo/README.md and register-lan-runner.sh.
The new runner is a plain restart:always container, not a Swarm service: a
Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would
be gone exactly when the cluster is what is broken. It runs from
/opt/forgejo-runner rather than in place, because the checkout is reset on
every prod deploy and one `git clean -fdx` there would destroy the
registration.
vps2 runs prod, so the socket mount is bounded rather than assumed benign:
capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no
job can bind-mount /etc/thermograph.env, and no thermograph network joined.
This defends against accident, not against a hostile workflow author -- stated
plainly in the compose header rather than implied.
The desktop runner stays registered as extra capacity. Nothing may assume it
is up.
2026-08-01 14:54:28 +00:00
|
|
|
|
|
|
|
|
# Jobs may mount NOTHING from the host. The workflows here do not need to:
|
|
|
|
|
# they check out into the job container's own workspace and reach the estate
|
|
|
|
|
# over SSH. An empty list is the difference between "a job can read
|
|
|
|
|
# /etc/thermograph.env" and "a job cannot", on the box where that file is
|
|
|
|
|
# prod's.
|
|
|
|
|
valid_volumes: []
|
|
|
|
|
|
|
|
|
|
privileged: false
|
|
|
|
|
|
|
|
|
|
# Mounts the host docker socket into the job container. build-push.yml runs
|
|
|
|
|
# `docker build`, so it needs a daemon; this points it at vps2's. See the
|
|
|
|
|
# README for what that does and does not mean for trust.
|
|
|
|
|
docker_host: "automount"
|