Jinemi/thermograph
4
0
Fork 0
Code Issues 2 Pull requests 9 Projects Releases Packages Wiki Activity Actions
thermograph/infra/deploy/forgejo/runner-vps2/README.md

152 lines
6.4 KiB
Markdown
Raw Normal View History

ci: add an always-on Actions runner on vps2 The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up.
2026-08-01 14:54:28 +00:00
# The always-on Actions runner (vps2)
The runner CI actually depends on. The desktop runner
(`../register-lan-runner.sh`) stays registered as extra capacity, but nothing in
the estate may assume it is up.
Labels: `docker` only. `thermograph-lan` is deliberately **not** registered here
— it was the host-native LAN-deploy label, dev moved to vps1 and is deployed
over SSH like beta and prod, and no workflow requests it any more.
## Why this exists
On 2026-07-31 16:31Z the desktop — then the estate's **only** registered runner
— went offline. For the next 21 hours nothing merged (every protected branch
requires a check that only a runner can produce), nothing deployed, and the
03:00Z `ops-cron` did not run. Forgejo held that scheduled run in `waiting`
rather than dropping it, so all three jobs completed on reconnect and no backup
was actually lost. A longer outage would not have been so kind.
`docker-stack.yml` and this directory's parent README both described an
"always-on Swarm-hosted runner" during that entire window. There wasn't one; an
early revision had it as a Docker-in-Docker sidecar and it was removed. That is
why the single point of failure went unnoticed for weeks, and it is the reason
the root `CLAUDE.md` treats a stale doc as a correctness bug rather than a
documentation one.
## Prerequisites
- vps2's docker daemon is logged in to `git.thermograph.org` (it is —
`/root/.docker/config.json`), so the daemon can pull the job image.
- `git.thermograph.org` resolves to the **mesh** address from vps2 for registry
traffic. Job containers get this via `--add-host` in `config.yaml`; the host
daemon already has it.
registry: move image and repo references to the Jinemi namespace Repos moved to the Jinemi org; the container packages did not follow, since Forgejo does not transfer packages with a repo. The deploy path still resolved `emi/thermograph/*` — a user_redirect to admin_emi — while build-push.yml derives its push path from ${github.repository}, now jinemi/thermograph. The next backend or frontend build would have published somewhere no deploy looks. Point the image paths at jinemi/thermograph/* (lowercase: OCI references admit no uppercase, which is why build-push.yml already pipes through tr), the clone URLs at Jinemi/thermograph, and the registry logins at admin_emi — the account that actually owns the tokens, rather than the redirect. The live tags and both ci-runner tags were copied into the Jinemi namespace first, so the switch has something to pull. thermograph-infra, thermograph-observability and the retired */app packages stay under admin_emi; they did not move.
2026-07-31 03:53:26 +00:00
- The job image exists: `git.thermograph.org/jinemi/thermograph/ci-runner:v2`.
ci: add an always-on Actions runner on vps2 The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up.
2026-08-01 14:54:28 +00:00
Built from `../ci-runner/Dockerfile`; see that file for when to rebuild.
## Where it is deployed, and why not from the checkout
**`/opt/forgejo-runner/` on vps2**, holding a copy of `docker-compose.yml` and
`config.yaml` from this directory, plus a `data/` the runner owns.
Deliberately *not* run in place from `/opt/thermograph/…/runner-vps2/`. That
checkout is `git reset --hard`'d on every prod deploy; untracked files do
survive that (it is the same property `deploy/.image-tags.env` relies on), but a
single `git clean -fdx` during troubleshooting would delete `data/.runner` and
de-register the runner. Putting the thing that repairs the estate inside the
tree the estate rewrites is the same mistake as scheduling it on the Swarm it
deploys.
So this directory is the source of truth; vps2 gets a copy:
forgejo: make the desktop runner unit restart-always, fix the vps2 config path Both runners exist and are up; these are the two defects found while verifying that. The desktop's systemd --user unit had Restart=on-failure. That is the same distinction that took Forgejo down for 27 hours on 2026-07-29 (docker-stack.yml records it for the db service): a daemon that exits 0 is not "finished successfully", and on-failure cannot tell that from a clean shutdown. The failure mode is silent — unit inactive (dead), runner offline in the UI, and every protected-branch merge blocked on a check nothing will produce. The StartLimit directives that bound that retry loop were in [Service], where systemd accepts them without complaint and ignores them; the live unit was running the 10s default rather than the intended 300s. Moved to [Unit], which is where they are read. runner-vps2/README told you to copy config.yaml next to docker-compose.yml, but the compose file mounts ./data:/data and loads --config /data/config.yaml, so a config there is invisible to the container — the daemon starts on defaults with no --add-host, and every registry push then fails as if the credential were wrong. vps2 had a stray copy at the documented path proving the instruction had been followed.
2026-08-01 18:50:42 +00:00
The two files do **not** go to the same place, and getting it wrong fails in a
confusing way. `docker-compose.yml` mounts `./data:/data` and starts the daemon
with `--config /data/config.yaml`, so `/data` *is* `data/` — a `config.yaml`
sitting beside the compose file is invisible to the container. The daemon then
either refuses to start or silently runs on defaults (no `--add-host`, so every
registry push fails as if the credential were wrong).
ci: add an always-on Actions runner on vps2 The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up.
2026-08-01 14:54:28 +00:00
```bash
forgejo: make the desktop runner unit restart-always, fix the vps2 config path Both runners exist and are up; these are the two defects found while verifying that. The desktop's systemd --user unit had Restart=on-failure. That is the same distinction that took Forgejo down for 27 hours on 2026-07-29 (docker-stack.yml records it for the db service): a daemon that exits 0 is not "finished successfully", and on-failure cannot tell that from a clean shutdown. The failure mode is silent — unit inactive (dead), runner offline in the UI, and every protected-branch merge blocked on a check nothing will produce. The StartLimit directives that bound that retry loop were in [Service], where systemd accepts them without complaint and ignores them; the live unit was running the 10s default rather than the intended 300s. Moved to [Unit], which is where they are read. runner-vps2/README told you to copy config.yaml next to docker-compose.yml, but the compose file mounts ./data:/data and loads --config /data/config.yaml, so a config there is invisible to the container — the daemon starts on defaults with no --add-host, and every registry push then fails as if the credential were wrong. vps2 had a stray copy at the documented path proving the instruction had been followed.
2026-08-01 18:50:42 +00:00
install -d -m 0755 /opt/forgejo-runner/data
ci: add an always-on Actions runner on vps2 The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up.
2026-08-01 14:54:28 +00:00
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \
/opt/forgejo-runner/
forgejo: make the desktop runner unit restart-always, fix the vps2 config path Both runners exist and are up; these are the two defects found while verifying that. The desktop's systemd --user unit had Restart=on-failure. That is the same distinction that took Forgejo down for 27 hours on 2026-07-29 (docker-stack.yml records it for the db service): a daemon that exits 0 is not "finished successfully", and on-failure cannot tell that from a clean shutdown. The failure mode is silent — unit inactive (dead), runner offline in the UI, and every protected-branch merge blocked on a check nothing will produce. The StartLimit directives that bound that retry loop were in [Service], where systemd accepts them without complaint and ignores them; the live unit was running the 10s default rather than the intended 300s. Moved to [Unit], which is where they are read. runner-vps2/README told you to copy config.yaml next to docker-compose.yml, but the compose file mounts ./data:/data and loads --config /data/config.yaml, so a config there is invisible to the container — the daemon starts on defaults with no --add-host, and every registry push then fails as if the credential were wrong. vps2 had a stray copy at the documented path proving the instruction had been followed.
2026-08-01 18:50:42 +00:00
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
/opt/forgejo-runner/data/
ci: add an always-on Actions runner on vps2 The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up.
2026-08-01 14:54:28 +00:00
```
forgejo: make the desktop runner unit restart-always, fix the vps2 config path Both runners exist and are up; these are the two defects found while verifying that. The desktop's systemd --user unit had Restart=on-failure. That is the same distinction that took Forgejo down for 27 hours on 2026-07-29 (docker-stack.yml records it for the db service): a daemon that exits 0 is not "finished successfully", and on-failure cannot tell that from a clean shutdown. The failure mode is silent — unit inactive (dead), runner offline in the UI, and every protected-branch merge blocked on a check nothing will produce. The StartLimit directives that bound that retry loop were in [Service], where systemd accepts them without complaint and ignores them; the live unit was running the 10s default rather than the intended 300s. Moved to [Unit], which is where they are read. runner-vps2/README told you to copy config.yaml next to docker-compose.yml, but the compose file mounts ./data:/data and loads --config /data/config.yaml, so a config there is invisible to the container — the daemon starts on defaults with no --add-host, and every registry push then fails as if the credential were wrong. vps2 had a stray copy at the documented path proving the instruction had been followed.
2026-08-01 18:50:42 +00:00
`data/` is also where `register` writes `.runner`, so the runner's whole state —
config plus registration — lives in the one directory the compose file mounts.
ci: add an always-on Actions runner on vps2 The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up.
2026-08-01 14:54:28 +00:00
Re-copy after changing either file here, then `docker compose up -d`. There is
no automation for that step and there should not be: a workflow that redeploys
the runner runs *on* the runner.
## One-time registration
The registration token is short-lived and single-purpose. Fetch it from the
Forgejo UI (**Site Administration → Actions → Runners → Create new Runner**) or
the API, and keep it out of shell history — it is a credential, however
briefly.
```bash
cd /opt/forgejo-runner
mkdir -p data
cp config.yaml data/config.yaml
read -rs REG_TOKEN && export REG_TOKEN # paste; not echoed, not in history
docker run --rm -it \
-v "$PWD/data:/data" -w /data --user root \
code.forgejo.org/forgejo/runner:6.3.1 \
forgejo-runner register --no-interactive \
--instance https://git.thermograph.org \
--token "$REG_TOKEN" \
--name thermograph-vps2 \
registry: move image and repo references to the Jinemi namespace Repos moved to the Jinemi org; the container packages did not follow, since Forgejo does not transfer packages with a repo. The deploy path still resolved `emi/thermograph/*` — a user_redirect to admin_emi — while build-push.yml derives its push path from ${github.repository}, now jinemi/thermograph. The next backend or frontend build would have published somewhere no deploy looks. Point the image paths at jinemi/thermograph/* (lowercase: OCI references admit no uppercase, which is why build-push.yml already pipes through tr), the clone URLs at Jinemi/thermograph, and the registry logins at admin_emi — the account that actually owns the tokens, rather than the redirect. The live tags and both ci-runner tags were copied into the Jinemi namespace first, so the switch has something to pull. thermograph-infra, thermograph-observability and the retired */app packages stay under admin_emi; they did not move.
2026-07-31 03:53:26 +00:00
--labels 'docker:docker://git.thermograph.org/jinemi/thermograph/ci-runner:v2'
ci: add an always-on Actions runner on vps2 The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up.
2026-08-01 14:54:28 +00:00
unset REG_TOKEN
```
That writes `data/.runner`, which holds the **long-lived** runner token. It is
`0600` and must stay out of git — `data/` is gitignored for exactly this reason.
Re-running `register` with a fresh registration token replaces it; the old
registration then shows offline in the UI and should be deleted there.
## Run it
```bash
docker compose up -d
docker compose logs -f --tail=50 runner # expect "runner: ... successfully activated"
```
Confirm Forgejo agrees, rather than trusting the log:
```bash
# on vps1, where Forgejo's database lives
fdb=$(docker ps -qf name=forgejo_db | head -1)
docker exec "$fdb" sh -c 'psql -U $POSTGRES_USER -d $POSTGRES_DB -c \
"select name, to_timestamp(last_online) as last_online, agent_labels from action_runner order by id;"'
```
Two rows, both recent. `last_online` is the only honest liveness signal — a
runner process that is up but cannot reach Forgejo looks healthy locally and is
useless.
## Upgrading
Pin changes go in `docker-compose.yml` (`image:`) and are applied with
`docker compose up -d`. The runner token survives an image change; it lives in
`data/.runner`, not in the image.
Changing the **job** image (the `docker://…ci-runner:vN` label) is a
re-registration or a hand-edit of `data/.runner`'s `labels` array — the same
caveat `../ci-runner/Dockerfile` records for the desktop runner. Keep the two
runners on the same job image or a workflow's behaviour starts depending on
which runner happened to pick it up.
## Troubleshooting
**Jobs queue forever with both runners online.** The workflow asked for a label
nobody registered. `runs-on: docker` is the only label this estate uses.
**`docker pull` fails inside a job, works on the host.** The `--add-host` line
in `config.yaml` is missing or wrong. vps1's Caddy serves `/v2/*` to the mesh
only, and the job container has no other way to learn the mesh address.
**Jobs fail with permission denied on the socket.** `user: root` was dropped
from the compose file.
**The runner is up but `last_online` is stale.** It cannot reach
`https://git.thermograph.org` — check egress from vps2 and that Forgejo on vps1
is serving.
Reference in a new issue Copy permalink