forgejo: make the desktop runner unit restart-always, fix the vps2 config path
All checks were successful
PR build (required check) / changes (pull_request) Successful in 9s
shell-lint / shellcheck (pull_request) Successful in 9s
secrets-guard / encrypted (pull_request) Successful in 11s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s

Both runners exist and are up; these are the two defects found while verifying
that.

The desktop's systemd --user unit had Restart=on-failure. That is the same
distinction that took Forgejo down for 27 hours on 2026-07-29 (docker-stack.yml
records it for the db service): a daemon that exits 0 is not "finished
successfully", and on-failure cannot tell that from a clean shutdown. The
failure mode is silent — unit inactive (dead), runner offline in the UI, and
every protected-branch merge blocked on a check nothing will produce.

The StartLimit directives that bound that retry loop were in [Service], where
systemd accepts them without complaint and ignores them; the live unit was
running the 10s default rather than the intended 300s. Moved to [Unit], which
is where they are read.

runner-vps2/README told you to copy config.yaml next to docker-compose.yml, but
the compose file mounts ./data:/data and loads --config /data/config.yaml, so a
config there is invisible to the container — the daemon starts on defaults with
no --add-host, and every registry push then fails as if the credential were
wrong. vps2 had a stray copy at the documented path proving the instruction had
been followed.
This commit is contained in:
Emi Griffith 2026-08-01 11:50:42 -07:00
parent 62ba381d06
commit f25466ca76
2 changed files with 30 additions and 3 deletions

View file

@ -93,6 +93,13 @@ cat > "$HOME/.config/systemd/user/forgejo-runner.service" <<EOF
[Unit]
Description=Forgejo Actions runner (docker + thermograph-lan)
After=network-online.target
# These bound the Restart=always loop below, so a genuinely broken config (bad
# token, docker unreachable) ends as a stopped unit rather than spinning
# forever. They belong in [Unit]: systemd accepts them in [Service] without
# complaint but IGNORES them, leaving the 10s default — \`systemctl --user show
# forgejo-runner -p StartLimitIntervalUSec\` is how you tell which one you got.
StartLimitIntervalSec=300
StartLimitBurst=5
[Service]
WorkingDirectory=${RUNNER_DIR}
@ -102,7 +109,16 @@ WorkingDirectory=${RUNNER_DIR}
# SOPS-configured) fails with "not installed" even though it plainly is.
Environment=PATH=%h/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
ExecStart=${RUNNER_DIR}/forgejo-runner daemon -c config.yaml
Restart=on-failure
# \`always\`, NOT \`on-failure\` — the same distinction that took Forgejo down for
# 27 hours on 2026-07-29 (see docker-stack.yml's restart_policy comment): an
# always-on daemon that exits **0** is not "finished successfully", it is a
# daemon that stopped and must come back. \`on-failure\` cannot tell those apart,
# and forgejo-runner exits 0 on several paths — a lost instance connection it
# gives up on, or a SIGTERM from a Docker restart it interprets as a clean
# shutdown. The failure mode is silent: the unit sits \`inactive (dead)\`, the UI
# shows the runner offline, and every protected-branch merge blocks on a check
# that will never be produced.
Restart=always
RestartSec=5
[Install]

View file

@ -49,13 +49,24 @@ deploys.
So this directory is the source of truth; vps2 gets a copy:
The two files do **not** go to the same place, and getting it wrong fails in a
confusing way. `docker-compose.yml` mounts `./data:/data` and starts the daemon
with `--config /data/config.yaml`, so `/data` *is* `data/` — a `config.yaml`
sitting beside the compose file is invisible to the container. The daemon then
either refuses to start or silently runs on defaults (no `--add-host`, so every
registry push fails as if the credential were wrong).
```bash
install -d -m 0755 /opt/forgejo-runner
install -d -m 0755 /opt/forgejo-runner/data
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \
/opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
/opt/forgejo-runner/
cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
/opt/forgejo-runner/data/
```
`data/` is also where `register` writes `.runner`, so the runner's whole state —
config plus registration — lives in the one directory the compose file mounts.
Re-copy after changing either file here, then `docker compose up -d`. There is
no automation for that step and there should not be: a workflow that redeploys
the runner runs *on* the runner.