forgejo: make the desktop runner unit restart-always, fix the vps2 config path #157

Merged

emi merged 2 commits from feat/forgejo-runners into dev

2026-08-01 18:52:52 +00:00

emi commented

2026-08-01 18:51:04 +00:00

Owner

Both runners already exist and are healthy — thermograph-vps2 (restart: always, executing tasks) and the desktop's systemd --user unit (active, enabled, Linger=yes). Nothing needed creating. These are the two real defects found while verifying that.

Desktop unit: `Restart=on-failure` -> `Restart=always`

The same distinction that took Forgejo down for 27 hours on 2026-07-29, which docker-stack.yml already records for the db service: a daemon that exits 0 is not "finished successfully", it is one that stopped and must come back. on-failure cannot tell those apart. The failure mode is silent — unit inactive (dead), runner offline in the UI, and every protected-branch merge blocked on a check nothing will ever produce. That is exactly the shape of the 2026-07-31 outage.

`StartLimit*` were being ignored

They sat in [Service], where systemd accepts them without complaint and then ignores them. The live unit was running the 10s default, not the intended 300s:

# before
StartLimitIntervalUSec=10s
# after (moved to [Unit])
StartLimitIntervalUSec=5min

`runner-vps2/README` sent `config.yaml` to a path the container cannot see

The README said to copy docker-compose.yml and config.yaml together into /opt/forgejo-runner/. But the compose file mounts ./data:/data and starts with --config /data/config.yaml, so /data is data/ — a config beside the compose file is invisible. The daemon then runs on defaults with no --add-host, and every registry push fails in a way that looks exactly like a bad credential.

vps2 had a stray copy at the documented path, so the instruction had been followed and worked around. Removed it; the README now sends the two files to their separate correct destinations and says why.

Applied live

desktop: Restart=always, StartLimitIntervalUSec=5min — via daemon-reload only, with no job running, so nothing was interrupted
vps2: removed the misleading /opt/forgejo-runner/config.yaml; container still up
generated unit re-rendered and checked with systemd-analyze verify; shellcheck clean

Both runners already exist and are healthy — `thermograph-vps2` (`restart: always`, executing tasks) and the desktop's systemd `--user` unit (active, enabled, `Linger=yes`). Nothing needed creating. These are the two real defects found while verifying that. ### Desktop unit: `Restart=on-failure` -> `Restart=always` The same distinction that took Forgejo down for 27 hours on 2026-07-29, which `docker-stack.yml` already records for the `db` service: a daemon that exits **0** is not "finished successfully", it is one that stopped and must come back. `on-failure` cannot tell those apart. The failure mode is silent — unit `inactive (dead)`, runner offline in the UI, and every protected-branch merge blocked on a check nothing will ever produce. That is exactly the shape of the 2026-07-31 outage. ### `StartLimit*` were being ignored They sat in `[Service]`, where systemd accepts them without complaint and then ignores them. The live unit was running the **10s** default, not the intended 300s: ``` # before StartLimitIntervalUSec=10s # after (moved to [Unit]) StartLimitIntervalUSec=5min ``` ### `runner-vps2/README` sent `config.yaml` to a path the container cannot see The README said to copy `docker-compose.yml` and `config.yaml` together into `/opt/forgejo-runner/`. But the compose file mounts `./data:/data` and starts with `--config /data/config.yaml`, so `/data` **is** `data/` — a config beside the compose file is invisible. The daemon then runs on defaults with no `--add-host`, and every registry push fails in a way that looks exactly like a bad credential. vps2 had a stray copy at the documented path, so the instruction had been followed and worked around. Removed it; the README now sends the two files to their separate correct destinations and says why. ### Applied live - desktop: `Restart=always`, `StartLimitIntervalUSec=5min` — via `daemon-reload` only, with no job running, so nothing was interrupted - vps2: removed the misleading `/opt/forgejo-runner/config.yaml`; container still up - generated unit re-rendered and checked with `systemd-analyze verify`; `shellcheck` clean

emi added 1 commit 2026-08-01 18:51:05 +00:00

forgejo: make the desktop runner unit restart-always, fix the vps2 config path

PR build (required check) / changes (pull_request) Successful in 9s

Details

shell-lint / shellcheck (pull_request) Successful in 9s

Details

secrets-guard / encrypted (pull_request) Successful in 11s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

f25466ca76

Both runners exist and are up; these are the two defects found while verifying
that.

The desktop's systemd --user unit had Restart=on-failure. That is the same
distinction that took Forgejo down for 27 hours on 2026-07-29 (docker-stack.yml
records it for the db service): a daemon that exits 0 is not "finished
successfully", and on-failure cannot tell that from a clean shutdown. The
failure mode is silent — unit inactive (dead), runner offline in the UI, and
every protected-branch merge blocked on a check nothing will produce.

The StartLimit directives that bound that retry loop were in [Service], where
systemd accepts them without complaint and ignores them; the live unit was
running the 10s default rather than the intended 300s. Moved to [Unit], which
is where they are read.

runner-vps2/README told you to copy config.yaml next to docker-compose.yml, but
the compose file mounts ./data:/data and loads --config /data/config.yaml, so a
config there is invisible to the container — the daemon starts on defaults with
no --add-host, and every registry push then fails as if the credential were
wrong. vps2 had a stray copy at the documented path proving the instruction had
been followed.