From f25466ca76350d5c7fdbe68fa4ac5d9bc07c1dea Mon Sep 17 00:00:00 2001
From: Emi Griffith <me@emigriffith.dev>
Date: Sat, 1 Aug 2026 11:50:42 -0700
Subject: [PATCH] forgejo: make the desktop runner unit restart-always, fix the
 vps2 config path
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Both runners exist and are up; these are the two defects found while verifying
that.

The desktop's systemd --user unit had Restart=on-failure. That is the same
distinction that took Forgejo down for 27 hours on 2026-07-29 (docker-stack.yml
records it for the db service): a daemon that exits 0 is not "finished
successfully", and on-failure cannot tell that from a clean shutdown. The
failure mode is silent — unit inactive (dead), runner offline in the UI, and
every protected-branch merge blocked on a check nothing will produce.

The StartLimit directives that bound that retry loop were in [Service], where
systemd accepts them without complaint and ignores them; the live unit was
running the 10s default rather than the intended 300s. Moved to [Unit], which
is where they are read.

runner-vps2/README told you to copy config.yaml next to docker-compose.yml, but
the compose file mounts ./data:/data and loads --config /data/config.yaml, so a
config there is invisible to the container — the daemon starts on defaults with
no --add-host, and every registry push then fails as if the credential were
wrong. vps2 had a stray copy at the documented path proving the instruction had
been followed.
---
 infra/deploy/forgejo/register-lan-runner.sh | 18 +++++++++++++++++-
 infra/deploy/forgejo/runner-vps2/README.md  | 15 +++++++++++++--
 2 files changed, 30 insertions(+), 3 deletions(-)

diff --git a/infra/deploy/forgejo/register-lan-runner.sh b/infra/deploy/forgejo/register-lan-runner.sh
index 611155d..8d63550 100755
--- a/infra/deploy/forgejo/register-lan-runner.sh
+++ b/infra/deploy/forgejo/register-lan-runner.sh
@@ -93,6 +93,13 @@ cat > "$HOME/.config/systemd/user/forgejo-runner.service" <<EOF
 [Unit]
 Description=Forgejo Actions runner (docker + thermograph-lan)
 After=network-online.target
+# These bound the Restart=always loop below, so a genuinely broken config (bad
+# token, docker unreachable) ends as a stopped unit rather than spinning
+# forever. They belong in [Unit]: systemd accepts them in [Service] without
+# complaint but IGNORES them, leaving the 10s default — \`systemctl --user show
+# forgejo-runner -p StartLimitIntervalUSec\` is how you tell which one you got.
+StartLimitIntervalSec=300
+StartLimitBurst=5
 
 [Service]
 WorkingDirectory=${RUNNER_DIR}
@@ -102,7 +109,16 @@ WorkingDirectory=${RUNNER_DIR}
 # SOPS-configured) fails with "not installed" even though it plainly is.
 Environment=PATH=%h/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
 ExecStart=${RUNNER_DIR}/forgejo-runner daemon -c config.yaml
-Restart=on-failure
+# \`always\`, NOT \`on-failure\` — the same distinction that took Forgejo down for
+# 27 hours on 2026-07-29 (see docker-stack.yml's restart_policy comment): an
+# always-on daemon that exits **0** is not "finished successfully", it is a
+# daemon that stopped and must come back. \`on-failure\` cannot tell those apart,
+# and forgejo-runner exits 0 on several paths — a lost instance connection it
+# gives up on, or a SIGTERM from a Docker restart it interprets as a clean
+# shutdown. The failure mode is silent: the unit sits \`inactive (dead)\`, the UI
+# shows the runner offline, and every protected-branch merge blocks on a check
+# that will never be produced.
+Restart=always
 RestartSec=5
 
 [Install]
diff --git a/infra/deploy/forgejo/runner-vps2/README.md b/infra/deploy/forgejo/runner-vps2/README.md
index 2b1204c..fa38ec5 100644
--- a/infra/deploy/forgejo/runner-vps2/README.md
+++ b/infra/deploy/forgejo/runner-vps2/README.md
@@ -49,13 +49,24 @@ deploys.
 
 So this directory is the source of truth; vps2 gets a copy:
 
+The two files do **not** go to the same place, and getting it wrong fails in a
+confusing way. `docker-compose.yml` mounts `./data:/data` and starts the daemon
+with `--config /data/config.yaml`, so `/data` *is* `data/` — a `config.yaml`
+sitting beside the compose file is invisible to the container. The daemon then
+either refuses to start or silently runs on defaults (no `--add-host`, so every
+registry push fails as if the credential were wrong).
+
 ```bash
-install -d -m 0755 /opt/forgejo-runner
+install -d -m 0755 /opt/forgejo-runner/data
 cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/docker-compose.yml \
-   /opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
    /opt/forgejo-runner/
+cp /opt/thermograph/infra/deploy/forgejo/runner-vps2/config.yaml \
+   /opt/forgejo-runner/data/
 ```
 
+`data/` is also where `register` writes `.runner`, so the runner's whole state —
+config plus registration — lives in the one directory the compose file mounts.
+
 Re-copy after changing either file here, then `docker compose up -d`. There is
 no automation for that step and there should not be: a workflow that redeploys
 the runner runs *on* the runner.