Add agent VPS access, a 2-node Docker Swarm, and Forgejo CI/CD (#234)
Three additive infrastructure layers on top of the two VPS boxes Terraform
already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old
VPS, 75.119.132.91). None of this touches backend/, Dockerfile,
docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the
app-containerization work in flight elsewhere; this is strictly the layer on
top. See INFRA.md for the full runbook and order of operations.
- deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login
(not raw root) for agent-driven ops — passwordless sudo under a distinct
username, sshd hardened to key-only auth, auditd logging every
root-effective command. One line to revoke.
- deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a
WireGuard tunnel rather than trusting the public internet with the
overlay data plane, which Docker's own guidance says should never face it
directly. Swarm ports locked to the tunnel interface once joined. This
cluster's only workload is Forgejo — it does not orchestrate the
Terraform-managed app deploys, so nothing here can strand the app's
single-writer database.
- deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a
Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus
Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom
auto-merge workflow step is dropped — it existed only to work around
GitHub's paywalled branch protection on private free-tier repos, which
Forgejo has no such tier for; native "auto merge when checks succeed"
replaces it, and as a real git push (unlike GitHub's non-triggering
token-merge) it fires the LAN deploy naturally with no double-trigger
logic needed. appleboy/ssh-action is referenced by full URL (not mirrored
on Forgejo's default action registry); actions/checkout and
actions/setup-python resolve unchanged.
Migration is mirror-first: the GitHub repo import and workflow files land
here, but cutting deploy secrets over and retiring GitHub happens only after
verification (INFRA.md 3d) — GitHub stays live as a fallback throughout.
One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but
terraform.tfvars.example names prod's deploy branch `release`. Left as a
faithful mirror rather than guessed at — reconcile with whoever's driving
Terraform/deploy.
2026-07-21 00:36:39 +00:00
|
|
|
#!/usr/bin/env bash
|
|
|
|
|
# Re-points the existing LAN dev self-hosted runner from GitHub Actions to
|
|
|
|
|
# Forgejo Actions. Run on the SAME machine that already runs the GitHub
|
|
|
|
|
# runner (see DEPLOY-DEV.md) — this replaces that runner, it doesn't add a
|
|
|
|
|
# second one. Sudo-free, systemd --user, same pattern as the app service.
|
|
|
|
|
#
|
2026-07-26 06:56:38 +00:00
|
|
|
# It registers with BOTH labels the workflows historically needed: general
|
|
|
|
|
# CI/build/deploy jobs (`docker`, containerized via this machine's own
|
|
|
|
|
# already-installed Docker — no Docker-in-Docker sidecar needed, since a real
|
|
|
|
|
# host with a real Docker install needs no such indirection) and the LAN-deploy
|
|
|
|
|
# job (`thermograph-lan`, bare/host-native — it wrote to ~/thermograph-dev and
|
|
|
|
|
# restarted a systemd --user service, which only worked running directly on the
|
Realign Swarm/Forgejo infra to the 3-node design (#243)
Bring the Swarm+Forgejo layer in line with the canonical topology in
docs/runbooks/implementation-handoff.md: three nodes (prod, beta, and
the desktop LAN dev machine) instead of two, with the Forgejo Actions
runner registered on the desktop as a plain systemd service rather than
a Swarm-scheduled Docker-in-Docker sidecar.
- setup-wireguard.sh: full N-peer mesh instead of point-to-point
- docker-stack.yml: drop the runner/runner-dind services, volumes, and
runner-token secret; only forgejo_db_password remains
- register-lan-runner.sh: register under both docker and
thermograph-lan labels, since one runner now covers both job types
- init-swarm.sh, join-swarm.sh, swarm/README.md, forgejo/README.md,
INFRA.md: updated node lists, order of operations, and access-state
table for three nodes
- deploy.yml: renamed to "Deploy to beta VPS" and reconciled the
main-vs-release branch question against terraform.tfvars.example
(this workflow already targets beta; a release-triggered prod
deploy doesn't exist yet and isn't invented here)
2026-07-21 01:43:19 +00:00
|
|
|
# host, not inside a container).
|
|
|
|
|
#
|
2026-07-26 06:56:38 +00:00
|
|
|
# `thermograph-lan` IS NOW OBSOLETE. Dev moved off this machine to vps1 and is
|
|
|
|
|
# deployed over SSH by the same `Deploy` workflow that ships beta and prod;
|
|
|
|
|
# there is no host-native LAN deploy job left for that label to serve. Nothing
|
|
|
|
|
# breaks by keeping it registered — no workflow requests it — but do not build
|
|
|
|
|
# anything new on it.
|
|
|
|
|
#
|
ci: add an always-on Actions runner on vps2
The estate had exactly one registered runner, on the desktop. It went offline
at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required
check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both
application databases and for Forgejo -- did not fire. Forgejo queued that
scheduled run rather than dropping it, so it completed on reconnect and nothing
was lost. A longer outage would have meant real gaps.
Three files claimed an "always-on Swarm-hosted runner" existed and that the
estate therefore no longer depended on the desktop. It did not exist: an early
revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was
removed. That claim is why a single point of failure sat unnoticed. Corrected
in docker-stack.yml, forgejo/README.md and register-lan-runner.sh.
The new runner is a plain restart:always container, not a Swarm service: a
Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would
be gone exactly when the cluster is what is broken. It runs from
/opt/forgejo-runner rather than in place, because the checkout is reset on
every prod deploy and one `git clean -fdx` there would destroy the
registration.
vps2 runs prod, so the socket mount is bounded rather than assumed benign:
capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no
job can bind-mount /etc/thermograph.env, and no thermograph network joined.
This defends against accident, not against a hostile workflow author -- stated
plainly in the compose header rather than implied.
The desktop runner stays registered as extra capacity. Nothing may assume it
is up.
2026-08-01 14:54:28 +00:00
|
|
|
# This machine keeps serving the `docker` label as EXTRA capacity, and that is
|
|
|
|
|
# now a settled call rather than an open one. The claim it used to make here —
|
|
|
|
|
# that a Swarm-hosted runner was always-on, so the estate no longer depended on
|
|
|
|
|
# this box — was false: no such runner existed. This was the only registered
|
|
|
|
|
# runner in the estate until 2026-08-01, and when it went offline on 2026-07-31
|
|
|
|
|
# every merge, deploy and backup stopped for 21 hours.
|
|
|
|
|
#
|
|
|
|
|
# The runner the estate depends on is deploy/forgejo/runner-vps2/. Keeping this
|
|
|
|
|
# one is worthwhile (a second runner means a queue drains instead of stalling),
|
|
|
|
|
# but nothing may be built on the assumption that this box is up.
|
2026-07-26 06:56:38 +00:00
|
|
|
#
|
Add agent VPS access, a 2-node Docker Swarm, and Forgejo CI/CD (#234)
Three additive infrastructure layers on top of the two VPS boxes Terraform
already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old
VPS, 75.119.132.91). None of this touches backend/, Dockerfile,
docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the
app-containerization work in flight elsewhere; this is strictly the layer on
top. See INFRA.md for the full runbook and order of operations.
- deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login
(not raw root) for agent-driven ops — passwordless sudo under a distinct
username, sshd hardened to key-only auth, auditd logging every
root-effective command. One line to revoke.
- deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a
WireGuard tunnel rather than trusting the public internet with the
overlay data plane, which Docker's own guidance says should never face it
directly. Swarm ports locked to the tunnel interface once joined. This
cluster's only workload is Forgejo — it does not orchestrate the
Terraform-managed app deploys, so nothing here can strand the app's
single-writer database.
- deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a
Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus
Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom
auto-merge workflow step is dropped — it existed only to work around
GitHub's paywalled branch protection on private free-tier repos, which
Forgejo has no such tier for; native "auto merge when checks succeed"
replaces it, and as a real git push (unlike GitHub's non-triggering
token-merge) it fires the LAN deploy naturally with no double-trigger
logic needed. appleboy/ssh-action is referenced by full URL (not mirrored
on Forgejo's default action registry); actions/checkout and
actions/setup-python resolve unchanged.
Migration is mirror-first: the GitHub repo import and workflow files land
here, but cutting deploy secrets over and retiring GitHub happens only after
verification (INFRA.md 3d) — GitHub stays live as a fallback throughout.
One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but
terraform.tfvars.example names prod's deploy branch `release`. Left as a
faithful mirror rather than guessed at — reconcile with whoever's driving
Terraform/deploy.
2026-07-21 00:36:39 +00:00
|
|
|
# bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
|
|
|
|
|
#
|
|
|
|
|
# Get <registration_token> from the Forgejo web UI:
|
|
|
|
|
# repo -> Settings -> Actions -> Runners -> Create new Runner
|
|
|
|
|
# (or an org/instance-level runner page, if you want it to serve more than
|
|
|
|
|
# this one repo — same as the GitHub runner did).
|
|
|
|
|
set -euo pipefail
|
|
|
|
|
|
|
|
|
|
FORGEJO_URL="${1:?usage: $0 <forgejo_url> <registration_token>}"
|
|
|
|
|
TOKEN="${2:?}"
|
|
|
|
|
RUNNER_DIR="${RUNNER_DIR:-$HOME/forgejo-runner}"
|
2026-07-31 03:53:26 +00:00
|
|
|
LABELS="${LABELS:-docker:docker://git.thermograph.org/jinemi/thermograph/ci-runner:v2,thermograph-lan}"
|
Add agent VPS access, a 2-node Docker Swarm, and Forgejo CI/CD (#234)
Three additive infrastructure layers on top of the two VPS boxes Terraform
already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old
VPS, 75.119.132.91). None of this touches backend/, Dockerfile,
docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the
app-containerization work in flight elsewhere; this is strictly the layer on
top. See INFRA.md for the full runbook and order of operations.
- deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login
(not raw root) for agent-driven ops — passwordless sudo under a distinct
username, sshd hardened to key-only auth, auditd logging every
root-effective command. One line to revoke.
- deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a
WireGuard tunnel rather than trusting the public internet with the
overlay data plane, which Docker's own guidance says should never face it
directly. Swarm ports locked to the tunnel interface once joined. This
cluster's only workload is Forgejo — it does not orchestrate the
Terraform-managed app deploys, so nothing here can strand the app's
single-writer database.
- deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a
Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus
Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom
auto-merge workflow step is dropped — it existed only to work around
GitHub's paywalled branch protection on private free-tier repos, which
Forgejo has no such tier for; native "auto merge when checks succeed"
replaces it, and as a real git push (unlike GitHub's non-triggering
token-merge) it fires the LAN deploy naturally with no double-trigger
logic needed. appleboy/ssh-action is referenced by full URL (not mirrored
on Forgejo's default action registry); actions/checkout and
actions/setup-python resolve unchanged.
Migration is mirror-first: the GitHub repo import and workflow files land
here, but cutting deploy secrets over and retiring GitHub happens only after
verification (INFRA.md 3d) — GitHub stays live as a fallback throughout.
One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but
terraform.tfvars.example names prod's deploy branch `release`. Left as a
faithful mirror rather than guessed at — reconcile with whoever's driving
Terraform/deploy.
2026-07-21 00:36:39 +00:00
|
|
|
|
|
|
|
|
echo "==> Stopping and disabling the old GitHub Actions runner service, if present"
|
|
|
|
|
systemctl --user stop github-actions-runner 2>/dev/null || true
|
|
|
|
|
systemctl --user disable github-actions-runner 2>/dev/null || true
|
|
|
|
|
|
Realign Swarm/Forgejo infra to the 3-node design (#243)
Bring the Swarm+Forgejo layer in line with the canonical topology in
docs/runbooks/implementation-handoff.md: three nodes (prod, beta, and
the desktop LAN dev machine) instead of two, with the Forgejo Actions
runner registered on the desktop as a plain systemd service rather than
a Swarm-scheduled Docker-in-Docker sidecar.
- setup-wireguard.sh: full N-peer mesh instead of point-to-point
- docker-stack.yml: drop the runner/runner-dind services, volumes, and
runner-token secret; only forgejo_db_password remains
- register-lan-runner.sh: register under both docker and
thermograph-lan labels, since one runner now covers both job types
- init-swarm.sh, join-swarm.sh, swarm/README.md, forgejo/README.md,
INFRA.md: updated node lists, order of operations, and access-state
table for three nodes
- deploy.yml: renamed to "Deploy to beta VPS" and reconciled the
main-vs-release branch question against terraform.tfvars.example
(this workflow already targets beta; a release-triggered prod
deploy doesn't exist yet and isn't invented here)
2026-07-21 01:43:19 +00:00
|
|
|
if ! id -nG "$USER" 2>/dev/null | grep -qw docker; then
|
|
|
|
|
echo "WARNING: $USER is not in the 'docker' group — the 'docker:docker://...'"
|
|
|
|
|
echo "labeled jobs (general CI) will fail to start a container until this is"
|
|
|
|
|
echo "fixed: sudo usermod -aG docker $USER && (log out and back in)."
|
|
|
|
|
fi
|
|
|
|
|
|
Add agent VPS access, a 2-node Docker Swarm, and Forgejo CI/CD (#234)
Three additive infrastructure layers on top of the two VPS boxes Terraform
already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old
VPS, 75.119.132.91). None of this touches backend/, Dockerfile,
docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the
app-containerization work in flight elsewhere; this is strictly the layer on
top. See INFRA.md for the full runbook and order of operations.
- deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login
(not raw root) for agent-driven ops — passwordless sudo under a distinct
username, sshd hardened to key-only auth, auditd logging every
root-effective command. One line to revoke.
- deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a
WireGuard tunnel rather than trusting the public internet with the
overlay data plane, which Docker's own guidance says should never face it
directly. Swarm ports locked to the tunnel interface once joined. This
cluster's only workload is Forgejo — it does not orchestrate the
Terraform-managed app deploys, so nothing here can strand the app's
single-writer database.
- deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a
Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus
Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom
auto-merge workflow step is dropped — it existed only to work around
GitHub's paywalled branch protection on private free-tier repos, which
Forgejo has no such tier for; native "auto merge when checks succeed"
replaces it, and as a real git push (unlike GitHub's non-triggering
token-merge) it fires the LAN deploy naturally with no double-trigger
logic needed. appleboy/ssh-action is referenced by full URL (not mirrored
on Forgejo's default action registry); actions/checkout and
actions/setup-python resolve unchanged.
Migration is mirror-first: the GitHub repo import and workflow files land
here, but cutting deploy secrets over and retiring GitHub happens only after
verification (INFRA.md 3d) — GitHub stays live as a fallback throughout.
One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but
terraform.tfvars.example names prod's deploy branch `release`. Left as a
faithful mirror rather than guessed at — reconcile with whoever's driving
Terraform/deploy.
2026-07-21 00:36:39 +00:00
|
|
|
echo "==> Installing forgejo-runner into $RUNNER_DIR"
|
|
|
|
|
mkdir -p "$RUNNER_DIR"
|
|
|
|
|
cd "$RUNNER_DIR"
|
|
|
|
|
if [ ! -x ./forgejo-runner ]; then
|
|
|
|
|
ARCH="$(uname -m)"
|
|
|
|
|
case "$ARCH" in
|
|
|
|
|
x86_64) BIN_ARCH=amd64 ;;
|
|
|
|
|
aarch64) BIN_ARCH=arm64 ;;
|
|
|
|
|
*) echo "Unsupported arch: $ARCH — download the right binary by hand from" \
|
|
|
|
|
"https://code.forgejo.org/forgejo/runner/releases" >&2; exit 1 ;;
|
|
|
|
|
esac
|
|
|
|
|
VER="${FORGEJO_RUNNER_VERSION:-6.3.1}"
|
|
|
|
|
curl -fsSL -o forgejo-runner \
|
|
|
|
|
"https://code.forgejo.org/forgejo/runner/releases/download/v${VER}/forgejo-runner-${VER}-linux-${BIN_ARCH}"
|
|
|
|
|
chmod +x forgejo-runner
|
|
|
|
|
fi
|
|
|
|
|
|
|
|
|
|
echo "==> Registering with $FORGEJO_URL (label: $LABELS)"
|
|
|
|
|
./forgejo-runner register --no-interactive \
|
|
|
|
|
--instance "$FORGEJO_URL" \
|
|
|
|
|
--token "$TOKEN" \
|
|
|
|
|
--name "thermograph-lan-$(hostname -s)" \
|
|
|
|
|
--labels "$LABELS"
|
|
|
|
|
|
2026-07-25 04:13:47 +00:00
|
|
|
echo "==> Runner config (docker.sock automount so docker:-labeled jobs like"
|
|
|
|
|
echo " build-push.yml can actually run 'docker build/push'; capacity raised"
|
|
|
|
|
echo " from the default of 1 -- a single PR push fires 3+ independent"
|
|
|
|
|
echo " workflows (pr-build, secrets-guard, shell-lint) simultaneously, so"
|
|
|
|
|
echo " anything less than that serializes jobs that could run in parallel)"
|
|
|
|
|
CAPACITY="${CAPACITY:-8}"
|
2026-07-21 06:46:04 +00:00
|
|
|
./forgejo-runner generate-config \
|
2026-07-25 04:13:47 +00:00
|
|
|
| sed -e 's/docker_host: "-"/docker_host: "automount"/' \
|
|
|
|
|
-e "s/capacity: 1/capacity: ${CAPACITY}/" \
|
2026-07-21 06:46:04 +00:00
|
|
|
> "${RUNNER_DIR}/config.yaml"
|
|
|
|
|
|
Add agent VPS access, a 2-node Docker Swarm, and Forgejo CI/CD (#234)
Three additive infrastructure layers on top of the two VPS boxes Terraform
already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old
VPS, 75.119.132.91). None of this touches backend/, Dockerfile,
docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the
app-containerization work in flight elsewhere; this is strictly the layer on
top. See INFRA.md for the full runbook and order of operations.
- deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login
(not raw root) for agent-driven ops — passwordless sudo under a distinct
username, sshd hardened to key-only auth, auditd logging every
root-effective command. One line to revoke.
- deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a
WireGuard tunnel rather than trusting the public internet with the
overlay data plane, which Docker's own guidance says should never face it
directly. Swarm ports locked to the tunnel interface once joined. This
cluster's only workload is Forgejo — it does not orchestrate the
Terraform-managed app deploys, so nothing here can strand the app's
single-writer database.
- deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a
Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus
Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom
auto-merge workflow step is dropped — it existed only to work around
GitHub's paywalled branch protection on private free-tier repos, which
Forgejo has no such tier for; native "auto merge when checks succeed"
replaces it, and as a real git push (unlike GitHub's non-triggering
token-merge) it fires the LAN deploy naturally with no double-trigger
logic needed. appleboy/ssh-action is referenced by full URL (not mirrored
on Forgejo's default action registry); actions/checkout and
actions/setup-python resolve unchanged.
Migration is mirror-first: the GitHub repo import and workflow files land
here, but cutting deploy secrets over and retiring GitHub happens only after
verification (INFRA.md 3d) — GitHub stays live as a fallback throughout.
One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but
terraform.tfvars.example names prod's deploy branch `release`. Left as a
faithful mirror rather than guessed at — reconcile with whoever's driving
Terraform/deploy.
2026-07-21 00:36:39 +00:00
|
|
|
echo "==> systemd --user unit"
|
|
|
|
|
mkdir -p "$HOME/.config/systemd/user"
|
|
|
|
|
cat > "$HOME/.config/systemd/user/forgejo-runner.service" <<EOF
|
|
|
|
|
[Unit]
|
Realign Swarm/Forgejo infra to the 3-node design (#243)
Bring the Swarm+Forgejo layer in line with the canonical topology in
docs/runbooks/implementation-handoff.md: three nodes (prod, beta, and
the desktop LAN dev machine) instead of two, with the Forgejo Actions
runner registered on the desktop as a plain systemd service rather than
a Swarm-scheduled Docker-in-Docker sidecar.
- setup-wireguard.sh: full N-peer mesh instead of point-to-point
- docker-stack.yml: drop the runner/runner-dind services, volumes, and
runner-token secret; only forgejo_db_password remains
- register-lan-runner.sh: register under both docker and
thermograph-lan labels, since one runner now covers both job types
- init-swarm.sh, join-swarm.sh, swarm/README.md, forgejo/README.md,
INFRA.md: updated node lists, order of operations, and access-state
table for three nodes
- deploy.yml: renamed to "Deploy to beta VPS" and reconciled the
main-vs-release branch question against terraform.tfvars.example
(this workflow already targets beta; a release-triggered prod
deploy doesn't exist yet and isn't invented here)
2026-07-21 01:43:19 +00:00
|
|
|
Description=Forgejo Actions runner (docker + thermograph-lan)
|
Add agent VPS access, a 2-node Docker Swarm, and Forgejo CI/CD (#234)
Three additive infrastructure layers on top of the two VPS boxes Terraform
already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old
VPS, 75.119.132.91). None of this touches backend/, Dockerfile,
docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the
app-containerization work in flight elsewhere; this is strictly the layer on
top. See INFRA.md for the full runbook and order of operations.
- deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login
(not raw root) for agent-driven ops — passwordless sudo under a distinct
username, sshd hardened to key-only auth, auditd logging every
root-effective command. One line to revoke.
- deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a
WireGuard tunnel rather than trusting the public internet with the
overlay data plane, which Docker's own guidance says should never face it
directly. Swarm ports locked to the tunnel interface once joined. This
cluster's only workload is Forgejo — it does not orchestrate the
Terraform-managed app deploys, so nothing here can strand the app's
single-writer database.
- deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a
Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus
Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom
auto-merge workflow step is dropped — it existed only to work around
GitHub's paywalled branch protection on private free-tier repos, which
Forgejo has no such tier for; native "auto merge when checks succeed"
replaces it, and as a real git push (unlike GitHub's non-triggering
token-merge) it fires the LAN deploy naturally with no double-trigger
logic needed. appleboy/ssh-action is referenced by full URL (not mirrored
on Forgejo's default action registry); actions/checkout and
actions/setup-python resolve unchanged.
Migration is mirror-first: the GitHub repo import and workflow files land
here, but cutting deploy secrets over and retiring GitHub happens only after
verification (INFRA.md 3d) — GitHub stays live as a fallback throughout.
One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but
terraform.tfvars.example names prod's deploy branch `release`. Left as a
faithful mirror rather than guessed at — reconcile with whoever's driving
Terraform/deploy.
2026-07-21 00:36:39 +00:00
|
|
|
After=network-online.target
|
2026-08-01 18:50:42 +00:00
|
|
|
# These bound the Restart=always loop below, so a genuinely broken config (bad
|
|
|
|
|
# token, docker unreachable) ends as a stopped unit rather than spinning
|
|
|
|
|
# forever. They belong in [Unit]: systemd accepts them in [Service] without
|
|
|
|
|
# complaint but IGNORES them, leaving the 10s default — \`systemctl --user show
|
|
|
|
|
# forgejo-runner -p StartLimitIntervalUSec\` is how you tell which one you got.
|
|
|
|
|
StartLimitIntervalSec=300
|
|
|
|
|
StartLimitBurst=5
|
Add agent VPS access, a 2-node Docker Swarm, and Forgejo CI/CD (#234)
Three additive infrastructure layers on top of the two VPS boxes Terraform
already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old
VPS, 75.119.132.91). None of this touches backend/, Dockerfile,
docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the
app-containerization work in flight elsewhere; this is strictly the layer on
top. See INFRA.md for the full runbook and order of operations.
- deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login
(not raw root) for agent-driven ops — passwordless sudo under a distinct
username, sshd hardened to key-only auth, auditd logging every
root-effective command. One line to revoke.
- deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a
WireGuard tunnel rather than trusting the public internet with the
overlay data plane, which Docker's own guidance says should never face it
directly. Swarm ports locked to the tunnel interface once joined. This
cluster's only workload is Forgejo — it does not orchestrate the
Terraform-managed app deploys, so nothing here can strand the app's
single-writer database.
- deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a
Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus
Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom
auto-merge workflow step is dropped — it existed only to work around
GitHub's paywalled branch protection on private free-tier repos, which
Forgejo has no such tier for; native "auto merge when checks succeed"
replaces it, and as a real git push (unlike GitHub's non-triggering
token-merge) it fires the LAN deploy naturally with no double-trigger
logic needed. appleboy/ssh-action is referenced by full URL (not mirrored
on Forgejo's default action registry); actions/checkout and
actions/setup-python resolve unchanged.
Migration is mirror-first: the GitHub repo import and workflow files land
here, but cutting deploy secrets over and retiring GitHub happens only after
verification (INFRA.md 3d) — GitHub stays live as a fallback throughout.
One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but
terraform.tfvars.example names prod's deploy branch `release`. Left as a
faithful mirror rather than guessed at — reconcile with whoever's driving
Terraform/deploy.
2026-07-21 00:36:39 +00:00
|
|
|
|
|
|
|
|
[Service]
|
|
|
|
|
WorkingDirectory=${RUNNER_DIR}
|
2026-07-25 06:59:21 +00:00
|
|
|
# CI job steps shell out to user-installed tools (sops, age, ...) that live in
|
|
|
|
|
# ~/.local/bin, not on systemd --user's default PATH. Without this, any
|
|
|
|
|
# workflow that needs one (e.g. render-secrets.sh once a host is
|
|
|
|
|
# SOPS-configured) fails with "not installed" even though it plainly is.
|
|
|
|
|
Environment=PATH=%h/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
|
2026-07-21 06:46:04 +00:00
|
|
|
ExecStart=${RUNNER_DIR}/forgejo-runner daemon -c config.yaml
|
2026-08-01 18:50:42 +00:00
|
|
|
# \`always\`, NOT \`on-failure\` — the same distinction that took Forgejo down for
|
|
|
|
|
# 27 hours on 2026-07-29 (see docker-stack.yml's restart_policy comment): an
|
|
|
|
|
# always-on daemon that exits **0** is not "finished successfully", it is a
|
|
|
|
|
# daemon that stopped and must come back. \`on-failure\` cannot tell those apart,
|
|
|
|
|
# and forgejo-runner exits 0 on several paths — a lost instance connection it
|
|
|
|
|
# gives up on, or a SIGTERM from a Docker restart it interprets as a clean
|
|
|
|
|
# shutdown. The failure mode is silent: the unit sits \`inactive (dead)\`, the UI
|
|
|
|
|
# shows the runner offline, and every protected-branch merge blocks on a check
|
|
|
|
|
# that will never be produced.
|
|
|
|
|
Restart=always
|
Add agent VPS access, a 2-node Docker Swarm, and Forgejo CI/CD (#234)
Three additive infrastructure layers on top of the two VPS boxes Terraform
already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old
VPS, 75.119.132.91). None of this touches backend/, Dockerfile,
docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the
app-containerization work in flight elsewhere; this is strictly the layer on
top. See INFRA.md for the full runbook and order of operations.
- deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login
(not raw root) for agent-driven ops — passwordless sudo under a distinct
username, sshd hardened to key-only auth, auditd logging every
root-effective command. One line to revoke.
- deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a
WireGuard tunnel rather than trusting the public internet with the
overlay data plane, which Docker's own guidance says should never face it
directly. Swarm ports locked to the tunnel interface once joined. This
cluster's only workload is Forgejo — it does not orchestrate the
Terraform-managed app deploys, so nothing here can strand the app's
single-writer database.
- deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a
Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus
Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom
auto-merge workflow step is dropped — it existed only to work around
GitHub's paywalled branch protection on private free-tier repos, which
Forgejo has no such tier for; native "auto merge when checks succeed"
replaces it, and as a real git push (unlike GitHub's non-triggering
token-merge) it fires the LAN deploy naturally with no double-trigger
logic needed. appleboy/ssh-action is referenced by full URL (not mirrored
on Forgejo's default action registry); actions/checkout and
actions/setup-python resolve unchanged.
Migration is mirror-first: the GitHub repo import and workflow files land
here, but cutting deploy secrets over and retiring GitHub happens only after
verification (INFRA.md 3d) — GitHub stays live as a fallback throughout.
One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but
terraform.tfvars.example names prod's deploy branch `release`. Left as a
faithful mirror rather than guessed at — reconcile with whoever's driving
Terraform/deploy.
2026-07-21 00:36:39 +00:00
|
|
|
RestartSec=5
|
|
|
|
|
|
|
|
|
|
[Install]
|
|
|
|
|
WantedBy=default.target
|
|
|
|
|
EOF
|
|
|
|
|
|
|
|
|
|
systemctl --user daemon-reload
|
|
|
|
|
systemctl --user enable --now forgejo-runner
|
|
|
|
|
loginctl enable-linger "$USER" 2>/dev/null || true
|
|
|
|
|
|
|
|
|
|
cat <<EOF
|
|
|
|
|
|
Realign Swarm/Forgejo infra to the 3-node design (#243)
Bring the Swarm+Forgejo layer in line with the canonical topology in
docs/runbooks/implementation-handoff.md: three nodes (prod, beta, and
the desktop LAN dev machine) instead of two, with the Forgejo Actions
runner registered on the desktop as a plain systemd service rather than
a Swarm-scheduled Docker-in-Docker sidecar.
- setup-wireguard.sh: full N-peer mesh instead of point-to-point
- docker-stack.yml: drop the runner/runner-dind services, volumes, and
runner-token secret; only forgejo_db_password remains
- register-lan-runner.sh: register under both docker and
thermograph-lan labels, since one runner now covers both job types
- init-swarm.sh, join-swarm.sh, swarm/README.md, forgejo/README.md,
INFRA.md: updated node lists, order of operations, and access-state
table for three nodes
- deploy.yml: renamed to "Deploy to beta VPS" and reconciled the
main-vs-release branch question against terraform.tfvars.example
(this workflow already targets beta; a release-triggered prod
deploy doesn't exist yet and isn't invented here)
2026-07-21 01:43:19 +00:00
|
|
|
Done. The runner now serves Forgejo, not GitHub — one runner, both labels
|
|
|
|
|
($LABELS), replacing what used to be a separate Swarm-hosted runner for
|
|
|
|
|
general CI plus this machine's own GitHub runner for LAN deploys.
|
Add agent VPS access, a 2-node Docker Swarm, and Forgejo CI/CD (#234)
Three additive infrastructure layers on top of the two VPS boxes Terraform
already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old
VPS, 75.119.132.91). None of this touches backend/, Dockerfile,
docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the
app-containerization work in flight elsewhere; this is strictly the layer on
top. See INFRA.md for the full runbook and order of operations.
- deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login
(not raw root) for agent-driven ops — passwordless sudo under a distinct
username, sshd hardened to key-only auth, auditd logging every
root-effective command. One line to revoke.
- deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a
WireGuard tunnel rather than trusting the public internet with the
overlay data plane, which Docker's own guidance says should never face it
directly. Swarm ports locked to the tunnel interface once joined. This
cluster's only workload is Forgejo — it does not orchestrate the
Terraform-managed app deploys, so nothing here can strand the app's
single-writer database.
- deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a
Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus
Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom
auto-merge workflow step is dropped — it existed only to work around
GitHub's paywalled branch protection on private free-tier repos, which
Forgejo has no such tier for; native "auto merge when checks succeed"
replaces it, and as a real git push (unlike GitHub's non-triggering
token-merge) it fires the LAN deploy naturally with no double-trigger
logic needed. appleboy/ssh-action is referenced by full URL (not mirrored
on Forgejo's default action registry); actions/checkout and
actions/setup-python resolve unchanged.
Migration is mirror-first: the GitHub repo import and workflow files land
here, but cutting deploy secrets over and retiring GitHub happens only after
verification (INFRA.md 3d) — GitHub stays live as a fallback throughout.
One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but
terraform.tfvars.example names prod's deploy branch `release`. Left as a
faithful mirror rather than guessed at — reconcile with whoever's driving
Terraform/deploy.
2026-07-21 00:36:39 +00:00
|
|
|
status: systemctl --user status forgejo-runner
|
|
|
|
|
logs: journalctl --user -u forgejo-runner -f
|
|
|
|
|
restart: systemctl --user restart forgejo-runner
|
|
|
|
|
|
|
|
|
|
The old github-actions-runner unit was stopped and disabled but not deleted —
|
|
|
|
|
remove ~/actions-runner by hand once you've confirmed Forgejo deploys work.
|
|
|
|
|
EOF
|