thermograph/infra/deploy/forgejo/README.md

Forgejo on the Swarm cluster

Runs as deploy/forgejo/docker-stack.yml — the only Swarm-scheduled workload this cluster carries (prod and beta's app stacks are separate docker stack deploys that happen to run on the manager node, vps2 — see DEPLOY.md). Pinned to the vps1 node (75.119.132.91) via the role=forge label from deploy/swarm/label-forge-node.sh.

The Actions runner is deliberately not part of this stack: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be unavailable exactly when the cluster is what needs repairing. An earlier revision did run it as a Swarm-scheduled Docker-in-Docker sidecar pinned to the Forgejo node; that is gone.

Runners live in two places, and only one of them counts:

• runner-vps2/ — the always-on runner, a plain restart: always container on vps2. This is the one CI depends on.
• register-lan-runner.sh — the desktop's systemd runner. Extra capacity. Nothing may assume it is up. It was the only registered runner in the estate until 2026-08-01, and its 21-hour outage on 2026-07-31 froze every merge and deploy and stopped the nightly backup from firing.

Prerequisites

1. All three nodes have joined the swarm (deploy/swarm/) and vps1 is labeled role=forge.
2. docker node ls (from the manager, vps2) shows all three Ready.

One-time setup: Swarm secret

One secret the stack expects to already exist (a Swarm secret, not a file — external: true in the stack file, so docker stack deploy never creates or sees the value, only references it):

# A strong random password for Forgejo's own Postgres (NOT related to
# Thermograph's app database — entirely separate instance/network).
openssl rand -base64 32 | docker secret create forgejo_db_password -

Deploy / update

docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo

Re-running is safe — Swarm only touches services whose spec actually changed. Do this before the DNS + Caddy step below — Caddy's reverse_proxy target (127.0.0.1:3080) needs the forgejo service actually listening first, or its first health check just fails harmlessly until it is.

This stack has no auto-deploy trigger — nothing in .forgejo/workflows/ redeploys it on push. A change to docker-stack.yml only takes effect once someone re-runs docker stack deploy by hand on the manager (vps2).

db/forgejo both carry resources.limits (defaults: db 1 CPU/1g, forgejo 2 CPU/2g — several times observed steady-state usage), overridable with FORGEJO_DB_CPUS/FORGEJO_DB_MEMORY/FORGEJO_CPUS/FORGEJO_MEMORY env vars before docker stack deploy, same convention as the app stack.

DNS + TLS: reusing vps1's existing Caddy, not a second reverse proxy

Forgejo is pinned to vps1 (role=forge) — the same box that also runs Grafana/Loki/Alloy and the emigriffith.dev portfolio site, each fronted by that host's own Caddy. A second ingress (Traefik) trying to bind the same ports would collide with it. So there's no Traefik in this stack: forgejo's web port publishes to 127.0.0.1:3080 only (host-local), and vps1's existing Caddy gets one more site block reverse-proxying to it — same pattern as its other site blocks, same automatic-HTTPS.

1. Point the Forgejo domain (default dev.jinemi.com; override with FORGEJO_DOMAIN=... before docker stack deploy) at vps1's public IP — that's where the task actually runs, not vps2's or the desktop's.
2. Append deploy/forgejo/caddy-git.conf to vps1's /etc/caddy/Caddyfile, adjusting the domain if you didn't use the default, then systemctl reload caddy.
3. That file also resolves the registry-exposure hazard (#15 in thermograph-docs/runbooks/hop1-forgejo-registry-cutover.md): /v2/* (the registry API) is blocked to everything except the WireGuard mesh CIDR (10.10.0.0/24); the git/web UI stays public. CI runners and Swarm nodes reach the registry over the mesh, not the public internet — see "Registry access from mesh clients" below.

Migrating the domain (git.thermograph.org -> dev.jinemi.com)

Web UI and OAuth: done. dev.jinemi.com is Forgejo's ROOT_URL (FORGEJO_DOMAIN in docker-stack.yml), so it is the name Forgejo generates in clone URLs, the Google OAuth callback, webhook payload URLs and mail links.

Registry: done in this repo, and it needs host-side steps to match. Every image name, REGISTRY_HOST default, runner label and --add-host pin under version control now says dev.jinemi.com. None of that is self-applying — see "Host-side steps" below before expecting a build to push.

Both names address the same Forgejo and therefore the same packages. A tag pushed as git.thermograph.org/jinemi/thermograph/backend:sha-abc is pullable as dev.jinemi.com/jinemi/thermograph/backend:sha-abc — the host part is only how you reach the registry, not part of the package's identity. That is what makes this migration safe to do in one step instead of a dual-push period: no image needs re-pushing, and a rollback to an old tag still resolves.

caddy-git.conf lists both names on a single site block. That is deliberate, not cosmetic: the /v2/* mesh-only matcher is per-block, so a second block for either name would re-expose the registry API publicly and undo hazard #15. Whatever else changes, keep the two names in one block.

git.thermograph.org must stay served. The list of reasons is shorter than it was, but it is not empty, and neither remaining item is a bookmark:

• image tags already pushed under the old prefix name that host, so a rollback to one asks for git.thermograph.org/... even though nothing builds that name any more;
• registered runners hold the instance URL they registered with (https://git.thermograph.org); they keep working while that name resolves, and stop the moment it doesn't. Re-registering them is the only way to retire the name, and it is not required for this migration.

Host-side steps

None of these live in the repo, and CI will not do them for you. In order:

1. /etc/hosts on every mesh client — pin dev.jinemi.com as well as git.thermograph.org (both, for the reasons under "Registry access from mesh clients" below). A client that resolves the new name publicly gets a 403 from the /v2/* matcher, which reads exactly like a bad credential.
2. docker login dev.jinemi.com on each host that pushes or pulls — the CI runners, vps2 (prod and beta), the desktop. A login against the old name does not carry over: docker keys stored credentials by registry host.
3. The Forgejo Actions variable REGISTRY_HOST → dev.jinemi.com (Site Administration → Actions → Variables, or the repo's own). This is what build-push.yml reads; the defaults in this repo are fallbacks for by-hand runs, so leaving the variable stale silently keeps CI on the old name.
4. Re-register the runners against https://dev.jinemi.com, optionally — see above. Until you do, leave git.thermograph.org served.

Verify with a push before assuming: the failure mode for a missed step 1 or 2 is unauthorized: reqPackageAccess, which names neither.

Before changing ROOT_URL again

Login is Google-SSO-only, so the Google OAuth client must already carry a matching redirect URI, or nobody can reach the UI — including to undo the change. Both of these are registered on client 337247886376-4360kepi20f24pue62ptg0lnqhcht46l.apps.googleusercontent.com:

https://dev.jinemi.com/user/oauth2/google/callback
https://git.thermograph.org/user/oauth2/google/callback

Verify rather than assume. Google answers this with no credentials: a registered URI serves the sign-in page, an unregistered one returns redirect_uri_mismatch. Substitute the host you intend to make canonical:

CID=337247886376-4360kepi20f24pue62ptg0lnqhcht46l.apps.googleusercontent.com
curl -sL "https://accounts.google.com/o/oauth2/v2/auth?client_id=$CID&redirect_uri=https%3A%2F%2F<host>%2Fuser%2Foauth2%2Fgoogle%2Fcallback&response_type=code&scope=openid+email+profile&state=probe" \
  | grep -q redirect_uri_mismatch && echo REJECTED || echo ACCEPTED

The stack has no auto-deploy: a FORGEJO_DOMAIN change here only takes effect when someone re-runs docker stack deploy by hand on the manager (vps2) — see "Deploy / update" above. Until then the running service keeps whatever ROOT_URL it was last deployed with, regardless of what this file says.

Registry access from mesh clients

Any node that needs docker login/push/pull against the registry (the CI runner building/pushing images, any Swarm node pulling them, prod or beta on vps2 pulling app images) must reach Forgejo over the WireGuard tunnel, not vps1's public IP — otherwise Caddy's /v2/* block above refuses the connection. Public DNS resolves both names to vps1's public IP, so add an /etc/hosts override on each such node pinning them to vps1's WireGuard address instead:

echo "10.10.0.2 dev.jinemi.com git.thermograph.org" | sudo tee -a /etc/hosts

(10.10.0.2 is vps1's WG address per deploy/swarm/README.md's peer numbering — adjust if you assigned it differently.) The git/web UI keeps working normally for everyone else since only /v2/* is restricted.

Both names, not just the one in the image. dev.jinemi.com now carries two independent jobs and needs the pin for each: it is the host in every image name, and it is ROOT_URL, from which Forgejo derives the registry's bearer-token realm — so a client is sent to https://<ROOT_URL host>/v2/token to collect a token no matter which name it dialled. git.thermograph.org stays pinned for a third reason: image tags pushed before the migration name it, and a rollback to one of those dials it directly.

This is not hypothetical. When ROOT_URL first became dev.jinemi.com while images were still named git.thermograph.org, the token request took the public route, the /v2/* matcher answered 403, and docker fell back to anonymous — every push failed with unauthorized: reqPackageAccess, which reads exactly like a revoked token or a missing scope. It was neither. Pinning one name and not the other reproduces it in either direction.

The realm moves whenever ROOT_URL does, so read it rather than assuming:

curl -sI https://dev.jinemi.com/v2/ | grep -i www-authenticate

These /etc/hosts pins are host state. Nothing in this repo writes them, so a reprovisioned node has to be pinned again before it can pull.

Register the Actions runner

Once Forgejo answers at its domain:

# On the runner host (see DEPLOY-DEV.md for where that is today):
#   Forgejo web UI -> Site Administration -> Actions -> Runners -> Create new Runner
#   (or, for a repo-scoped runner: <repo> -> Settings -> Actions -> Runners)
#   copy the registration token, then:
bash deploy/forgejo/register-lan-runner.sh https://<forgejo-domain> <token>

See that script's header for exactly what it replaces (the pre-Forgejo GitHub self-hosted runner) and why it registers with two labels where there used to be two separate runners.

config.yaml's runner.capacity is raised from the tool's default of 1 to 8 (override with CAPACITY=) — a single PR push fires pr-build, secrets-guard, and shell-lint simultaneously (3 independent workflows, no needs: between them), so capacity 1 serializes work that could run in parallel, and even capacity 3 (an earlier, undocumented hand-tune) leaves build-backend/build-frontend/validate-observability queued behind those three before they get a slot.

This paragraph used to say a runner must never go on vps2. That was reversed deliberately on 2026-08-01, and the reasoning is worth keeping rather than quietly deleting.

The original objection stands on its merits: container.docker_host: automount gives job containers the host's Docker socket, so a runner on vps2 means any CI job has root-equivalent access to both the prod and beta stacks running there. An earlier revision of this stack ran the runner as a Swarm-hosted container on the Forgejo node and was reverted for the same class of reason.

What changed is that the alternative turned out to be worse. The desktop was the estate's only registered runner, and when it went offline on 2026-07-31 nothing merged, nothing deployed, and the nightly backup did not fire for 21 hours — on a repo where every branch is protected and every change is a PR, a single absent runner freezes the whole estate. Availability of CI is not a convenience here; it is what the backup and the deploy path both hang off.

So the runner on vps2 (runner-vps2/) is a considered trade, not an oversight, and it is bounded rather than assumed benign:

• capacity: 1 — one job at a time, so a CI burst cannot contend with thermograph_web for the box's six cores.
• --cpus=2 --memory=4g on job containers, set in container.options. Job containers are siblings of the runner rather than children, so the runner's own compose limits do not reach them; that option is the only lever that does.
• valid_volumes: [] — a job cannot bind-mount a host path, which is the difference between "a job can read /etc/thermograph.env" and "a job cannot", on the box where that file is prod's.
• The compose project joins no thermograph network.

Be honest about what that buys: it is defence against accident, not against a hostile workflow author. Forgejo's own database on vps1 already stores VPS2_SSH_KEY, which is root on vps2, so the material was already reachable — what changed is that it is now reachable by a job rather than only at rest. On a two-person estate where both people can already SSH to that box as root, that is the honest boundary.

If you are adding capacity rather than redundancy, raise capacity or add a runner on a box that hosts nothing. The vps2 runner exists so that CI has a second home, not because prod is a good place to run CI.

Custom CI job image (ci-runner/)

ci-runner/Dockerfile still bases on node:20-bookworm — Node is a hard requirement, not leftover: Forgejo's runner executes actions/checkout@v4 (used by every workflow) as node dist/index.js inside the job container, regardless of whether the workflow itself uses npm/node. (v1 of this image tried a Node-free Debian-slim base and broke every job's checkout step — node: executable file not found — within a minute of going live; reverted immediately.) What it actually fixes: every docker-labeled build-push job currently re-installs the Docker CLI on each run (apt-get install docker.io), which pulls in the classic builder rather than BuildKit (the classic builder mishandles COPY --chown=<name> group resolution — a real bug hit during the frontend Go rewrite). ci-runner adds docker-ce-cli + docker-buildx-plugin (BuildKit) on top of the same Node base, plus git/python3/python3-yaml for the other jobs that need them (shell-lint, observability-validate).

Current tag: dev.jinemi.com/jinemi/thermograph/ci-runner:v2 (v1 is broken — do not register any runner against it). Rebuild/push (requires a PAT with write:package scope — the embedded git-remote token lacks it, same requirement documented in build-push.yml):

docker build -t dev.jinemi.com/jinemi/thermograph/ci-runner:vN deploy/forgejo/ci-runner
docker push dev.jinemi.com/jinemi/thermograph/ci-runner:vN

register-lan-runner.sh's LABELS default points at the current tag, so fresh registrations pick it up automatically. The live runner is cut over by editing the labels array in ~/forgejo-runner/.runner on the runner host (same runner id/token, no re-registration needed) and restarting the service — verify a real job runs green under the new image before relying on it, same way v1's break was caught. Only after that verification should the now-redundant apt-get install docker.io / python3-yaml steps be removed from the workflows that had them — removing them first would break every job still running on the stock node:20-bookworm image.

Why Postgres here and not the Thermograph app's TimescaleDB

Separate instance, separate network (forgejo_net, not the app's overlay network), separate volume. Forgejo is a distinct product with its own schema and its own backup/restore lifecycle — sharing a database with the app would couple two things that should be able to fail, migrate, and restore independently.

Verifying

docker service ls                       # forgejo_db, forgejo_forgejo both Running, 1/1
curl -I https://dev.jinemi.com/         # 200, valid cert (Caddy's, not a new one)
curl -I https://dev.jinemi.com/v2/      # 403 from anywhere off the WireGuard mesh
curl -I https://git.thermograph.org/    # 200 too — the alias must stay served
# On the runner host, after registering the runner:
systemctl --user status forgejo-runner   # active, both labels registered

Rollback / removal

docker stack rm forgejo
# volumes (forgejo_data, forgejo_db, ...) survive a stack rm — remove them
# explicitly only if you actually want to destroy the Forgejo instance's data.