thermograph/infra/deploy/forgejo/README.md

# Forgejo on the Swarm cluster

Runs as `deploy/forgejo/docker-stack.yml` — the only Swarm-scheduled workload
this cluster carries (prod and beta's app stacks are separate `docker stack
deploy`s that happen to run on the manager node, vps2 — see
`DEPLOY.md`). Pinned to the **vps1** node
(`75.119.132.91`) via the `role=forge` label from
`deploy/swarm/label-forge-node.sh`.

The Actions **runner** is deliberately *not* part of this stack: a
Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would
be unavailable exactly when the cluster is what needs repairing. An earlier
revision did run it as a Swarm-scheduled Docker-in-Docker sidecar pinned to the
Forgejo node; that is gone.

Runners live in two places, and only one of them counts:

- **`runner-vps2/`** — the always-on runner, a plain `restart: always` container
  on vps2. This is the one CI depends on.
- **`register-lan-runner.sh`** — the desktop's systemd runner. Extra capacity.
  **Nothing may assume it is up.** It was the only registered runner in the
  estate until 2026-08-01, and its 21-hour outage on 2026-07-31 froze every
  merge and deploy and stopped the nightly backup from firing.

## Prerequisites

1. All three nodes have joined the swarm (`deploy/swarm/`) and vps1 is
   labeled `role=forge`.
2. `docker node ls` (from the manager, vps2) shows all three `Ready`.

## One-time setup: Swarm secret

One secret the stack expects to already exist (a Swarm secret, not a file —
`external: true` in the stack file, so `docker stack deploy` never creates or
sees the value, only references it):

```bash
# A strong random password for Forgejo's own Postgres (NOT related to
# Thermograph's app database — entirely separate instance/network).
openssl rand -base64 32 | docker secret create forgejo_db_password -
```

## Deploy / update

```bash
docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo
```

Re-running is safe — Swarm only touches services whose spec actually changed.
Do this **before** the DNS + Caddy step below — Caddy's reverse_proxy target
(`127.0.0.1:3080`) needs the `forgejo` service actually listening first, or
its first health check just fails harmlessly until it is.

This stack has **no auto-deploy trigger** — nothing in `.forgejo/workflows/`
redeploys it on push. A change to `docker-stack.yml` only takes effect once
someone re-runs `docker stack deploy` by hand on the manager (vps2).

`db`/`forgejo` both carry `resources.limits` (defaults: db 1 CPU/1g, forgejo 2
CPU/2g — several times observed steady-state usage), overridable with
`FORGEJO_DB_CPUS`/`FORGEJO_DB_MEMORY`/`FORGEJO_CPUS`/`FORGEJO_MEMORY` env vars
before `docker stack deploy`, same convention as the app stack.

## DNS + TLS: reusing vps1's existing Caddy, not a second reverse proxy

Forgejo is pinned to vps1 (`role=forge`) — the same box that also runs
Grafana/Loki/Alloy and the `emigriffith.dev` portfolio site, each fronted by
that host's own Caddy. A second ingress (Traefik) trying to bind the same
ports would collide with it. So there's no Traefik in this stack: `forgejo`'s
web port publishes to `127.0.0.1:3080` only (host-local), and vps1's
*existing* Caddy gets one more site block reverse-proxying to it — same
pattern as its other site blocks, same automatic-HTTPS.

1. Point the Forgejo domain (default `git.thermograph.org`; override with
   `FORGEJO_DOMAIN=...` before `docker stack deploy`) at **vps1's** public IP
   — that's where the task actually runs, not vps2's or the desktop's.
2. Append `deploy/forgejo/caddy-git.conf` to vps1's `/etc/caddy/Caddyfile`,
   adjusting the domain if you didn't use the default, then `systemctl reload
   caddy`.
3. That file also resolves the registry-exposure hazard (#15 in
   `thermograph-docs/runbooks/hop1-forgejo-registry-cutover.md`): `/v2/*` (the registry
   API) is blocked to everything except the WireGuard mesh CIDR
   (`10.10.0.0/24`); the git/web UI stays public. CI runners and Swarm nodes
   reach the registry over the mesh, not the public internet — see "Registry
   access from mesh clients" below.

## Migrating the domain (git.thermograph.org -> dev.jinemi.com)

**Web UI and OAuth: done.** `dev.jinemi.com` is Forgejo's `ROOT_URL`
(`FORGEJO_DOMAIN` in `docker-stack.yml`), so it is the name Forgejo generates in
clone URLs, the Google OAuth callback, webhook payload URLs and mail links.

**Registry: deliberately not done.** Renaming the registry host is a separate
migration from renaming the web UI, and the two need not happen together.

`caddy-git.conf` lists both names on a *single* site block. That is deliberate,
not cosmetic: the `/v2/*` mesh-only matcher is per-block, so a second block for
either name would re-expose the registry API publicly and undo hazard #15.
Whatever else changes, keep the two names in one block.

**`git.thermograph.org` must stay served.** It is not a courtesy redirect for
old bookmarks — CI resolves it:

- images are named by registry host, and the `git.thermograph.org/` prefix is
  baked into the runner labels (`register-lan-runner.sh`,
  `runner-vps2/README.md`), the CI-runner image (`ci-runner/Dockerfile`),
  `REGISTRY_HOST` in `infra/.env.example`, and every already-pushed tag;
- every mesh client in "Registry access from mesh clients" below pins
  `git.thermograph.org` to `10.10.0.2` in `/etc/hosts`, which is what makes
  Caddy's `/v2/*` matcher see a mesh source IP instead of returning 403;
- registered runners hold the instance URL they registered with
  (`https://git.thermograph.org`); they keep working while that name resolves.

### Before changing ROOT_URL again

Login is Google-SSO-only, so the Google OAuth client must already carry a
matching redirect URI, or nobody can reach the UI — including to undo the
change. Both of these are registered on client
`337247886376-4360kepi20f24pue62ptg0lnqhcht46l.apps.googleusercontent.com`:

    https://dev.jinemi.com/user/oauth2/google/callback
    https://git.thermograph.org/user/oauth2/google/callback

Verify rather than assume. Google answers this with no credentials: a
registered URI serves the sign-in page, an unregistered one returns
`redirect_uri_mismatch`. Substitute the host you intend to make canonical:

```sh
CID=337247886376-4360kepi20f24pue62ptg0lnqhcht46l.apps.googleusercontent.com
curl -sL "https://accounts.google.com/o/oauth2/v2/auth?client_id=$CID&redirect_uri=https%3A%2F%2F<host>%2Fuser%2Foauth2%2Fgoogle%2Fcallback&response_type=code&scope=openid+email+profile&state=probe" \
  | grep -q redirect_uri_mismatch && echo REJECTED || echo ACCEPTED
```

The stack has **no auto-deploy**: a `FORGEJO_DOMAIN` change here only takes
effect when someone re-runs `docker stack deploy` by hand on the manager (vps2)
— see "Deploy / update" above. Until then the running service keeps whatever
`ROOT_URL` it was last deployed with, regardless of what this file says.

## Registry access from mesh clients

Any node that needs `docker login`/push/pull against the registry (the CI
runner building/pushing images, any Swarm node pulling them, prod or beta on
vps2 pulling app images) must reach `git.thermograph.org` **over the
WireGuard tunnel**, not vps1's public IP — otherwise Caddy's `/v2/*` block
above refuses the connection. Public DNS resolves the domain to vps1's public
IP, so add a `/etc/hosts` override on each such node pinning it to vps1's
WireGuard address instead:

```
echo "10.10.0.2 git.thermograph.org" | sudo tee -a /etc/hosts
```

(`10.10.0.2` is vps1's WG address per `deploy/swarm/README.md`'s peer
numbering — adjust if you assigned it differently.) The git/web UI keeps
working normally for everyone else since only `/v2/*` is restricted.

**Pin the canonical name too, not just the image host.** Forgejo derives the
registry's bearer-token realm from `ROOT_URL`, so a client that already
resolves `git.thermograph.org` over the mesh is still sent to
`https://<ROOT_URL host>/v2/token` to collect a token. When `ROOT_URL` became
`dev.jinemi.com`, that second request took the public route, the `/v2/*`
matcher above answered 403, and docker fell back to anonymous — every push then
failed with `unauthorized: reqPackageAccess`, which reads exactly like a
revoked token or a missing scope. It is neither. Pin both names:

```
echo "10.10.0.2 git.thermograph.org dev.jinemi.com" | sudo tee -a /etc/hosts
```

The realm moves whenever `ROOT_URL` does, so read it rather than assuming:

```sh
curl -sI https://git.thermograph.org/v2/ | grep -i www-authenticate
```

These `/etc/hosts` pins are host state. Nothing in this repo writes them, so a
reprovisioned node has to be pinned again before it can pull.

## Register the Actions runner

Once Forgejo answers at its domain:

```bash
# On the runner host (see DEPLOY-DEV.md for where that is today):
#   Forgejo web UI -> Site Administration -> Actions -> Runners -> Create new Runner
#   (or, for a repo-scoped runner: <repo> -> Settings -> Actions -> Runners)
#   copy the registration token, then:
bash deploy/forgejo/register-lan-runner.sh https://<forgejo-domain> <token>
```

See that script's header for exactly what it replaces (the pre-Forgejo GitHub
self-hosted runner) and why it registers with two labels where there used to
be two separate runners.

`config.yaml`'s `runner.capacity` is raised from the tool's default of 1 to 8
(override with `CAPACITY=`) — a single PR push fires `pr-build`,
`secrets-guard`, and `shell-lint` simultaneously (3 independent workflows,
no `needs:` between them), so capacity 1 serializes work that could run in
parallel, and even capacity 3 (an earlier, undocumented hand-tune) leaves
`build-backend`/`build-frontend`/`validate-observability` queued behind
those three before they get a slot.

**This paragraph used to say a runner must never go on vps2. That was reversed
deliberately on 2026-08-01, and the reasoning is worth keeping rather than
quietly deleting.**

The original objection stands on its merits: `container.docker_host: automount`
gives job containers the *host's* Docker socket, so a runner on vps2 means any
CI job has root-equivalent access to both the prod and beta stacks running
there. An earlier revision of this stack ran the runner as a Swarm-hosted
container on the Forgejo node and was reverted for the same class of reason.

What changed is that the alternative turned out to be worse. The desktop was
the estate's **only** registered runner, and when it went offline on
2026-07-31 nothing merged, nothing deployed, and the nightly backup did not
fire for 21 hours — on a repo where every branch is protected and every change
is a PR, a single absent runner freezes the whole estate. Availability of CI
is not a convenience here; it is what the backup and the deploy path both
hang off.

So the runner on vps2 (`runner-vps2/`) is a considered trade, not an
oversight, and it is bounded rather than assumed benign:

- `capacity: 1` — one job at a time, so a CI burst cannot contend with
  `thermograph_web` for the box's six cores.
- `--cpus=2 --memory=4g` on job containers, set in `container.options`. Job
  containers are siblings of the runner rather than children, so the runner's
  own compose limits do not reach them; that option is the only lever that
  does.
- `valid_volumes: []` — a job cannot bind-mount a host path, which is the
  difference between "a job can read `/etc/thermograph.env`" and "a job
  cannot", on the box where that file is prod's.
- The compose project joins no `thermograph` network.

Be honest about what that buys: it is defence against **accident**, not
against a hostile workflow author. Forgejo's own database on vps1 already
stores `VPS2_SSH_KEY`, which is root on vps2, so the material was already
reachable — what changed is that it is now reachable by a *job* rather than
only at rest. On a two-person estate where both people can already SSH to that
box as root, that is the honest boundary.

**If you are adding capacity rather than redundancy, raise `capacity` or add a
runner on a box that hosts nothing.** The vps2 runner exists so that CI has a
second home, not because prod is a good place to run CI.

## Custom CI job image (`ci-runner/`)

`ci-runner/Dockerfile` still bases on `node:20-bookworm` — Node is a hard
requirement, not leftover: Forgejo's runner executes `actions/checkout@v4`
(used by every workflow) as `node dist/index.js` *inside the job container*,
regardless of whether the workflow itself uses npm/node. (v1 of this image
tried a Node-free Debian-slim base and broke every job's checkout step —
`node: executable file not found` — within a minute of going live; reverted
immediately.) What it actually fixes: every `docker`-labeled build-push job
currently re-installs the Docker CLI on each run (`apt-get install
docker.io`), which pulls in the classic builder rather than BuildKit (the
classic builder mishandles `COPY --chown=<name>` group resolution — a real
bug hit during the frontend Go rewrite). `ci-runner` adds `docker-ce-cli` +
`docker-buildx-plugin` (BuildKit) on top of the same Node base, plus
`git`/`python3`/`python3-yaml` for the other jobs that need them
(`shell-lint`, `observability-validate`).

Current tag: `git.thermograph.org/jinemi/thermograph/ci-runner:v2` (`v1` is
broken — do not register any runner against it). Rebuild/push (requires a PAT
with `write:package` scope — the embedded git-remote token lacks it, same
requirement documented in `build-push.yml`):

```bash
docker build -t git.thermograph.org/jinemi/thermograph/ci-runner:vN deploy/forgejo/ci-runner
docker push git.thermograph.org/jinemi/thermograph/ci-runner:vN
```

`register-lan-runner.sh`'s `LABELS` default points at the current tag, so
fresh registrations pick it up automatically. The live runner is cut over by
editing the `labels` array in `~/forgejo-runner/.runner` on the runner host
(same runner id/token, no re-registration needed) and restarting the service —
**verify a real job runs green under the new image before relying on it**,
same way v1's break was caught. Only after that verification should the
now-redundant `apt-get install docker.io` / `python3-yaml` steps be removed
from the workflows that had them — removing them first would break every job
still running on the stock `node:20-bookworm` image.

## Why Postgres here and not the Thermograph app's TimescaleDB

Separate instance, separate network (`forgejo_net`, not the app's overlay
network), separate volume. Forgejo is a distinct product with its own schema
and its own backup/restore lifecycle — sharing a database with the app would
couple two things that should be able to fail, migrate, and restore
independently.

## Verifying

```bash
docker service ls                       # forgejo_db, forgejo_forgejo both Running, 1/1
curl -I https://git.thermograph.org/    # 200, valid cert (Caddy's, not a new one)
curl -I https://git.thermograph.org/v2/ # 403 from anywhere off the WireGuard mesh
# On the runner host, after registering the runner:
systemctl --user status forgejo-runner   # active, both labels registered
```

## Rollback / removal

```bash
docker stack rm forgejo
# volumes (forgejo_data, forgejo_db, ...) survive a stack rm — remove them
# explicitly only if you actually want to destroy the Forgejo instance's data.
```