Add agent VPS access, a 2-node Docker Swarm, and Forgejo CI/CD (#234)

Three additive infrastructure layers on top of the two VPS boxes Terraform already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old VPS, 75.119.132.91). None of this touches backend/, Dockerfile, docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the app-containerization work in flight elsewhere; this is strictly the layer on top. See INFRA.md for the full runbook and order of operations. - deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login (not raw root) for agent-driven ops — passwordless sudo under a distinct username, sshd hardened to key-only auth, auditd logging every root-effective command. One line to revoke. - deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a WireGuard tunnel rather than trusting the public internet with the overlay data plane, which Docker's own guidance says should never face it directly. Swarm ports locked to the tunnel interface once joined. This cluster's only workload is Forgejo — it does not orchestrate the Terraform-managed app deploys, so nothing here can strand the app's single-writer database. - deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom auto-merge workflow step is dropped — it existed only to work around GitHub's paywalled branch protection on private free-tier repos, which Forgejo has no such tier for; native "auto merge when checks succeed" replaces it, and as a real git push (unlike GitHub's non-triggering token-merge) it fires the LAN deploy naturally with no double-trigger logic needed. appleboy/ssh-action is referenced by full URL (not mirrored on Forgejo's default action registry); actions/checkout and actions/setup-python resolve unchanged. Migration is mirror-first: the GitHub repo import and workflow files land here, but cutting deploy secrets over and retiring GitHub happens only after verification (INFRA.md 3d) — GitHub stays live as a fallback throughout. One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but terraform.tfvars.example names prod's deploy branch `release`. Left as a faithful mirror rather than guessed at — reconcile with whoever's driving Terraform/deploy.
2026-07-20 17:36:39 -07:00 · 2026-07-20 17:36:39 -07:00 · 0d8fc9f4d0
commit 0d8fc9f4d0
parent 213b3818e7
11 changed files with 761 additions and 0 deletions
--- a/INFRA.md
+++ b/INFRA.md
@ -0,0 +1,96 @@
+# Agent access, Docker Swarm, and Forgejo CI/CD
+
+This covers three additive infrastructure layers on top of the two VPS boxes
+Terraform already provisions (see `terraform/README.md` and
+`terraform.tfvars.example`, where they're named **prod** — the new 48 GB /
+12-core box, `thermograph.org` — and **beta** — the old VPS,
+`75.119.132.91`). **None of this touches `backend/`, `Dockerfile`,
+`docker-compose*.yml`, `terraform/`, or `deploy/db/`** — those stay owned by
+the app-containerization work; this is strictly the layer above it.
+
+```
+Track 1: deploy/provision-agent-access.sh   — a dedicated full-root login for me
+Track 2: deploy/swarm/                      — Swarm cluster spanning both boxes
+Track 3: deploy/forgejo/ + .forgejo/         — Forgejo + Forgejo Actions, replacing GitHub
+```
+
+## Track 1 — Agent access
+
+Run `sudo bash deploy/provision-agent-access.sh` on **both** boxes. Creates a
+dedicated `agent` user (not raw root login — a distinct name gives a clean
+audit trail) with passwordless sudo, installs the agent's public key, disables
+SSH password auth fleet-wide, and turns on `auditd` logging of every
+root-effective command. See the script's header comment for the full
+rationale and revocation steps (one line to delete, or delete the whole
+account — your own access is never affected).
+
+## Track 2 — Swarm
+
+See `deploy/swarm/README.md` for the exact order of operations (WireGuard
+tunnel first, then swarm init/join, then lock the Swarm ports down to the
+tunnel interface, then label beta for Forgejo placement). This cluster's only
+job is hosting Forgejo — it does not orchestrate the Terraform-managed app
+deploys.
+
+## Track 3 — Forgejo, replacing GitHub
+
+### 3a. Stand up Forgejo
+See `deploy/forgejo/README.md` — deploys the stack, walks through minting the
+two Swarm secrets it needs (`forgejo_db_password`, `forgejo_runner_token`).
+
+### 3b. Migrate the repo (mirror first, verify, then cut over)
+1. In Forgejo: **+ New Migration → GitHub**. Point it at
+   `https://github.com/griffemi/thermograph`, pull code + issues + PRs +
+   releases + labels. This is non-destructive to GitHub — do this as a mirror
+   and leave GitHub live and untouched until you're confident.
+2. Spot-check: branch list matches, a few PRs/issues render correctly, LFS (if
+   any) came across.
+3. **Don't retarget any secret or disable a GitHub workflow yet** — see 3d.
+
+### 3c. Workflows
+`.forgejo/workflows/*.yml` already mirror `.github/workflows/*.yml` in this
+PR — copied with the mechanical changes Forgejo needs:
+- `runs-on: ubuntu-latest` → `runs-on: docker` (the label the Swarm-hosted
+  runner in `deploy/forgejo/docker-stack.yml` registers under).
+- `appleboy/ssh-action` referenced by full GitHub URL (not mirrored in
+  Forgejo's default action registry, unlike `actions/checkout` /
+  `actions/setup-python`, which resolve unchanged).
+- **The custom auto-merge workflow step is gone.** It existed only because
+  GitHub's native branch protection/auto-merge is paywalled on private
+  free-tier repos. Forgejo has no such tier — configure it directly:
+  **Settings → Branches → protect `dev`  → Enable Status Check → list `build`
+  → merge style: squash.** Then a ready PR gets a real "Auto merge when checks
+  succeed" button.
+- One knock-on simplification: a native auto-merge is an ordinary git push
+  (unlike GitHub's token-based merge commit, which doesn't re-trigger `push`),
+  so `deploy-dev.yml`'s push trigger fires naturally after auto-merge — no
+  more "direct push vs. PR-merge" double-trigger logic to maintain.
+- `.forgejo/workflows/deploy.yml` still targets `main` faithfully mirroring
+  the current GitHub workflow, but `terraform.tfvars.example` describes prod's
+  branch as `release`. That mismatch is flagged in the file's own header
+  comment — reconcile with whoever's driving the Terraform/deploy side rather
+  than guessing here.
+
+### 3d. Cut over
+Only after 3a–3c are verified:
+1. Move the deploy secrets (`SSH_HOST`, `SSH_USER`, `SSH_KEY`, `SSH_PORT`) into
+   Forgejo's repo Secrets (Settings → Actions → Secrets).
+2. Re-point the LAN dev runner: `bash deploy/forgejo/register-lan-runner.sh
+   <forgejo_url> <token>` — stops/disables the old GitHub runner service on
+   that same machine and registers `forgejo-runner` in its place, same
+   `systemd --user` pattern as before (see `DEPLOY-DEV.md` for the original;
+   this supersedes it for the runner specifically).
+3. Open a test PR against `dev` in Forgejo, confirm the required check runs,
+   auto-merge fires, and the LAN deploy lands.
+4. Only once that's solid: disable/archive the `.github/workflows/*.yml`
+   triggers (or the whole GitHub repo) — GitHub was the fallback during this
+   whole migration; retire it deliberately, not by accident.
+
+## Verification checklist
+- [ ] `ssh agent@<prod>` and `ssh agent@<beta>` both work; `sudo whoami` → `root`
+- [ ] Password SSH auth confirmed dead on both boxes
+- [ ] `docker node ls` (from prod) shows both nodes `Ready`
+- [ ] Swarm ports (2377/7946/4789) unreachable from outside the WireGuard tunnel
+- [ ] `https://git.thermograph.org` (or your chosen domain) serves Forgejo over TLS
+- [ ] A test PR: required check runs → auto-merges → LAN dev deploy lands
+- [ ] GitHub retired only after all of the above hold
--- a/deploy/forgejo/README.md
+++ b/deploy/forgejo/README.md
@ -0,0 +1,74 @@
+# Forgejo on the Swarm cluster
+
+Runs as `deploy/forgejo/docker-stack.yml` — the only workload this Swarm
+cluster carries (the Thermograph app itself stays on the Terraform-managed
+`docker compose` deploys; see `terraform/README.md`). Pinned to the **beta**
+node (old VPS) via the `role=forge` label from
+`deploy/swarm/label-forge-node.sh`.
+
+## Prerequisites
+
+1. Both boxes have joined the swarm (`deploy/swarm/`) and beta is labeled
+   `role=forge`.
+2. `docker node ls` (from the manager) shows both `Ready`.
+
+## One-time setup: Swarm secrets
+
+Two secrets the stack expects to already exist (Swarm secrets, not files —
+`external: true` in the stack file, so `docker stack deploy` never creates or
+sees the values, only references them):
+
+```bash
+# A strong random password for Forgejo's own Postgres (NOT related to
+# Thermograph's app database — entirely separate instance/network).
+openssl rand -base64 32 | docker secret create forgejo_db_password -
+
+# The runner registration token. Forgejo can't issue one before it's running,
+# so this is a two-step dance the first time:
+docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo   # 1. bring Forgejo up (runner will crashloop briefly — expected)
+# 2. once Forgejo answers at the domain, log in, go to
+#    Site Administration -> Actions -> Runners -> Create new Runner
+#    (or, for a repo-scoped runner: <repo> -> Settings -> Actions -> Runners),
+#    copy the token, then:
+echo -n "PASTE_TOKEN_HERE" | docker secret create forgejo_runner_token -
+docker service update --force forgejo_runner   # picks up the new secret and registers
+```
+
+## Deploy / update
+
+```bash
+docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo
+```
+
+Re-running is safe — Swarm only touches services whose spec actually changed.
+
+## DNS
+
+Point the Forgejo domain (default `git.thermograph.org`; override with
+`FORGEJO_DOMAIN=...` before `docker stack deploy`, Swarm reads it from the
+deploying shell's environment) at **either** node's public IP — the routing
+mesh forwards published ports to wherever the task actually landed.
+
+## Why Postgres here and not the Thermograph app's TimescaleDB
+
+Separate instance, separate network (`forgejo_net`, not the app's compose
+network), separate volume. Forgejo is a distinct product with its own schema
+and its own backup/restore lifecycle — sharing a database with the app would
+couple two things that should be able to fail, migrate, and restore
+independently.
+
+## Verifying
+
+```bash
+docker service ls                       # all forgejo_* services Running, 1/1
+curl -I https://git.thermograph.org/    # 200, valid cert
+docker service logs forgejo_runner --tail 50   # "runner: successfully registered" then idle
+```
+
+## Rollback / removal
+
+```bash
+docker stack rm forgejo
+# volumes (forgejo_data, forgejo_db, ...) survive a stack rm — remove them
+# explicitly only if you actually want to destroy the Forgejo instance's data.
+```
--- a/deploy/forgejo/docker-stack.yml
+++ b/deploy/forgejo/docker-stack.yml
@ -0,0 +1,174 @@
+# Forgejo (self-hosted Git + CI/CD) as a Docker Swarm stack — the only workload
+# this Swarm cluster runs (see deploy/swarm/README.md). Deliberately separate
+# from the Terraform-managed docker-compose.yml that runs the Thermograph app
+# itself: this stack's only job is Forgejo, its runner, and the ingress in
+# front of it.
+#
+# Deploy from the manager node (prod), after both boxes have joined the swarm
+# and beta is labeled role=forge:
+#
+#   docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo
+#
+# Requires these Swarm secrets to exist first (see deploy/forgejo/README.md):
+#   forgejo_db_password, forgejo_runner_token
+#
+# DNS: point git.thermograph.org (or your chosen domain — see plan) at EITHER
+# node's public IP. Swarm's routing mesh accepts published ports on every node
+# and forwards to wherever the task actually runs, so either IP works.
+
+services:
+  db:
+    image: postgres:16-alpine
+    environment:
+      POSTGRES_USER: forgejo
+      POSTGRES_DB: forgejo
+      POSTGRES_PASSWORD_FILE: /run/secrets/forgejo_db_password
+    secrets: [forgejo_db_password]
+    volumes:
+      - forgejo_db:/var/lib/postgresql/data
+    networks: [forgejo_net]
+    deploy:
+      placement:
+        constraints: [node.labels.role == forge]
+      restart_policy:
+        condition: on-failure
+
+  forgejo:
+    image: codeberg.org/forgejo/forgejo:9-rootless
+    depends_on: [db]
+    environment:
+      FORGEJO__database__DB_TYPE: postgres
+      FORGEJO__database__HOST: db:5432
+      FORGEJO__database__NAME: forgejo
+      FORGEJO__database__USER: forgejo
+      FORGEJO__server__DOMAIN: "${FORGEJO_DOMAIN:-git.thermograph.org}"
+      FORGEJO__server__ROOT_URL: "https://${FORGEJO_DOMAIN:-git.thermograph.org}/"
+      FORGEJO__server__SSH_PORT: "2222"
+      # Actions on: repo/org/user-level runners register against this instance.
+      FORGEJO__actions__ENABLED: "true"
+      # Gitea/Forgejo's app.ini env-mapping honors a __FILE suffix to read a
+      # value from a file instead of the literal env var — same convention as
+      # the official Postgres image's POSTGRES_PASSWORD_FILE above.
+      FORGEJO__database__PASSWD__FILE: /run/secrets/forgejo_db_password
+      FORGEJO__database__USER: forgejo
+    secrets:
+      - source: forgejo_db_password
+        target: forgejo_db_password
+    volumes:
+      - forgejo_data:/var/lib/gitea
+    networks: [forgejo_net]
+    ports:
+      # SSH for git@ clones — Swarm's routing mesh publishes this on both nodes.
+      - target: 2222
+        published: 2222
+        protocol: tcp
+        mode: host
+    deploy:
+      placement:
+        constraints: [node.labels.role == forge]
+      restart_policy:
+        condition: on-failure
+      labels:
+        - traefik.enable=true
+        - traefik.http.routers.forgejo.rule=Host(`${FORGEJO_DOMAIN:-git.thermograph.org}`)
+        - traefik.http.routers.forgejo.entrypoints=websecure
+        - traefik.http.routers.forgejo.tls.certresolver=le
+        - traefik.http.services.forgejo.loadbalancer.server.port=3000
+
+  # Docker-in-Docker sidecar for the runner. Chosen over mounting the host
+  # docker.sock (root-equivalent host access, zero isolation) — DinD is the
+  # documented mitigation for a trusted single-maintainer runner that only
+  # ever executes this repo's own workflows.
+  runner-dind:
+    image: docker:27-dind
+    privileged: true
+    environment:
+      DOCKER_TLS_CERTDIR: ""
+    volumes:
+      - runner_dind_certs:/certs
+    networks: [forgejo_net]
+    deploy:
+      placement:
+        constraints: [node.labels.role == forge]
+      restart_policy:
+        condition: on-failure
+
+  runner:
+    image: code.forgejo.org/forgejo/runner:6
+    depends_on: [runner-dind, forgejo]
+    environment:
+      DOCKER_HOST: tcp://runner-dind:2375
+      FORGEJO_INSTANCE_URL: "https://${FORGEJO_DOMAIN:-git.thermograph.org}"
+    secrets:
+      - source: forgejo_runner_token
+        target: forgejo_runner_token
+    # Registers on first start using the token secret (see README for how to
+    # mint one), then just executes jobs — no docker.sock mount.
+    entrypoint: >
+      sh -c '
+        if [ ! -f /data/.runner ]; then
+          forgejo-runner register --no-interactive
+            --instance "$$FORGEJO_INSTANCE_URL"
+            --token "$$(cat /run/secrets/forgejo_runner_token)"
+            --name swarm-runner --labels docker:docker://node:20-bookworm;
+        fi;
+        forgejo-runner daemon
+      '
+    volumes:
+      - runner_data:/data
+    networks: [forgejo_net]
+    deploy:
+      placement:
+        constraints: [node.labels.role == forge]
+      restart_policy:
+        condition: on-failure
+
+  traefik:
+    image: traefik:v3.1
+    command:
+      - --providers.swarm.endpoint=unix:///var/run/docker.sock
+      - --providers.swarm.exposedbydefault=false
+      - --entrypoints.web.address=:80
+      - --entrypoints.websecure.address=:443
+      - --entrypoints.web.http.redirections.entrypoint.to=websecure
+      - --entrypoints.web.http.redirections.entrypoint.scheme=https
+      - --certificatesresolvers.le.acme.email=${ACME_EMAIL:-admin@thermograph.org}
+      - --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
+      - --certificatesresolvers.le.acme.httpchallenge.entrypoint=web
+    ports:
+      - target: 80
+        published: 80
+        mode: host
+      - target: 443
+        published: 443
+        mode: host
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - traefik_certs:/letsencrypt
+    networks: [forgejo_net]
+    deploy:
+      placement:
+        # Pinned to the same node as the ACME cert store volume — a floating
+        # Traefik would re-request certs from Let's Encrypt on every
+        # reschedule and hit its rate limit.
+        constraints: [node.labels.role == forge]
+      restart_policy:
+        condition: on-failure
+
+networks:
+  forgejo_net:
+    driver: overlay
+    attachable: false
+
+volumes:
+  forgejo_db:
+  forgejo_data:
+  runner_data:
+  runner_dind_certs:
+  traefik_certs:
+
+secrets:
+  forgejo_db_password:
+    external: true
+  forgejo_runner_token:
+    external: true
--- a/deploy/forgejo/register-lan-runner.sh
+++ b/deploy/forgejo/register-lan-runner.sh
@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+# Re-points the existing LAN dev self-hosted runner from GitHub Actions to
+# Forgejo Actions. Run on the SAME machine that already runs the GitHub
+# runner (see DEPLOY-DEV.md) — this replaces that runner, it doesn't add a
+# second one. Sudo-free, systemd --user, same pattern as the app service.
+#
+#   bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
+#
+# Get <registration_token> from the Forgejo web UI:
+#   repo -> Settings -> Actions -> Runners -> Create new Runner
+# (or an org/instance-level runner page, if you want it to serve more than
+# this one repo — same as the GitHub runner did).
+set -euo pipefail
+
+FORGEJO_URL="${1:?usage: $0 <forgejo_url> <registration_token>}"
+TOKEN="${2:?}"
+RUNNER_DIR="${RUNNER_DIR:-$HOME/forgejo-runner}"
+LABELS="${LABELS:-thermograph-lan}"
+
+echo "==> Stopping and disabling the old GitHub Actions runner service, if present"
+systemctl --user stop github-actions-runner 2>/dev/null || true
+systemctl --user disable github-actions-runner 2>/dev/null || true
+
+echo "==> Installing forgejo-runner into $RUNNER_DIR"
+mkdir -p "$RUNNER_DIR"
+cd "$RUNNER_DIR"
+if [ ! -x ./forgejo-runner ]; then
+  ARCH="$(uname -m)"
+  case "$ARCH" in
+    x86_64) BIN_ARCH=amd64 ;;
+    aarch64) BIN_ARCH=arm64 ;;
+    *) echo "Unsupported arch: $ARCH — download the right binary by hand from" \
+            "https://code.forgejo.org/forgejo/runner/releases" >&2; exit 1 ;;
+  esac
+  VER="${FORGEJO_RUNNER_VERSION:-6.3.1}"
+  curl -fsSL -o forgejo-runner \
+    "https://code.forgejo.org/forgejo/runner/releases/download/v${VER}/forgejo-runner-${VER}-linux-${BIN_ARCH}"
+  chmod +x forgejo-runner
+fi
+
+echo "==> Registering with $FORGEJO_URL (label: $LABELS)"
+./forgejo-runner register --no-interactive \
+  --instance "$FORGEJO_URL" \
+  --token "$TOKEN" \
+  --name "thermograph-lan-$(hostname -s)" \
+  --labels "$LABELS"
+
+echo "==> systemd --user unit"
+mkdir -p "$HOME/.config/systemd/user"
+cat > "$HOME/.config/systemd/user/forgejo-runner.service" <<EOF
+[Unit]
+Description=Forgejo Actions runner (thermograph-lan)
+After=network-online.target
+
+[Service]
+WorkingDirectory=${RUNNER_DIR}
+ExecStart=${RUNNER_DIR}/forgejo-runner daemon
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=default.target
+EOF
+
+systemctl --user daemon-reload
+systemctl --user enable --now forgejo-runner
+loginctl enable-linger "$USER" 2>/dev/null || true
+
+cat <<EOF
+
+Done. The runner now serves Forgejo, not GitHub.
+  status:  systemctl --user status forgejo-runner
+  logs:    journalctl --user -u forgejo-runner -f
+  restart: systemctl --user restart forgejo-runner
+
+The old github-actions-runner unit was stopped and disabled but not deleted —
+remove ~/actions-runner by hand once you've confirmed Forgejo deploys work.
+EOF
--- a/deploy/provision-agent-access.sh
+++ b/deploy/provision-agent-access.sh
@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+# Provisions a dedicated, full-sudo login for an AI coding agent on this VPS.
+# Idempotent — safe to re-run (e.g. to rotate the key or re-apply hardening
+# after an OS upgrade). Run as root (or via an existing sudo-capable user):
+#
+#   sudo bash /opt/thermograph/deploy/provision-agent-access.sh
+#
+# The agent's public key is embedded below (AGENT_PUBKEY) rather than passed
+# as an argument, so this file alone is the whole bootstrap — copy the current
+# one from the repo and run it; there's nothing else to fetch or fill in.
+#
+# What this does, and why it's a SEPARATE user rather than direct root login:
+#   - A distinct login name ("agent") gives a clean "who did this" audit trail
+#     that shared/root login destroys. It is still full-root-capable via
+#     passwordless sudo — this is a full-trust grant, just an attributable and
+#     independently revocable one.
+#   - Revocation is one line: delete AGENT_PUBKEY's line from
+#     /home/agent/.ssh/authorized_keys, or `deluser --remove-home agent` +
+#     `rm /etc/sudoers.d/agent`. Your own access is untouched either way.
+#
+# This script does NOT touch the app, Docker, Postgres, or Terraform-managed
+# state — it only manages the "agent" login and sshd/auditd hardening.
+set -euo pipefail
+
+# --- fill this in when rotating the key; the current value is the one
+# generated for this engagement --------------------------------------------
+AGENT_PUBKEY="${AGENT_PUBKEY:-ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICUwrp/neW5UJVmsK9SRveghJ+ayX3r59qgkL9Zo0mqO claude-agent@thermograph-infra}"
+
+AGENT_USER="${AGENT_USER:-agent}"
+
+if [ "$(id -u)" -ne 0 ]; then
+  echo "Run as root (sudo bash $0)." >&2
+  exit 1
+fi
+
+echo "==> User: $AGENT_USER"
+if ! id "$AGENT_USER" >/dev/null 2>&1; then
+  adduser --disabled-password --gecos "" "$AGENT_USER"
+fi
+
+echo "==> Passwordless sudo (full, by design — see header comment)"
+echo "$AGENT_USER ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/"$AGENT_USER"
+chmod 440 /etc/sudoers.d/"$AGENT_USER"
+visudo -cf /etc/sudoers.d/"$AGENT_USER"   # fail loudly on a syntax error rather than silently locking sudo
+
+echo "==> Authorized key"
+install -d -m 700 -o "$AGENT_USER" -g "$AGENT_USER" /home/"$AGENT_USER"/.ssh
+# Replace wholesale rather than append, so re-running after a key rotation
+# doesn't accumulate stale keys.
+echo "$AGENT_PUBKEY" > /home/"$AGENT_USER"/.ssh/authorized_keys
+chown "$AGENT_USER":"$AGENT_USER" /home/"$AGENT_USER"/.ssh/authorized_keys
+chmod 600 /home/"$AGENT_USER"/.ssh/authorized_keys
+
+echo "==> sshd hardening (fleet-wide, not just for this account)"
+SSHD_DROPIN=/etc/ssh/sshd_config.d/99-agent-hardening.conf
+cat > "$SSHD_DROPIN" <<'EOF'
+# Managed by deploy/provision-agent-access.sh — key-only login, no passwords.
+# Root itself still can't password in; PermitRootLogin prohibit-password
+# leaves direct root key-login available for a human who already has a key
+# there, without opening a password path for anyone.
+PasswordAuthentication no
+KbdInteractiveAuthentication no
+PermitRootLogin prohibit-password
+EOF
+sshd -t   # validate before reload; a bad config here can lock out the whole box
+systemctl reload sshd 2>/dev/null || systemctl reload ssh 2>/dev/null || true
+
+echo "==> auditd: log root-effective commands (survives shell history -c)"
+if ! command -v auditctl >/dev/null 2>&1; then
+  apt-get update -y -q
+  apt-get install -y -q auditd audispd-plugins
+fi
+AUDIT_RULES=/etc/audit/rules.d/agent-root-cmds.rules
+cat > "$AUDIT_RULES" <<'EOF'
+# Every root-effective exec, tagged for `ausearch -k agentcmd`.
+-a always,exit -F arch=b64 -S execve -F euid=0 -k agentcmd
+-a always,exit -F arch=b32 -S execve -F euid=0 -k agentcmd
+EOF
+augenrules --load 2>/dev/null || true
+systemctl enable --now auditd 2>/dev/null || true
+
+echo
+echo "Done. Verify from your own machine (not this box):"
+echo "  ssh -i <agent-private-key> $AGENT_USER@<this-host> sudo whoami   # -> root"
+echo "  ssh -o PasswordAuthentication=no <any-user>@<this-host>          # password path confirmed dead"
+echo
+echo "Revoke any time: delete the key line in"
+echo "  /home/$AGENT_USER/.ssh/authorized_keys"
+echo "or remove the account entirely:"
+echo "  deluser --remove-home $AGENT_USER && rm -f /etc/sudoers.d/$AGENT_USER"
--- a/deploy/swarm/README.md
+++ b/deploy/swarm/README.md
@ -0,0 +1,65 @@
+# 2-node Docker Swarm (prod + beta), for hosting Forgejo
+
+This Swarm cluster's only job is to run Forgejo (`deploy/forgejo/`) — it does
+**not** orchestrate the Thermograph app itself, which stays on the
+Terraform-managed `docker compose` deploys on each box independently (see
+`terraform/README.md`). Keeping those separate means nothing here can strand
+or interfere with the app's already-working, single-writer Postgres/TimescaleDB
+deploys.
+
+**Nodes:**
+- **manager** — prod, the new 48 GB / 12-core box (more headroom).
+- **worker** — beta, the old VPS (`75.119.132.91`).
+
+1 manager + 1 worker, not 2 managers: Raft needs 3 nodes for real quorum-based
+HA, so a second manager here would add complexity (split-brain risk) without
+adding actual failover. If the manager goes down, the worker keeps running
+whatever was already scheduled on it (Forgejo, since it's pinned there) but
+the cluster can't reschedule anything until the manager's back — acceptable
+for a 2-box hobby/small-team cluster whose only job is CI/CD.
+
+## Order of operations
+
+1. **Agent access first** (`deploy/provision-agent-access.sh`) on both boxes —
+   everything below is run through that access.
+2. **WireGuard tunnel** (`setup-wireguard.sh`) — run on both boxes; see the
+   script's header for the two-pass key-exchange dance. Verify with
+   `ping <peer_wg_ip>` before continuing.
+3. **Swarm init** (`init-swarm.sh <manager_wg_ip>`) on the manager (prod) only.
+4. **Swarm join** (`join-swarm.sh <manager_wg_ip> <token>`) on the worker
+   (beta) only, using the token `init-swarm.sh` printed.
+5. **Firewall lockdown** (`firewall-swarm.sh`) on **both** boxes — closes
+   2377/7946/4789 to everything except the WireGuard interface. Do this
+   *after* joining is confirmed working, not before (locking the ports first
+   would make the join itself fail).
+6. **Label beta** (`label-forge-node.sh <beta-node-name>`) on the manager —
+   `docker node ls` shows beta's node name/ID.
+7. Deploy Forgejo: see `deploy/forgejo/README.md`.
+
+## Why WireGuard instead of relying on Swarm's built-in TLS alone
+
+Swarm's control plane (port 2377) is TLS-encrypted and mutually authenticated
+by default. Its overlay data plane (VXLAN, port 4789) is **not** encrypted by
+default, and Docker's own guidance is that port must never face the public
+internet — these two boxes are on different providers' public IPs, not a
+private LAN, so the tunnel is the network boundary the Swarm ports advertise
+into, rather than trusting the public internet directly.
+
+## Verifying
+
+```bash
+# On the manager:
+docker node ls                      # both nodes Ready
+docker node inspect <beta-node> --format '{{.Spec.Labels}}'   # role:forge
+
+# From outside the WireGuard interface (e.g. your own machine), confirm the
+# Swarm ports are NOT reachable on the public IP:
+nc -zv -w2 <public_ip> 2377   # should fail/timeout
+nc -zvu -w2 <public_ip> 4789  # should fail/timeout
+```
+
+## Adding a node label back out (undo)
+
+```bash
+docker node update --label-rm role <beta-node>
+```
--- a/deploy/swarm/firewall-swarm.sh
+++ b/deploy/swarm/firewall-swarm.sh
@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Locks the Swarm ports (2377 control, 7946 gossip, 4789 overlay VXLAN) to the
+# WireGuard interface only — they must never be reachable from the public
+# internet. Run on BOTH boxes after joining the swarm. Existing app-facing
+# rules (80/443, SSH, etc.) are untouched.
+set -euo pipefail
+
+WG_IFACE="${WG_IFACE:-wg0}"
+
+if ! command -v ufw >/dev/null 2>&1; then
+  echo "ufw not found — apply the equivalent iptables/nftables rules by hand:" >&2
+  echo "  allow 2377/tcp, 7946/tcp, 7946/udp, 4789/udp only on interface ${WG_IFACE}" >&2
+  echo "  deny  2377/tcp, 7946/tcp, 7946/udp, 4789/udp on every other interface" >&2
+  exit 1
+fi
+
+echo "==> Allowing Swarm ports on ${WG_IFACE} only"
+ufw allow in on "$WG_IFACE" to any port 2377 proto tcp
+ufw allow in on "$WG_IFACE" to any port 7946 proto tcp
+ufw allow in on "$WG_IFACE" to any port 7946 proto udp
+ufw allow in on "$WG_IFACE" to any port 4789 proto udp
+
+echo "==> Explicitly denying the same ports on every other interface"
+ufw deny 2377/tcp
+ufw deny 7946/tcp
+ufw deny 7946/udp
+ufw deny 4789/udp
+
+echo
+echo "Rules added (not yet necessarily active — check ufw status):"
+ufw status numbered | grep -E "2377|7946|4789|Status" || true
+echo
+echo "If ufw is inactive, enable it: ufw enable   (make sure your SSH port is"
+echo "allowed FIRST, or you will lock yourself out)."
--- a/deploy/swarm/init-swarm.sh
+++ b/deploy/swarm/init-swarm.sh
@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+# Run ONCE, on the manager node only (prod — the new 48 GB box). Initializes
+# the Swarm advertising the WireGuard address, so cluster traffic never
+# touches the public interface. Run setup-wireguard.sh on both boxes first.
+set -euo pipefail
+
+MY_WG_IP="${1:?usage: $0 <my_wg_ip>}"
+
+if docker info 2>/dev/null | grep -q "Swarm: active"; then
+  echo "Swarm is already active on this node. Current state:"
+  docker node ls
+  echo
+  echo "Worker join token:"
+  docker swarm join-token -q worker
+  exit 0
+fi
+
+echo "==> docker swarm init, advertising ${MY_WG_IP} (the WireGuard address, not the public IP)"
+docker swarm init --advertise-addr "$MY_WG_IP" --listen-addr "${MY_WG_IP}:2377"
+
+echo
+echo "==> Worker join command (run this on the worker/beta node):"
+TOKEN="$(docker swarm join-token -q worker)"
+echo "    docker swarm join --token ${TOKEN} ${MY_WG_IP}:2377"
+echo
+echo "(Or run deploy/swarm/join-swarm.sh <manager_wg_ip> <token> on the worker.)"
--- a/deploy/swarm/join-swarm.sh
+++ b/deploy/swarm/join-swarm.sh
@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Run ONCE, on the worker node only (beta — the old VPS). Joins the Swarm
+# initialized by init-swarm.sh on the manager, over the WireGuard tunnel.
+set -euo pipefail
+
+MANAGER_WG_IP="${1:?usage: $0 <manager_wg_ip> <join_token>}"
+JOIN_TOKEN="${2:?}"
+
+if docker info 2>/dev/null | grep -q "Swarm: active"; then
+  echo "This node is already part of a swarm:"
+  docker info 2>/dev/null | grep -A2 "Swarm:"
+  exit 0
+fi
+
+docker swarm join --token "$JOIN_TOKEN" "${MANAGER_WG_IP}:2377"
+
+echo
+echo "Joined. Verify from the manager: docker node ls"
--- a/deploy/swarm/label-forge-node.sh
+++ b/deploy/swarm/label-forge-node.sh
@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+# Run ONCE, on the manager node, after both boxes have joined the swarm.
+# Labels the worker (beta) so the Forgejo stack's placement constraint
+# (node.labels.role == forge) schedules there and nowhere else.
+set -euo pipefail
+
+WORKER_HOSTNAME="${1:?usage: $0 <worker_node_hostname_or_id> (see: docker node ls)}"
+
+docker node update --label-add role=forge "$WORKER_HOSTNAME"
+
+echo
+echo "Labels on ${WORKER_HOSTNAME}:"
+docker node inspect "$WORKER_HOSTNAME" --format '{{.Spec.Labels}}'
--- a/deploy/swarm/setup-wireguard.sh
+++ b/deploy/swarm/setup-wireguard.sh
@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+# Sets up a point-to-point WireGuard tunnel between the two Swarm hosts. Docker
+# Swarm's control plane (2377/tcp) is TLS-encrypted by default, but the overlay
+# data plane (VXLAN, 4789/udp) is NOT — and it should never face the public
+# internet. Swarm is joined over this tunnel's private IPs instead.
+#
+# Run on EACH box, in either order. Each run generates that box's own WireGuard
+# keypair (if missing) and prints its public key — paste it into the OTHER
+# box's invocation. Two runs total, one per box:
+#
+#   box A (prod):  bash setup-wireguard.sh 10.10.0.1 10.10.0.2 <PROD_PUBLIC_IP>  <BETA_PUBLIC_IP>  <BETA_WG_PUBKEY_or_empty>
+#   box B (beta):  bash setup-wireguard.sh 10.10.0.2 10.10.0.1 <BETA_PUBLIC_IP>  <PROD_PUBLIC_IP>  <PROD_WG_PUBKEY>
+#
+# First run on either box: leave the last argument empty, note the printed
+# pubkey, then run the second box with that value, then re-run the first box
+# once more with the second box's now-known pubkey. (A local key never
+# changes across re-runs — only the peer section updates.)
+set -euo pipefail
+
+MY_WG_IP="${1:?usage: $0 <my_wg_ip> <peer_wg_ip> <my_public_ip> <peer_public_ip> [peer_wg_pubkey]}"
+PEER_WG_IP="${2:?}"
+MY_PUBLIC_IP="${3:?}"
+PEER_PUBLIC_IP="${4:?}"
+PEER_PUBKEY="${5:-}"
+
+WG_PORT="${WG_PORT:-51820}"
+WG_DIR=/etc/wireguard
+
+if [ "$(id -u)" -ne 0 ]; then
+  echo "Run as root (or via the agent user's sudo)." >&2
+  exit 1
+fi
+
+echo "==> Installing WireGuard if needed"
+command -v wg >/dev/null 2>&1 || { apt-get update -y -q && apt-get install -y -q wireguard; }
+
+install -d -m 700 "$WG_DIR"
+if [ ! -f "$WG_DIR/privatekey" ]; then
+  echo "==> Generating this box's WireGuard keypair"
+  umask 077
+  wg genkey | tee "$WG_DIR/privatekey" | wg pubkey > "$WG_DIR/publickey"
+fi
+MY_PRIVKEY="$(cat "$WG_DIR/privatekey")"
+MY_PUBKEY="$(cat "$WG_DIR/publickey")"
+
+echo
+echo "==> This box's WireGuard public key (give this to the OTHER box's run):"
+echo "    $MY_PUBKEY"
+echo
+
+if [ -z "$PEER_PUBKEY" ]; then
+  echo "No peer public key supplied yet — writing a config with no [Peer] section."
+  echo "Run this same command again once you have it (from the other box's output above)."
+  cat > "$WG_DIR/wg0.conf" <<EOF
+[Interface]
+Address = ${MY_WG_IP}/24
+PrivateKey = ${MY_PRIVKEY}
+ListenPort = ${WG_PORT}
+EOF
+else
+  cat > "$WG_DIR/wg0.conf" <<EOF
+[Interface]
+Address = ${MY_WG_IP}/24
+PrivateKey = ${MY_PRIVKEY}
+ListenPort = ${WG_PORT}
+
+[Peer]
+PublicKey = ${PEER_PUBKEY}
+Endpoint = ${PEER_PUBLIC_IP}:${WG_PORT}
+AllowedIPs = ${PEER_WG_IP}/32
+PersistentKeepalive = 25
+EOF
+fi
+chmod 600 "$WG_DIR/wg0.conf"
+
+echo "==> Bringing up wg0"
+systemctl enable --now wg-quick@wg0 2>/dev/null || (wg-quick down wg0 2>/dev/null; wg-quick up wg0)
+# Re-apply if the config changed on an already-up interface.
+wg syncconf wg0 <(wg-quick strip wg0) 2>/dev/null || true
+
+echo "==> Firewall: only let the OTHER box's public IP reach the WireGuard port"
+if command -v ufw >/dev/null 2>&1; then
+  ufw allow from "$PEER_PUBLIC_IP" to any port "$WG_PORT" proto udp comment "wireguard peer"
+fi
+
+echo
+echo "wg0 status:"
+wg show wg0 || true
+echo
+if [ -n "$PEER_PUBKEY" ]; then
+  echo "==> Verify from here once BOTH boxes have run with each other's pubkey:"
+  echo "    ping -c2 ${PEER_WG_IP}"
+fi