Emi Griffith 0d8fc9f4d0 Add agent VPS access, a 2-node Docker Swarm, and Forgejo CI/CD (#234 )

Three additive infrastructure layers on top of the two VPS boxes Terraform
already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old
VPS, 75.119.132.91). None of this touches backend/, Dockerfile,
docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the
app-containerization work in flight elsewhere; this is strictly the layer on
top. See INFRA.md for the full runbook and order of operations.

- deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login
  (not raw root) for agent-driven ops — passwordless sudo under a distinct
  username, sshd hardened to key-only auth, auditd logging every
  root-effective command. One line to revoke.

- deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a
  WireGuard tunnel rather than trusting the public internet with the
  overlay data plane, which Docker's own guidance says should never face it
  directly. Swarm ports locked to the tunnel interface once joined. This
  cluster's only workload is Forgejo — it does not orchestrate the
  Terraform-managed app deploys, so nothing here can strand the app's
  single-writer database.

- deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a
  Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus
  Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom
  auto-merge workflow step is dropped — it existed only to work around
  GitHub's paywalled branch protection on private free-tier repos, which
  Forgejo has no such tier for; native "auto merge when checks succeed"
  replaces it, and as a real git push (unlike GitHub's non-triggering
  token-merge) it fires the LAN deploy naturally with no double-trigger
  logic needed. appleboy/ssh-action is referenced by full URL (not mirrored
  on Forgejo's default action registry); actions/checkout and
  actions/setup-python resolve unchanged.

Migration is mirror-first: the GitHub repo import and workflow files land
here, but cutting deploy secrets over and retiring GitHub happens only after
verification (INFRA.md 3d) — GitHub stays live as a fallback throughout.

One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but
terraform.tfvars.example names prod's deploy branch `release`. Left as a
faithful mirror rather than guessed at — reconcile with whoever's driving
Terraform/deploy.

2026-07-21 00:36:39 +00:00

5.2 KiB

Raw Blame History

Agent access, Docker Swarm, and Forgejo CI/CD

This covers three additive infrastructure layers on top of the two VPS boxes Terraform already provisions (see terraform/README.md and terraform.tfvars.example, where they're named prod — the new 48 GB / 12-core box, thermograph.org — and beta — the old VPS, 75.119.132.91). None of this touches backend/, Dockerfile, docker-compose*.yml, terraform/, or deploy/db/ — those stay owned by the app-containerization work; this is strictly the layer above it.

Track 1: deploy/provision-agent-access.sh   — a dedicated full-root login for me
Track 2: deploy/swarm/                      — Swarm cluster spanning both boxes
Track 3: deploy/forgejo/ + .forgejo/         — Forgejo + Forgejo Actions, replacing GitHub

Track 1 — Agent access

Run sudo bash deploy/provision-agent-access.sh on both boxes. Creates a dedicated agent user (not raw root login — a distinct name gives a clean audit trail) with passwordless sudo, installs the agent's public key, disables SSH password auth fleet-wide, and turns on auditd logging of every root-effective command. See the script's header comment for the full rationale and revocation steps (one line to delete, or delete the whole account — your own access is never affected).

Track 2 — Swarm

See deploy/swarm/README.md for the exact order of operations (WireGuard tunnel first, then swarm init/join, then lock the Swarm ports down to the tunnel interface, then label beta for Forgejo placement). This cluster's only job is hosting Forgejo — it does not orchestrate the Terraform-managed app deploys.

Track 3 — Forgejo, replacing GitHub

3a. Stand up Forgejo

See deploy/forgejo/README.md — deploys the stack, walks through minting the two Swarm secrets it needs (forgejo_db_password, forgejo_runner_token).

3b. Migrate the repo (mirror first, verify, then cut over)

In Forgejo: + New Migration → GitHub. Point it at https://github.com/griffemi/thermograph, pull code + issues + PRs + releases + labels. This is non-destructive to GitHub — do this as a mirror and leave GitHub live and untouched until you're confident.
Spot-check: branch list matches, a few PRs/issues render correctly, LFS (if any) came across.
Don't retarget any secret or disable a GitHub workflow yet — see 3d.

3c. Workflows

.forgejo/workflows/*.yml already mirror .github/workflows/*.yml in this PR — copied with the mechanical changes Forgejo needs:

runs-on: ubuntu-latest → runs-on: docker (the label the Swarm-hosted runner in deploy/forgejo/docker-stack.yml registers under).
appleboy/ssh-action referenced by full GitHub URL (not mirrored in Forgejo's default action registry, unlike actions/checkout / actions/setup-python, which resolve unchanged).
The custom auto-merge workflow step is gone. It existed only because GitHub's native branch protection/auto-merge is paywalled on private free-tier repos. Forgejo has no such tier — configure it directly: Settings → Branches → protect dev → Enable Status Check → list build → merge style: squash. Then a ready PR gets a real "Auto merge when checks succeed" button.
One knock-on simplification: a native auto-merge is an ordinary git push (unlike GitHub's token-based merge commit, which doesn't re-trigger push), so deploy-dev.yml's push trigger fires naturally after auto-merge — no more "direct push vs. PR-merge" double-trigger logic to maintain.
.forgejo/workflows/deploy.yml still targets main faithfully mirroring the current GitHub workflow, but terraform.tfvars.example describes prod's branch as release. That mismatch is flagged in the file's own header comment — reconcile with whoever's driving the Terraform/deploy side rather than guessing here.

3d. Cut over

Only after 3a–3c are verified:

Move the deploy secrets (SSH_HOST, SSH_USER, SSH_KEY, SSH_PORT) into Forgejo's repo Secrets (Settings → Actions → Secrets).
Re-point the LAN dev runner: bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <token> — stops/disables the old GitHub runner service on that same machine and registers forgejo-runner in its place, same systemd --user pattern as before (see DEPLOY-DEV.md for the original; this supersedes it for the runner specifically).
Open a test PR against dev in Forgejo, confirm the required check runs, auto-merge fires, and the LAN deploy lands.
Only once that's solid: disable/archive the .github/workflows/*.yml triggers (or the whole GitHub repo) — GitHub was the fallback during this whole migration; retire it deliberately, not by accident.

Verification checklist

ssh agent@<prod> and ssh agent@<beta> both work; sudo whoami → root
Password SSH auth confirmed dead on both boxes
docker node ls (from prod) shows both nodes Ready
Swarm ports (2377/7946/4789) unreachable from outside the WireGuard tunnel
https://git.thermograph.org (or your chosen domain) serves Forgejo over TLS
A test PR: required check runs → auto-merges → LAN dev deploy lands
GitHub retired only after all of the above hold

5.2 KiB Raw Blame History Unescape Escape