thermograph/INFRA.md

Agent access, Docker Swarm, and Forgejo CI/CD

This covers three additive infrastructure layers on top of the two VPS boxes Terraform already provisions (see terraform/README.md and terraform.tfvars.example, where they're named prod — the new 48 GB / 12-core box, thermograph.org — and beta — the old VPS, 75.119.132.91). None of this touches backend/, Dockerfile, docker-compose*.yml, terraform/, or deploy/db/ — those stay owned by the app-containerization work; this is strictly the layer above it.

Track 1: deploy/provision-agent-access.sh   — a dedicated full-root login for me
Track 2: deploy/swarm/                      — Swarm cluster spanning both boxes
Track 3: deploy/forgejo/ + .forgejo/         — Forgejo + Forgejo Actions, replacing GitHub

Track 1 — Agent access

Run sudo bash deploy/provision-agent-access.sh on both boxes. Creates a dedicated agent user (not raw root login — a distinct name gives a clean audit trail) with passwordless sudo, installs the agent's public key, disables SSH password auth fleet-wide, and turns on auditd logging of every root-effective command. See the script's header comment for the full rationale and revocation steps (one line to delete, or delete the whole account — your own access is never affected).

Current access state (live)

Both boxes are provisioned and reachable as of this writing:

Host	Role	Public IP	Login
prod	manager (Swarm), Thermograph's eventual home	169.58.46.181	agent
beta	worker (Swarm), old VPS	75.119.132.91	agent

ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181   # prod
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91    # beta

The private half of the agent's dedicated keypair lives at ~/.ssh/thermograph_agent_ed25519 on the operator's own machine (never in this repo, never in a CI secret) — same convention DEPLOY.md already uses for the separate CI deploy key (~/.ssh/thermograph_ci). If you're setting this up fresh elsewhere, generate a new pair the same way (ssh-keygen -t ed25519 -a 100 -C "claude-agent@thermograph-infra" -f ~/.ssh/thermograph_agent_ed25519 -N ""), embed the new public half in AGENT_PUBKEY at the top of provision-agent-access.sh, and re-run it on both boxes — that replaces the authorized key wholesale rather than appending, so an old key stops working the moment the new one lands.

Keep the private key somewhere that survives — not a scratch/temp directory that gets cleaned up, not anywhere-in-repo. Losing it just means regenerating and re-running the provisioning script; it does not lock either box, since your own login is untouched.

As of this writing, neither box has Docker installed yet and Terraform has not been applied to either (prod is a bare OS install; beta has an old /opt/thermograph checkout but no Docker) — Track 2 is paused on that.

Track 2 — Swarm

See deploy/swarm/README.md for the exact order of operations (WireGuard tunnel first, then swarm init/join, then lock the Swarm ports down to the tunnel interface, then label beta for Forgejo placement). This cluster's only job is hosting Forgejo — it does not orchestrate the Terraform-managed app deploys.

Track 3 — Forgejo, replacing GitHub

3a. Stand up Forgejo

See deploy/forgejo/README.md — deploys the stack, walks through minting the two Swarm secrets it needs (forgejo_db_password, forgejo_runner_token).

3b. Migrate the repo (mirror first, verify, then cut over)

1. In Forgejo: + New Migration → GitHub. Point it at https://github.com/griffemi/thermograph, pull code + issues + PRs + releases + labels. This is non-destructive to GitHub — do this as a mirror and leave GitHub live and untouched until you're confident.
2. Spot-check: branch list matches, a few PRs/issues render correctly, LFS (if any) came across.
3. Don't retarget any secret or disable a GitHub workflow yet — see 3d.

3c. Workflows

.forgejo/workflows/*.yml already mirror .github/workflows/*.yml in this PR — copied with the mechanical changes Forgejo needs:

• runs-on: ubuntu-latest → runs-on: docker (the label the Swarm-hosted runner in deploy/forgejo/docker-stack.yml registers under).
• appleboy/ssh-action referenced by full GitHub URL (not mirrored in Forgejo's default action registry, unlike actions/checkout / actions/setup-python, which resolve unchanged).
• The custom auto-merge workflow step is gone. It existed only because GitHub's native branch protection/auto-merge is paywalled on private free-tier repos. Forgejo has no such tier — configure it directly: Settings → Branches → protect dev → Enable Status Check → list build → merge style: squash. Then a ready PR gets a real "Auto merge when checks succeed" button.
• One knock-on simplification: a native auto-merge is an ordinary git push (unlike GitHub's token-based merge commit, which doesn't re-trigger push), so deploy-dev.yml's push trigger fires naturally after auto-merge — no more "direct push vs. PR-merge" double-trigger logic to maintain.
• .forgejo/workflows/deploy.yml still targets main faithfully mirroring the current GitHub workflow, but terraform.tfvars.example describes prod's branch as release. That mismatch is flagged in the file's own header comment — reconcile with whoever's driving the Terraform/deploy side rather than guessing here.

3d. Cut over

Only after 3a–3c are verified:

1. Move the deploy secrets (SSH_HOST, SSH_USER, SSH_KEY, SSH_PORT) into Forgejo's repo Secrets (Settings → Actions → Secrets).
2. Re-point the LAN dev runner: bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <token> — stops/disables the old GitHub runner service on that same machine and registers forgejo-runner in its place, same systemd --user pattern as before (see DEPLOY-DEV.md for the original; this supersedes it for the runner specifically).
3. Open a test PR against dev in Forgejo, confirm the required check runs, auto-merge fires, and the LAN deploy lands.
4. Only once that's solid: disable/archive the .github/workflows/*.yml triggers (or the whole GitHub repo) — GitHub was the fallback during this whole migration; retire it deliberately, not by accident.

Verification checklist

• ssh agent@<prod> and ssh agent@<beta> both work; sudo whoami → root
• Password SSH auth confirmed dead on both boxes (tested with an actual failed login attempt, not just config inspection)
• docker node ls (from prod) shows both nodes Ready
• Swarm ports (2377/7946/4789) unreachable from outside the WireGuard tunnel
• https://git.thermograph.org (or your chosen domain) serves Forgejo over TLS
• A test PR: required check runs → auto-merges → LAN dev deploy lands
• GitHub retired only after all of the above hold