Jinemi/thermograph
4
0
Fork 0
Code Issues 2 Pull requests 9 Projects Releases Packages Wiki Activity Actions 1
thermograph/INFRA.md

8.9 KiB
Raw Blame History

Agent access, Docker Swarm, and Forgejo CI/CD

This covers three additive infrastructure layers on top of the app containerization / Postgres / Terraform work described in thermograph-docs/runbooks/implementation-handoff.md (read that first — it's the canonical Track A/Track B plan this whole effort implements) and thermograph-docs/architecture/repo-topology-and-infrastructure.md. Terraform provisions prod (the new 48 GB / 12-core box, thermograph.org) and beta (the old VPS, 75.119.132.91) — see terraform/README.md / terraform.tfvars.example. None of this touches backend/, Dockerfile, docker-compose*.yml, terraform/, or deploy/db/ — those stay owned by the app-containerization work; this is strictly the layer above it.

Track 1: deploy/provision-agent-access.sh   — a dedicated full-root login for me
Track 2: deploy/swarm/                      — Swarm cluster spanning THREE nodes
Track 3: deploy/forgejo/ + .forgejo/         — Forgejo + Forgejo Actions, replacing GitHub

Three nodes, not two: prod, beta, and the desktop (the LAN dev machine — same box that already runs the pre-Forgejo GitHub self-hosted runner). This matches the canonical Track B design in the handoff doc; an earlier revision of this whole effort covered just prod+beta and has been realigned.

Track 1 — Agent access

Run sudo bash deploy/provision-agent-access.sh on prod and beta (not the desktop — that's wherever you're already working from, no separate access-provisioning step needed there). Creates a dedicated agent user (not raw root login — a distinct name gives a clean audit trail) with passwordless sudo, installs the agent's public key, disables SSH password auth fleet-wide, and turns on auditd logging of every root-effective command. See the script's header comment for the full rationale and revocation steps (one line to delete, or delete the whole account — your own access is never affected).

Current access state (live)

Both VPS boxes are provisioned and reachable as of this writing:

Host Role Public IP Login
prod Swarm manager; Thermograph's live prod home (release, thermograph.org) 169.58.46.181 agent
beta Swarm worker; main/beta.thermograph.org + hosts Forgejo 75.119.132.91 agent
desktop Swarm worker, LAN dev machine + Forgejo Actions runner (local) (already have access) 
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181   # prod
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91    # beta

The private half of the agent's dedicated keypair lives at ~/.ssh/thermograph_agent_ed25519 on the operator's own machine (never in this repo, never in a CI secret) — same convention DEPLOY.md already uses for the separate CI deploy key (~/.ssh/thermograph_ci). If you're setting this up fresh elsewhere, generate a new pair the same way (ssh-keygen -t ed25519 -a 100 -C "claude-agent@thermograph-infra" -f ~/.ssh/thermograph_agent_ed25519 -N ""), embed the new public half in AGENT_PUBKEY at the top of provision-agent-access.sh, and re-run it on both boxes — that replaces the authorized key wholesale rather than appending, so an old key stops working the moment the new one lands.

Keep the private key somewhere that survives — not a scratch/temp directory that gets cleaned up, not anywhere-in-repo. Losing it just means regenerating and re-running the provisioning script; it does not lock either box, since your own login is untouched.

Live as of 2026-07-21 (was: "no Docker / Terraform not applied / Swarm inactive" — all now done): Docker is installed on all three nodes; the WireGuard mesh and Docker Swarm are up with all three nodes Ready (prod manager, beta + desktop workers); Terraform has been applied to both VPS boxes (prod stood up fresh on release, beta rebuilt on main); Forgejo is serving at git.thermograph.org; and the desktop runs the Forgejo Actions runner. See the verification checklist at the end.

Track 2 — Swarm

See deploy/swarm/README.md for the exact order of operations across all three nodes (WireGuard mesh first, then swarm init/join, then lock the Swarm ports down to the tunnel interface, then label beta for Forgejo placement). This cluster's only job is hosting Forgejo — it does not orchestrate the Terraform-managed app deploys.

Track 3 — Forgejo, replacing GitHub

3a. Stand up Forgejo

See deploy/forgejo/README.md — deploys the stack (pinned to beta), then walks through registering the Actions runner on the desktop as a plain systemd service (register-lan-runner.sh), not as a Swarm-scheduled container. Only one Swarm secret to mint now (forgejo_db_password) — the runner token is no longer a Swarm secret, since the runner isn't a Swarm service anymore.

3b. Migrate the repo (mirror first, verify, then cut over)

  1. In Forgejo: + New Migration → GitHub. Point it at https://github.com/griffemi/thermograph, pull code + issues + PRs + releases + labels. This is non-destructive to GitHub — do this as a mirror and leave GitHub live and untouched until you're confident.
  2. Spot-check: branch list matches, a few PRs/issues render correctly, LFS (if any) came across.
  3. Don't retarget any secret or disable a GitHub workflow yet — see 3d.

3c. Workflows

.forgejo/workflows/{build,pr-build,deploy-dev,deploy}.yml mirror .github/workflows/*.yml, copied with the mechanical changes Forgejo needs:

  • runs-on: ubuntu-latest → runs-on: docker (one of the two labels the desktop's runner registers under — see 3a; general CI/deploy jobs get a fresh container, the LAN-deploy job in deploy-dev.yml runs bare/host-native under the thermograph-lan label instead, since it needs real filesystem access).
  • appleboy/ssh-action referenced by full GitHub URL (not mirrored in Forgejo's default action registry, unlike actions/checkout / actions/setup-python, which resolve unchanged).
  • The custom auto-merge workflow step is gone. It existed only because GitHub's native branch protection/auto-merge is paywalled on private free-tier repos. Forgejo has no such tier: dev is protected with the build status check required (Settings → Branches). Forgejo does not auto-merge on green by itself — a ready, green, mergeable PR is merged explicitly (POST .../pulls/{n}/merge, {"Do": "squash"}), per CLAUDE.md. That merge is an ordinary push, so deploy-dev.yml's push trigger fires naturally afterward.
  • Branch/deploy mapping (settled): .forgejo/workflows/deploy.yml ("Deploy to beta VPS") triggers on main and SSHes to beta (75.119.132.91) running deploy/deploy.sh. Prod is not deployed by a push-triggered workflow — it's deployed with terraform apply (terraform/README.md) on the release branch. So the promotion chain is dev → main (this workflow deploys beta) → release (Terraform deploys prod). There is deliberately no release-triggered workflow.

A second, independent set of Forgejo workflows (.forgejo/workflows/{ci,build-push,ops-cron}.yml) was added by a parallel effort implementing thermograph-docs/runbooks/implementation-handoff.md Track A chunk 7 — see that doc and its own reconciliation PR (which already fixed its runner-label and ssh-action guesses to match the real infrastructure here). ci.yml duplicated this PR's build.yml exactly and was dropped in that reconciliation; build-push.yml (image → Forgejo's built-in registry) and ops-cron.yml (backup + IndexNow) are novel and still present.

3d. Cut over — DONE (2026-07-21)

The GitHub → Forgejo cutover is complete; this records what was done:

  1. Deploy secrets (SSH_HOST, SSH_USER, SSH_KEY, SSH_PORT) live in Forgejo's repo Secrets, pointing at beta with a dedicated CI deploy key.
  2. The LAN dev runner was re-pointed to Forgejo via deploy/forgejo/register-lan-runner.sh (systemd --user, on the desktop).
  3. The repo was migrated into Forgejo; dev/main/release all promoted through Forgejo and deploys verified live.
  4. GitHub is retired as git host and CI (PR "Retire GitHub as the git host and CI platform") — origin remains only as a read-only mirror of history.

Verification checklist — all met (2026-07-21)

  • ssh agent@<prod> and ssh agent@<beta> both work; sudo whoami → root
  • Password SSH auth confirmed dead on both boxes (tested with an actual failed login attempt, not just config inspection)
  • docker node ls (from prod) shows all three nodes Ready
  • Swarm ports (2377/7946/4789) unreachable from outside the WireGuard tunnel
  • https://git.thermograph.org serves Forgejo over TLS; /v2/ registry API is 403 from off-mesh (firewalled to the WireGuard CIDR)
  • A test PR flow verified: required build check runs → explicit squash merge → LAN dev deploy lands
  • GitHub retired as git host + CI; kept only as a read-only history mirror