Emi Griffith 0d8fc9f4d0 Add agent VPS access, a 2-node Docker Swarm, and Forgejo CI/CD (#234 )

Three additive infrastructure layers on top of the two VPS boxes Terraform
already provisions (prod: new 48 GB/12-core box, thermograph.org; beta: old
VPS, 75.119.132.91). None of this touches backend/, Dockerfile,
docker-compose*.yml, terraform/, or deploy/db/ — that stays owned by the
app-containerization work in flight elsewhere; this is strictly the layer on
top. See INFRA.md for the full runbook and order of operations.

- deploy/provision-agent-access.sh: a dedicated, auditable full-sudo login
  (not raw root) for agent-driven ops — passwordless sudo under a distinct
  username, sshd hardened to key-only auth, auditd logging every
  root-effective command. One line to revoke.

- deploy/swarm/: a 2-node Swarm (prod=manager, beta=worker) joined over a
  WireGuard tunnel rather than trusting the public internet with the
  overlay data plane, which Docker's own guidance says should never face it
  directly. Swarm ports locked to the tunnel interface once joined. This
  cluster's only workload is Forgejo — it does not orchestrate the
  Terraform-managed app deploys, so nothing here can strand the app's
  single-writer database.

- deploy/forgejo/ + .forgejo/workflows/: Forgejo + Traefik + a
  Docker-in-Docker-sandboxed runner as a Swarm stack pinned to beta, plus
  Forgejo Actions workflows mirroring .github/workflows/*.yml. The custom
  auto-merge workflow step is dropped — it existed only to work around
  GitHub's paywalled branch protection on private free-tier repos, which
  Forgejo has no such tier for; native "auto merge when checks succeed"
  replaces it, and as a real git push (unlike GitHub's non-triggering
  token-merge) it fires the LAN deploy naturally with no double-trigger
  logic needed. appleboy/ssh-action is referenced by full URL (not mirrored
  on Forgejo's default action registry); actions/checkout and
  actions/setup-python resolve unchanged.

Migration is mirror-first: the GitHub repo import and workflow files land
here, but cutting deploy secrets over and retiring GitHub happens only after
verification (INFRA.md 3d) — GitHub stays live as a fallback throughout.

One flagged, unresolved mismatch: deploy.yml still triggers on `main`, but
terraform.tfvars.example names prod's deploy branch `release`. Left as a
faithful mirror rather than guessed at — reconcile with whoever's driving
Terraform/deploy.

2026-07-21 00:36:39 +00:00

2.9 KiB

Raw Blame History

Forgejo on the Swarm cluster

Runs as deploy/forgejo/docker-stack.yml — the only workload this Swarm cluster carries (the Thermograph app itself stays on the Terraform-managed docker compose deploys; see terraform/README.md). Pinned to the beta node (old VPS) via the role=forge label from deploy/swarm/label-forge-node.sh.

Prerequisites

Both boxes have joined the swarm (deploy/swarm/) and beta is labeled role=forge.
docker node ls (from the manager) shows both Ready.

One-time setup: Swarm secrets

Two secrets the stack expects to already exist (Swarm secrets, not files — external: true in the stack file, so docker stack deploy never creates or sees the values, only references them):

# A strong random password for Forgejo's own Postgres (NOT related to
# Thermograph's app database — entirely separate instance/network).
openssl rand -base64 32 | docker secret create forgejo_db_password -

# The runner registration token. Forgejo can't issue one before it's running,
# so this is a two-step dance the first time:
docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo   # 1. bring Forgejo up (runner will crashloop briefly — expected)
# 2. once Forgejo answers at the domain, log in, go to
#    Site Administration -> Actions -> Runners -> Create new Runner
#    (or, for a repo-scoped runner: <repo> -> Settings -> Actions -> Runners),
#    copy the token, then:
echo -n "PASTE_TOKEN_HERE" | docker secret create forgejo_runner_token -
docker service update --force forgejo_runner   # picks up the new secret and registers

Deploy / update

docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo

Re-running is safe — Swarm only touches services whose spec actually changed.

DNS

Point the Forgejo domain (default git.thermograph.org; override with FORGEJO_DOMAIN=... before docker stack deploy, Swarm reads it from the deploying shell's environment) at either node's public IP — the routing mesh forwards published ports to wherever the task actually landed.

Why Postgres here and not the Thermograph app's TimescaleDB

Separate instance, separate network (forgejo_net, not the app's compose network), separate volume. Forgejo is a distinct product with its own schema and its own backup/restore lifecycle — sharing a database with the app would couple two things that should be able to fail, migrate, and restore independently.

Verifying

docker service ls                       # all forgejo_* services Running, 1/1
curl -I https://git.thermograph.org/    # 200, valid cert
docker service logs forgejo_runner --tail 50   # "runner: successfully registered" then idle

Rollback / removal

docker stack rm forgejo
# volumes (forgejo_data, forgejo_db, ...) survive a stack rm — remove them
# explicitly only if you actually want to destroy the Forgejo instance's data.

2.9 KiB Raw Blame History