thermograph/infra/terraform/README.md

Thermograph — Terraform (host provisioning)

Terraform that provisions and configures VPS hosts and hands the app off to docker compose, which pulls the published app image (built + pushed by the app repo's CI) and runs it — no app source is ever built or checked out on a host. By default it does not create servers: modules/thermograph-host is SSH-provisioner-driven against a host that already exists. An optional modules/gcp-host can additionally create the VM on GCP first (see "GCP scaffold" below) — scaffold-only today, no live resources until you opt in.

What it manages

One reusable module (modules/thermograph-host) is instantiated per (host, environment) pair via for_each (main.tf's local.all_hosts, merging var.hosts — SSH-managed, already-existing boxes — with any var.gcp_hosts Terraform created itself). var.hosts is keyed by HOST, not by environment, because vps2 alone runs two environments (prod and beta) as separate Docker Swarm stacks on the same box — see the hosts variable's file header in variables.tf for why that stopped being "one entry = one environment" and what still needs to change in main.tf to consume the new shape (flagged there as a TODO(cutover)). This config manages two VPS hosts, three environments, today:

host	environment	VPS	branch	domain	notes
vps1	dev	75.119.132.91	dev (of the APP repos — see backend_image_tag / frontend_image_tag)	none (mesh-only)	Also hosts Forgejo, Grafana/Loki/Alloy, emigriffith.dev — not Terraform-managed
vps2	prod	48 GB / 12-core box (169.58.46.181)	main	thermograph.org	Caddy TLS (this module owns /etc/caddy/Caddyfile); sized up (8/8/4/16g)
vps2	beta	the SAME box as prod (169.58.46.181)	main	none (see below)	Separate Swarm stack + checkout, shares prod's TimescaleDB instance on a separate database/role

Only ONE environment per host may set a domain: the module installs a full /etc/caddy/Caddyfile, and a second terraform apply with its own domain set would silently overwrite the first's site instead of adding to it (there is no merge). Prod claims that slot on vps2 today, so beta's public reverse-proxy (if and when beta.thermograph.org is exposed) has to be a site block in vps2's shared, hand-maintained Caddy config instead — the same pattern deploy/forgejo/docker-stack.yml already uses to put git.thermograph.org in front of Forgejo's loopback port on vps1, alongside that box's own Caddy-fronted emigriffith.dev.

Each environment's own checkout on disk (app_dir, git_branch) is this infra repo, not the app repo — the "branch" column above is which app-repo tag an environment is meant to track conceptually; the actual pinned versions are var.hosts[*].environments[*].backend_image_tag / frontend_image_tag (e.g. "sha-<12 hex>" each — the app is two separately-published images, emi/thermograph-backend/app and emi/thermograph-frontend/app), since the host has no app checkout to derive a tag from. app_dir is required with no default specifically so vps2's two environments can never collide on the same checkout path (prod: /opt/thermograph, beta: /opt/thermograph-beta). The LAN-laptop dev rehearsal (make dev-up) is out of scope here — it's a local-only compose overlay, not this Terraform-managed dev environment.

Per (host, environment), over SSH provisioners, Terraform:

• installs Docker + the compose plugin if missing;
• configures a ufw firewall (22/80/443 always; on a host with no domain it also opens the app port 8137);
• ensures the git checkout at app_dir exists (clones this repo on a fresh box) and resets it to the host's git_branch;
• renders /etc/thermograph-topology.env from Terraform variables (non-secret sizing only — WORKERS/APP_CPUS/DB_CPUS/DB_MEMORY/etc.), then runs deploy/render-secrets.sh (from that same freshly-synced checkout) to render /etc/thermograph.env from the SOPS+age vault — Terraform itself never sees or carries an app secret (see "Secrets" below);
• for a host with a domain, installs a rendered Caddyfile and reloads Caddy;
• brings the stack up on the explicit per-service tags (exports BACKEND_IMAGE_TAG / FRONTEND_IMAGE_TAG for the compose file): docker login to the registry, docker compose <-f each compose file> pull backend frontend, then ... up -d --remove-orphans, running docker as root with both env files sourced;
• health-checks http://127.0.0.1:8137/.

A change to the rendered topology env, the compose files, the branch, the app image tag, or the sizing flips the null_resource trigger and re-runs the provisioners on next apply.

GCP scaffold (no live resources)

modules/gcp-host can create a GCE VM (+ a dedicated VPC, subnet, and a firewall opening 22/80/443) instead of assuming the host already exists — see main.tf's module.gcp_vm. It contributes nothing to provisioning: its only output (external_ip) feeds into the exact same thermograph-host module every SSH-managed host uses, so there's one provisioning path regardless of how a host came to exist. var.gcp_hosts defaults to {}, so by default no google_* resource is ever planned and the google provider is never invoked — terraform plan/validate succeed with no GCP credentials configured at all. To actually use it: populate an entry in terraform.tfvars (see the commented example in terraform.tfvars.example) and authenticate via gcloud auth application-default login or GOOGLE_APPLICATION_CREDENTIALS. This is the same create-then-provision composition a future Proxmox module (the architecture doc's longer-term target — see §6/§9 there) would use.

Container sizing is env-driven

docker-compose.yml reads WORKERS, APP_CPUS, DB_CPUS, and DB_MEMORY from the environment (defaults 4 / 4 / 2 / 8g, identical to before). Terraform sets them per host through /etc/thermograph-topology.env, so the big prod box can run larger caps without a compose edit. The Postgres internal memory budget (shared_buffers, effective_cache_size, work_mem, maintenance_work_mem) is derived from the same DB_MEMORY by deploy/db/init/20-tuning.sh — so raising db_memory scales the container cap and the tuning together (prod 16g → shared_buffers 4 GB). The tuning applies on a fresh DB volume; on an existing volume re-run it by hand (see the script header).

Prerequisites

• Terraform >= 1.6 (v1.15 is installed).
• SSH key access to both hosts as a sudo-capable user (the live boxes use the dedicated agent account, ~/.ssh/thermograph_agent_ed25519 — see deploy/provision-agent-access.sh). Point ssh_private_key_path at that key (~ is expanded).
• The hosts are Debian/Ubuntu with apt and outbound internet (Docker/Caddy installs pull from the network). Docker may already be present — installs are conditional.
• For the prod host: DNS for thermograph.org must point at the new box before apply, or Caddy's first-request cert issuance will fail.

Use

cd terraform
cp terraform.tfvars.example terraform.tfvars   # then edit: real IPs + secrets
terraform init
terraform plan
terraform apply

Target one host with -target='module.host["prod"]' if you want to apply to just one (the module's for_each key changes once main.tf is updated to flatten var.hosts's new per-host environments map — see the TODO(cutover) on the hosts variable in variables.tf — at which point the key for beta becomes something like module.host["vps2-beta"], not module.host["beta"]).

Secrets

App secrets (POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET, VAPID keys, REGISTRY_TOKEN, Discord/SMTP credentials, …) are not Terraform variables — they live in the git-native SOPS+age vault at ../deploy/secrets/*.yaml (committed, encrypted) and are rendered into /etc/thermograph.env at deploy time by ../deploy/render-secrets.sh, which the provisioner's deploy step runs from the freshly-synced checkout. Terraform only renders the non-secret /etc/thermograph-topology.env (sizing/routing). See ../deploy/secrets/README.md to rotate a secret or add a new one — it's a sops edit + commit + deploy, no Terraform apply involved.

om_rclone_conf (object-storage bucket credentials for the self-hosted ERA5 archive) is the one exception still supplied via Terraform, in terraform.tfvars — folding it into the vault too is a reasonable future step, not done here.

Local state

The backend is local: terraform.tfstate is written next to the config. It's gitignored (terraform/.gitignore and the root .gitignore) — keep it off shared disks and back it up somewhere private. terraform.tfvars is likewise gitignored (it may carry a repo_url credential and always carries om_rclone_conf); only terraform.tfvars.example (dummy values) is committed. .terraform.lock.hcl is committed so provider versions are pinned across machines.

WARNING — applying against live prod

terraform apply runs remote-exec on the server: it resets the checkout to the branch, renders topology config + secrets, and runs docker compose pull && up -d (pulling the pinned backend_image_tag / frontend_image_tag and recreating containers — a brief app restart). Against the live production host this is a real deploy. Review the plan, apply in a maintenance window, and prefer -target to touch one host at a time.

This is separate from the Postgres data cutover in deploy/POSTGRES-MIGRATION.md. Terraform provisions the host and starts the stack; it does not migrate the SQLite→Postgres accounts data. Sequence them deliberately: for a first cutover on a host, follow the migration doc's freeze/backup/copy steps around the point where Terraform brings the stack up — don't let Terraform recreate containers mid-migration.

Assumptions / notes

• beta has no public domain here, and can't without a different Caddy strategy. With compose_files = ["docker-compose.yml"] the app binds 127.0.0.1:8137 (loopback), so opening the port in ufw alone does not expose it — and setting domain on beta the way prod does would make this module overwrite the SAME /etc/caddy/Caddyfile prod's apply just installed (see the table above). Reach beta via an SSH tunnel, or front it with a site block in vps2's own hand-maintained Caddy config (outside Terraform), the same way Forgejo's git.thermograph.org reaches Forgejo's loopback port on vps1. COOKIE_SECURE is auto-set to 0 when there's no domain (a Secure cookie is never sent over plain HTTP) and 1 behind Caddy TLS.
• The rendered Caddyfile only reverse-proxies the app. The repo's deploy/Caddyfile additionally serves the emigriffith.dev portfolio and legacy redirects; those are host-specific and not templated here.
• Provisioner-based by design: the hosts already exist, so this is not a create-from-scratch cloud config. Re-applying is idempotent (installs are guarded, git reset --hard, compose up reconciles).