e5137e777d
Terraform generates the secrets that have no external meaning
(POSTGRES_PASSWORD, AUTH_SECRET, METRICS_TOKEN, INDEXNOW_KEY) via the random
provider instead of requiring the operator to hand-generate and paste each
into terraform.tfvars. Each is pinned with a static keepers value (secrets.tf)
so apply never regenerates a value already in use - the exact incident class
this guards against: every session invalidated, the app<->DB password
mismatched. Rotation is now a deliberate keepers edit, never a side effect.
postgres_password/auth_secret move from required inputs to optional (default
"") - explicit var wins when supplied (seeding an EXISTING live secret during
a migration onto Terraform, hop-1 cutover runbook Stage 0), else Terraform
generates and owns it. metrics_token/indexnow_key are new: neither existed in
Terraform before, both previously left for the app's own fallback generation.
VAPID deliberately stays a required, non-generated input - an EC keypair
where regeneration breaks every existing push subscription outright, unlike
an opaque token.
Sizing tiers: a locals.sizes t-shirt map (nano/small/medium/large ->
{workers, app_cpus, db_cpus, db_memory}), toward the target Proxmox
sizing-tier model (architecture doc SS6) ahead of actually provisioning VMs -
Proxmox itself stays deferred; today a tier just sizes container caps on the
existing SSH-managed hosts. A host can reference one by name (hosts.<name>.
size) or keep hand-picking the four fields, so existing tfvars are
unaffected; prod's example now uses size = "large" (identical numbers),
beta keeps explicit numbers, and a commented uat example demonstrates the
shortcut for a future ephemeral host.
Strengthened terraform/README.md's local-state caveat: more Terraform-
generated secrets landing in tfstate raises the stakes of the existing
never-commit-cleartext-state guidance, not just the sizing.
Verified: terraform validate + fmt clean. A real `terraform plan` against
fake hosts (prod/beta/uat, mixing size="large"/explicit-numbers/size="nano")
resolved every sizing correctly (prod 8/8/4/16g, beta 4/4/2/8g, uat
1/1/1/1g) and planned exactly one instance of each random_password/random_id
resource. Applied just those four resources (real generation, -target to
avoid touching the fake SSH-only host resources) and re-planned: "No
changes" - confirming the keepers pinning holds. Adding an explicit
postgres_password override afterward left the random_password resource
itself completely untouched (0 replace/destroy), confirming the override
path never disturbs the generated resource.
120 lines
6.5 KiB
Markdown
120 lines
6.5 KiB
Markdown
# Thermograph — Terraform (host provisioning)
|
|
|
|
Terraform that **provisions and configures the existing VPS hosts** and hands the app
|
|
off to `docker compose`. It does *not* create servers (no cloud provider) and does not
|
|
replace compose — it prepares each host (Docker, firewall, checkout, secrets, Caddy)
|
|
and runs `docker compose up`.
|
|
|
|
## What it manages
|
|
|
|
One reusable module (`modules/thermograph-host`) is instantiated per host via
|
|
`for_each`. This config manages **two VPS hosts**:
|
|
|
|
| key | role | VPS | branch | domain | notes |
|
|
|--------|--------|-------------------------|-----------|-------------------|------------------------------------|
|
|
| `prod` | prod | NEW 48 GB / 12-core box | `release` | `thermograph.org` | Caddy TLS; sized up (8/4/16g) |
|
|
| `beta` | beta | old box `75.119.132.91` | `main` | *(none)* | testing/beta; no public domain yet |
|
|
|
|
The `dev` branch is **out of scope here** — it deploys to the LAN dev server via
|
|
`deploy/deploy-dev.sh` (a self-hosted GitHub Actions runner), not Terraform.
|
|
|
|
Per host, over SSH provisioners, Terraform:
|
|
|
|
- installs Docker + the compose plugin if missing;
|
|
- configures a `ufw` firewall (22/80/443 always; on a host with **no** domain it also
|
|
opens the app port `8137`);
|
|
- ensures the git checkout at `app_dir` exists (clones on a fresh box) and resets it to
|
|
the host's branch;
|
|
- renders `/etc/thermograph.env` from Terraform variables (secrets injected from tfvars,
|
|
pushed via provisioner `content` so they never touch local disk) and installs it
|
|
root-owned `0640`;
|
|
- for a host **with** a domain, installs a rendered Caddyfile and reloads Caddy;
|
|
- brings the stack up: `docker compose <-f each compose file> up -d --build`, running
|
|
docker as root with `/etc/thermograph.env` sourced in the same shell;
|
|
- health-checks `http://127.0.0.1:8137/`.
|
|
|
|
A change to the rendered env, the compose files, the branch, or the sizing flips the
|
|
`null_resource` trigger and re-runs the provisioners on next apply.
|
|
|
|
### Container sizing is env-driven
|
|
|
|
`docker-compose.yml` reads `WORKERS`, `APP_CPUS`, `DB_CPUS`, and `DB_MEMORY` from the
|
|
environment (defaults `4 / 4 / 2 / 8g`, identical to before). Terraform sets them per
|
|
host through `/etc/thermograph.env`, so the big prod box can run larger caps without a
|
|
compose edit. The Postgres *internal* memory budget (`shared_buffers`,
|
|
`effective_cache_size`, `work_mem`, `maintenance_work_mem`) is derived from the same
|
|
`DB_MEMORY` by `deploy/db/init/20-tuning.sh` — so raising `db_memory` scales the
|
|
container cap and the tuning together (prod 16g → shared_buffers 4 GB). The tuning
|
|
applies on a fresh DB volume; on an existing volume re-run it by hand (see the script
|
|
header).
|
|
|
|
## Prerequisites
|
|
|
|
- Terraform >= 1.6 (v1.15 is installed).
|
|
- SSH key access to **both** hosts as a **sudo-capable** user (default `deploy`). Point
|
|
`ssh_private_key_path` at that key (`~` is expanded).
|
|
- The hosts are Debian/Ubuntu with `apt` and outbound internet (Docker/Caddy installs
|
|
pull from the network). Docker may already be present — installs are conditional.
|
|
- For the `prod` host: DNS for `thermograph.org` must point at the new box before apply,
|
|
or Caddy's first-request cert issuance will fail.
|
|
|
|
## Use
|
|
|
|
```sh
|
|
cd terraform
|
|
cp terraform.tfvars.example terraform.tfvars # then edit: real IPs + secrets
|
|
terraform init
|
|
terraform plan
|
|
terraform apply
|
|
```
|
|
|
|
Target one host with `-target='module.host["beta"]'` if you want to apply to just one.
|
|
|
|
## Local state + secrets caveat
|
|
|
|
The backend is **local**: `terraform.tfstate` is written next to the config and holds
|
|
every secret in cleartext (the rendered env, VAPID keys, DB password, …). It is
|
|
gitignored (`terraform/.gitignore` and the root `.gitignore`). Keep it off shared disks
|
|
and back it up somewhere private. `terraform.tfvars` is likewise gitignored; only
|
|
`terraform.tfvars.example` (dummy values) is committed. `.terraform.lock.hcl` **is**
|
|
committed so provider versions are pinned across machines.
|
|
|
|
**Terraform now originates, not just carries, some of these secrets**
|
|
(`postgres_password`/`auth_secret`/`metrics_token`/`indexnow_key` — see `secrets.tf`):
|
|
left unset, each is generated by the `random` provider and its value lives *only* in
|
|
`terraform.tfstate` — there is no other copy until it's rendered to a host's
|
|
`/etc/thermograph.env`. More generated secrets landing in state makes the local-state
|
|
posture above matter more, not less: **never commit or sync `terraform.tfstate` in
|
|
cleartext**, and before this config is used from a shared machine, CI, or a remote
|
|
backend, stand up an encrypted backend (or SOPS-wrap the state) first — see the hop-1
|
|
cutover runbook's hazard #15 and Track B step 7. Until then, treat `terraform.tfstate`
|
|
with the same care as the secrets it contains.
|
|
|
|
## WARNING — applying against live prod
|
|
|
|
`terraform apply` runs `remote-exec` **on the server**: it resets the checkout to the
|
|
branch, rewrites `/etc/thermograph.env`, and runs `docker compose up -d --build`
|
|
(rebuilding images and recreating containers — a brief app restart). Against the live
|
|
production host this is a real deploy. Review the plan, apply in a maintenance window,
|
|
and prefer `-target` to touch one host at a time.
|
|
|
|
This is **separate from the Postgres data cutover** in `deploy/POSTGRES-MIGRATION.md`.
|
|
Terraform provisions the host and starts the stack; it does **not** migrate the
|
|
SQLite→Postgres accounts data. Sequence them deliberately: for a first cutover on a
|
|
host, follow the migration doc's freeze/backup/copy steps around the point where
|
|
Terraform brings the stack up — don't let Terraform recreate containers mid-migration.
|
|
|
|
## Assumptions / notes
|
|
|
|
- **beta has no public domain by default.** With `compose_files = ["docker-compose.yml"]`
|
|
the app binds `127.0.0.1:8137` (loopback), so opening the port in `ufw` alone does not
|
|
expose it. Reach beta via an SSH tunnel, or set `domain = "beta.thermograph.org"` (adds
|
|
Caddy TLS) — or add the `0.0.0.0`-publishing dev overlay to `compose_files` — to make
|
|
it reachable. `COOKIE_SECURE` is auto-set to `0` when there's no domain (a Secure
|
|
cookie is never sent over plain HTTP) and `1` behind Caddy TLS.
|
|
- The rendered Caddyfile only reverse-proxies the app. The repo's `deploy/Caddyfile`
|
|
additionally serves the `emigriffith.dev` portfolio and legacy redirects; those are
|
|
host-specific and not templated here.
|
|
- Provisioner-based by design: the hosts already exist, so this is not a
|
|
create-from-scratch cloud config. Re-applying is idempotent (installs are guarded,
|
|
`git reset --hard`, compose `up` reconciles).
|