Have Terraform generate its own internal secrets, with sizing tiers (#239)

Terraform generates the secrets that have no external meaning
(POSTGRES_PASSWORD, AUTH_SECRET, METRICS_TOKEN, INDEXNOW_KEY) via the random
provider instead of requiring the operator to hand-generate and paste each
into terraform.tfvars. Each is pinned with a static keepers value (secrets.tf)
so apply never regenerates a value already in use - the exact incident class
this guards against: every session invalidated, the app<->DB password
mismatched. Rotation is now a deliberate keepers edit, never a side effect.

postgres_password/auth_secret move from required inputs to optional (default
"") - explicit var wins when supplied (seeding an EXISTING live secret during
a migration onto Terraform, hop-1 cutover runbook Stage 0), else Terraform
generates and owns it. metrics_token/indexnow_key are new: neither existed in
Terraform before, both previously left for the app's own fallback generation.

VAPID deliberately stays a required, non-generated input - an EC keypair
where regeneration breaks every existing push subscription outright, unlike
an opaque token.

Sizing tiers: a locals.sizes t-shirt map (nano/small/medium/large ->
{workers, app_cpus, db_cpus, db_memory}), toward the target Proxmox
sizing-tier model (architecture doc SS6) ahead of actually provisioning VMs -
Proxmox itself stays deferred; today a tier just sizes container caps on the
existing SSH-managed hosts. A host can reference one by name (hosts.<name>.
size) or keep hand-picking the four fields, so existing tfvars are
unaffected; prod's example now uses size = "large" (identical numbers),
beta keeps explicit numbers, and a commented uat example demonstrates the
shortcut for a future ephemeral host.

Strengthened terraform/README.md's local-state caveat: more Terraform-
generated secrets landing in tfstate raises the stakes of the existing
never-commit-cleartext-state guidance, not just the sizing.

Verified: terraform validate + fmt clean. A real `terraform plan` against
fake hosts (prod/beta/uat, mixing size="large"/explicit-numbers/size="nano")
resolved every sizing correctly (prod 8/8/4/16g, beta 4/4/2/8g, uat
1/1/1/1g) and planned exactly one instance of each random_password/random_id
resource. Applied just those four resources (real generation, -target to
avoid touching the fake SSH-only host resources) and re-planned: "No
changes" - confirming the keepers pinning holds. Adding an explicit
postgres_password override afterward left the random_password resource
itself completely untouched (0 replace/destroy), confirming the override
path never disturbs the generated resource.

terraform/README.md

@@ -79,6 +79,17 @@ and back it up somewhere private. `terraform.tfvars` is likewise gitignored; onl
 `terraform.tfvars.example` (dummy values) is committed. `.terraform.lock.hcl` **is**
 committed so provider versions are pinned across machines.
 
+**Terraform now originates, not just carries, some of these secrets**
+(`postgres_password`/`auth_secret`/`metrics_token`/`indexnow_key` — see `secrets.tf`):
+left unset, each is generated by the `random` provider and its value lives *only* in
+`terraform.tfstate` — there is no other copy until it's rendered to a host's
+`/etc/thermograph.env`. More generated secrets landing in state makes the local-state
+posture above matter more, not less: **never commit or sync `terraform.tfstate` in
+cleartext**, and before this config is used from a shared machine, CI, or a remote
+backend, stand up an encrypted backend (or SOPS-wrap the state) first — see the hop-1
+cutover runbook's hazard #15 and Track B step 7. Until then, treat `terraform.tfstate`
+with the same care as the secrets it contains.
+
 ## WARNING — applying against live prod
 
 `terraform apply` runs `remote-exec` **on the server**: it resets the checkout to the

terraform/main.tf

@@ -3,6 +3,32 @@ locals {
   # files here so a compose change re-triggers the remote deploy, and this is the
   # tree the host's checkout mirrors over git.
   repo_root = abspath("${path.root}/..")
+
+  # Reusable size presets — a host can reference one by name (hosts.<name>.size)
+  # instead of hand-picking workers/app_cpus/db_cpus/db_memory separately. Mirrors
+  # the target Proxmox sizing-tier model (architecture doc §6: nano/small/medium/
+  # large -> {cores, mem}) ahead of actually provisioning VMs — Proxmox itself is
+  # deferred; today these tiers just size the container caps on the existing
+  # SSH-managed VPS hosts. "large" matches the current 48 GB prod box's numbers.
+  sizes = {
+    nano   = { workers = 1, app_cpus = 1, db_cpus = 1, db_memory = "1g" }
+    small  = { workers = 4, app_cpus = 4, db_cpus = 2, db_memory = "8g" }
+    medium = { workers = 6, app_cpus = 6, db_cpus = 3, db_memory = "12g" }
+    large  = { workers = 8, app_cpus = 8, db_cpus = 4, db_memory = "16g" }
+  }
+
+  # Per-host resolved sizing: a named tier (host.size) wins when set; otherwise
+  # the host's own workers/app_cpus/db_cpus/db_memory fields (each individually
+  # defaulted in variables.tf) — so existing tfvars with explicit numbers are
+  # unaffected, and a tier is purely an opt-in shortcut.
+  host_sizing = {
+    for name, h in var.hosts : name => h.size != null ? local.sizes[h.size] : {
+      workers   = h.workers
+      app_cpus  = h.app_cpus
+      db_cpus   = h.db_cpus
+      db_memory = h.db_memory
+    }
+  }
 }
 
 # One module instance per host. The module is entirely SSH-provisioner driven — it
@@ -21,10 +47,10 @@ module "host" {
   domain               = each.value.domain
   compose_files        = each.value.compose_files
   app_dir              = each.value.app_dir
-  workers              = each.value.workers
-  app_cpus             = each.value.app_cpus
-  db_cpus              = each.value.db_cpus
-  db_memory            = each.value.db_memory
+  workers              = local.host_sizing[each.key].workers
+  app_cpus             = local.host_sizing[each.key].app_cpus
+  db_cpus              = local.host_sizing[each.key].db_cpus
+  db_memory            = local.host_sizing[each.key].db_memory
   timescaledb_tag      = each.value.timescaledb_tag
   openmeteo            = each.value.openmeteo
   om_data_dir          = each.value.om_data_dir
@@ -39,9 +65,13 @@ module "host" {
   om_rclone_conf   = var.om_rclone_conf
   om_vfs_cache_max = var.om_vfs_cache_max
 
-  # Shared secrets -> /etc/thermograph.env
-  postgres_password     = var.postgres_password
-  auth_secret           = var.auth_secret
+  # Shared secrets -> /etc/thermograph.env. postgres_password/auth_secret/
+  # metrics_token/indexnow_key resolve through secrets.tf's locals (explicit
+  # var wins if supplied, else Terraform-generated) rather than the raw vars.
+  postgres_password     = local.postgres_password
+  auth_secret           = local.auth_secret
+  metrics_token         = local.metrics_token
+  indexnow_key          = local.indexnow_key
   vapid_private_key     = var.vapid_private_key
   vapid_public_key      = var.vapid_public_key
   vapid_contact         = var.vapid_contact

terraform/modules/thermograph-host/main.tf

@@ -26,6 +26,8 @@ locals {
     postgres_password     = var.postgres_password
     database_url          = local.database_url
     auth_secret           = var.auth_secret
+    metrics_token         = var.metrics_token
+    indexnow_key          = var.indexnow_key
     workers               = var.workers
     app_cpus              = var.app_cpus
     db_cpus               = var.db_cpus

terraform/modules/thermograph-host/templates/thermograph.env.tftpl

@@ -39,6 +39,14 @@ THERMOGRAPH_COOKIE_SECURE=${cookie_secure}
 # Signs email-confirm / password-reset tokens; pinned so links survive restarts.
 THERMOGRAPH_AUTH_SECRET=${auth_secret}
 
+# --- Ops metrics + IndexNow (Terraform-generated; see ../../secrets.tf) ---------
+# Lets the ops dashboard authenticate remotely (over an SSH tunnel or otherwise)
+# instead of requiring direct loopback access.
+THERMOGRAPH_METRICS_TOKEN=${metrics_token}
+# The IndexNow key, published at /{key}.txt as proof of ownership. Freely
+# rotatable — nothing external depends on this specific value staying put.
+THERMOGRAPH_INDEXNOW_KEY=${indexnow_key}
+
 # --- Web Push (VAPID) -----------------------------------------------------------
 # Pinned so existing browser subscriptions keep receiving across restarts.
 THERMOGRAPH_VAPID_PRIVATE_KEY=${vapid_private_key}

terraform/modules/thermograph-host/variables.tf

@@ -129,6 +129,18 @@ variable "auth_secret" {
   sensitive = true
 }
 
+variable "metrics_token" {
+  type      = string
+  sensitive = true
+  default   = ""
+}
+
+variable "indexnow_key" {
+  type      = string
+  sensitive = true
+  default   = ""
+}
+
 variable "vapid_private_key" {
   type      = string
   sensitive = true

terraform/secrets.tf

@@ -0,0 +1,65 @@
+# Terraform-generated internal secrets — values with no meaning outside this
+# deployment, so Terraform owns generating them instead of requiring the operator
+# to hand-generate and paste each one into terraform.tfvars.
+#
+# Every resource here is pinned with a static `keepers` value so `terraform
+# apply` never regenerates a value already in use — regeneration would be a real
+# incident (every session invalidated, the app<->DB password mismatched, a
+# bookmarked dashboard token broken). Rotation is a deliberate edit to a
+# resource's `keepers` value here, never a side effect of anything else
+# changing. See docs/architecture/repo-topology-and-infrastructure.md §6.
+#
+# The corresponding variable (variables.tf) defaults to "" (Terraform
+# generates); supply an explicit value only to seed an EXISTING live secret
+# during a migration onto Terraform (hop-1 cutover runbook Stage 0) —
+# `terraform plan` must show no change to the local below once the value
+# matches what's already deployed.
+#
+# VAPID is deliberately NOT here: it's an EC keypair (var.vapid_private_key /
+# vapid_public_key), generated once out-of-band and held as a stable REQUIRED
+# input. Unlike an opaque token, there is no safe "Terraform regenerates it"
+# story — regeneration breaks every existing push subscription outright.
+#
+# Generated values land in tfstate. tfstate must never be committed or synced in
+# cleartext — stand up an encrypted backend (or keep these out of Terraform and
+# supply them as pre-created Swarm secrets instead, per the hop-1 runbook's
+# hazard #15) before relying on this in a shared or remote-state setup.
+
+resource "random_password" "postgres_password" {
+  length  = 32
+  special = false # lands raw (no URL-encoding) in THERMOGRAPH_DATABASE_URL
+  keepers = {
+    rotation = "1"
+  }
+}
+
+resource "random_id" "auth_secret" {
+  byte_length = 32 # matches users.py's own secrets.token_urlsafe(32) fallback shape
+  keepers = {
+    rotation = "1"
+  }
+}
+
+resource "random_password" "metrics_token" {
+  length  = 32
+  special = false # may travel as a query param (?token=...); stay URL-safe
+  keepers = {
+    rotation = "1"
+  }
+}
+
+resource "random_id" "indexnow_key" {
+  byte_length = 16 # 32 hex chars, matching indexnow.py's own secrets.token_hex(16)
+  keepers = {
+    rotation = "1"
+  }
+}
+
+locals {
+  # var.X wins when explicitly supplied (migration: seed the existing live
+  # value); otherwise the Terraform-generated one — see the file header.
+  postgres_password = var.postgres_password != "" ? var.postgres_password : random_password.postgres_password.result
+  auth_secret       = var.auth_secret != "" ? var.auth_secret : random_id.auth_secret.b64_url
+  metrics_token     = var.metrics_token != "" ? var.metrics_token : random_password.metrics_token.result
+  indexnow_key      = var.indexnow_key != "" ? var.indexnow_key : random_id.indexnow_key.hex
+}

terraform/terraform.tfvars.example

@@ -19,12 +19,11 @@ hosts = {
     domain               = "thermograph.org" # Caddy TLS in front, app on loopback
     compose_files        = ["docker-compose.yml"]
     app_dir              = "/opt/thermograph"
-    # Sized up for the big box — tune freely. The Postgres internal budget scales from
-    # db_memory automatically (deploy/db/init/20-tuning.sh); no separate tuning edit.
-    workers   = 8
-    app_cpus  = 8
-    db_cpus   = 4
-    db_memory = "16g"
+    # "large" is the named size tier for this box (locals.sizes in main.tf) — same
+    # numbers as hand-picking workers=8/app_cpus=8/db_cpus=4/db_memory="16g" below,
+    # via the shortcut. The Postgres internal budget scales from db_memory
+    # automatically (deploy/db/init/20-tuning.sh); no separate tuning edit.
+    size = "large"
     # Self-host the ERA5 archive here: layers docker-compose.openmeteo.yml and
     # provisions the rclone mount of the object-storage bucket (om_* vars below).
     openmeteo   = true
@@ -45,11 +44,28 @@ hosts = {
     domain        = ""
     compose_files = ["docker-compose.yml"]
     app_dir       = "/opt/thermograph"
+    # Explicit numbers, not a size tier — both styles work on any host; a tier is
+    # purely an opt-in shortcut (see prod's `size = "large"` above).
     workers       = 4
     app_cpus      = 4
     db_cpus       = 2
     db_memory     = "8g"
   }
+
+  # UAT: an ephemeral, single-node environment — same images/topology shape as
+  # prod, not prod's scale (design doc §6/§9). Uncomment once a UAT box exists;
+  # not managed until then. "nano" keeps it cheap since it's destroyed when idle.
+  # uat = {
+  #   host                 = "REPLACE_WITH_UAT_VM_IP"
+  #   ssh_user             = "deploy"
+  #   ssh_private_key_path = "~/.ssh/id_ed25519"
+  #   role                 = "uat"
+  #   git_branch           = "release"
+  #   domain               = ""
+  #   compose_files        = ["docker-compose.yml"]
+  #   app_dir              = "/opt/thermograph"
+  #   size                 = "nano"
+  # }
 }
 
 # Optional overrides (shown with their defaults):
@@ -74,11 +90,20 @@ om_rclone_conf   = <<-RCLONE
   RCLONE
 
 # ---------------------------------------------------------------------------------
-# Shared secrets (DUMMY values — replace, and keep this file out of git)
+# Shared secrets (keep this file out of git)
 # ---------------------------------------------------------------------------------
-postgres_password = "REPLACE_WITH_A_STRONG_DB_PASSWORD"
-# python -c "import secrets; print(secrets.token_urlsafe(48))"
-auth_secret = "REPLACE_WITH_A_LONG_RANDOM_STRING"
+# postgres_password / auth_secret / metrics_token / indexnow_key are OPTIONAL —
+# left unset (or ""), Terraform generates and owns each one (see secrets.tf),
+# pinned so `apply` never regenerates a value already in use. Uncomment and set
+# one only to seed an EXISTING live secret when migrating onto Terraform (hop-1
+# cutover runbook Stage 0) — `terraform plan` must then show no change to it.
+# postgres_password = "REPLACE_WITH_THE_EXISTING_LIVE_DB_PASSWORD"
+# auth_secret        = "REPLACE_WITH_THE_EXISTING_LIVE_AUTH_SECRET"
+# metrics_token      = "REPLACE_WITH_THE_EXISTING_LIVE_METRICS_TOKEN"
+# indexnow_key       = "REPLACE_WITH_THE_EXISTING_LIVE_INDEXNOW_KEY"
+
+# VAPID is NOT Terraform-generated — an EC keypair, generate once out-of-band and
+# hold it as a stable input; regeneration breaks every existing push subscription.
 # cd backend && ../.venv/bin/python -c "import push,json;k=push._generate();print(k['private_key']);print(k['public_key'])"
 vapid_private_key = "REPLACE_WITH_VAPID_PRIVATE_KEY"
 vapid_public_key  = "REPLACE_WITH_VAPID_PUBLIC_KEY"

terraform/variables.tf

@@ -31,6 +31,11 @@ variable "hosts" {
     app_cpus             = optional(number, 4)                  # app container CPU cap (APP_CPUS)
     db_cpus              = optional(number, 2)                  # db container CPU cap (DB_CPUS)
     db_memory            = optional(string, "8g")               # db container memory cap (DB_MEMORY)
+    # A named size tier (see locals.sizes in main.tf: nano/small/medium/large) —
+    # when set, overrides the four fields above with the tier's preset. Leave
+    # null (default) to keep hand-picking workers/app_cpus/db_cpus/db_memory,
+    # as prod/beta already do below.
+    size = optional(string, null)
     # The floating tag matches today's behavior everywhere until you pin it. Pin to
     # an exact minor (SELECT extversion FROM pg_extension WHERE extname='timescaledb'
     # on the live DB) before any host of this stack could ever replicate with
@@ -88,16 +93,37 @@ variable "om_vfs_cache_max" {
 # These live in terraform.tfvars (gitignored) and land in local state — never commit
 # real values. Every host in this config shares the same secret set; split the config
 # if prod and dev must diverge on a credential.
+#
+# postgres_password / auth_secret / metrics_token / indexnow_key default to "" —
+# Terraform generates and owns each one (see secrets.tf), pinned so `apply` never
+# regenerates a live value. Supply an explicit value only to seed an EXISTING live
+# secret during a migration onto Terraform (hop-1 cutover runbook Stage 0).
 variable "postgres_password" {
-  description = "PostgreSQL password (initializes the db container AND builds THERMOGRAPH_DATABASE_URL)."
+  description = "PostgreSQL password (initializes the db container AND builds THERMOGRAPH_DATABASE_URL). \"\" (default) => Terraform-generated."
   type        = string
   sensitive   = true
+  default     = ""
 }
 
 variable "auth_secret" {
-  description = "THERMOGRAPH_AUTH_SECRET — signs email-confirm / password-reset tokens. Pin it."
+  description = "THERMOGRAPH_AUTH_SECRET — signs email-confirm / password-reset tokens. \"\" (default) => Terraform-generated."
   type        = string
   sensitive   = true
+  default     = ""
+}
+
+variable "metrics_token" {
+  description = "THERMOGRAPH_METRICS_TOKEN — optional remote-access token for the ops metrics endpoint. \"\" (default) => Terraform-generated."
+  type        = string
+  sensitive   = true
+  default     = ""
+}
+
+variable "indexnow_key" {
+  description = "THERMOGRAPH_INDEXNOW_KEY — IndexNow key, published at /{key}.txt. \"\" (default) => Terraform-generated."
+  type        = string
+  sensitive   = true
+  default     = ""
 }
 
 variable "vapid_private_key" {