All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Successful in 18s
PR build (required check) / gate (pull_request) Successful in 2s
Environments stop being machines. Until now each box WAS an environment --
"beta" named both a deploy target and a host, "the desktop" named both the
operator's computer and the dev server -- so every path could assume one
environment per host and hardcode /opt/thermograph, /etc/thermograph.env and
ports 8137/8080. That assumption ends here:
vps1 75.119.132.91 Forgejo, Grafana/Loki, the portfolio site, and DEV
(own Postgres, mesh-only on 10.10.0.2:8137)
vps2 169.58.46.181 PROD and BETA as two Swarm stacks sharing one
TimescaleDB instance, plus Centralis, Postfix, backups
desktop AI model hosting + flex Swarm capacity, no environment
Nothing live has moved; the ordered cutover is in
infra/deploy/RUNBOOK-vps1-vps2-cutover.md.
deploy/env-topology.sh is the single source of truth: env -> host, checkout,
branch, deploy mode, stack name, env file, LB ports, DB role/database, service
prefix. THERMOGRAPH_ENV is the input, and on vps2 it is the only thing
distinguishing a beta deploy from a prod one -- deploy.sh refuses to run when it
disagrees with the checkout it was invoked from. The host-wide secrets-env and
deploy-mode markers survive only as a fallback for a single-environment box.
Beta gets its own stack file rather than an overlay (a merge cannot REMOVE the
db service, and beta having no database of its own is the point). Its services
are prefixed beta-*: Swarm registers a service's short name as a DNS alias on
every network it joins, so two stacks both calling a service `web` on the shared
data network would let prod's frontend resolve a beta task. Prod's stack, env
and LB config are untouched.
One Postgres, two databases with two roles: deploy/db/provision-env-db.sh
creates thermograph_beta (NOSUPERUSER, owns only its own database, CONNECT
revoked from PUBLIC) plus a <role>_ro for ad-hoc queries, and refuses to run for
the environment that owns the instance -- doing so would demote prod's bootstrap
superuser.
Fixes that co-residency would otherwise have broken silently:
- CI secrets are keyed by host (VPS1_SSH_*, VPS2_SSH_*). SSH_* meant "beta" and
also "the Forgejo box" because those shared a machine; that conflation is what
once had the prod backup dumping beta.
- The nightly backup dumps BOTH application databases to separate off-box
prefixes, and fails loudly on a missing one instead of skipping it.
- Alloy stopped deriving the `host` label from the node -- on vps2 that would
have filed every beta line as prod, feeding prod's alert rules with beta's
traffic. It is now derived per source, with a new `node` label for the machine,
and beta's log volume is mounted via a vps2-only overlay.
- ops/dbq.sh, ops/iceberg.sh and the secrets seed scripts derived their SSH
target and env-file path from the topology instead of hardcoding beta to
75.119.132.91 -- which is vps1's address now.
- Dev's overlay binds DEV_BIND_ADDR (loopback by default, the mesh address on
vps1) instead of 0.0.0.0: correct for a home LAN box, a public exposure of
unreviewed branches on a VPS.
Terraform's hosts variable is keyed by machine with a nested environments map,
and main.tf flattens (host, environment) pairs so two environments on one box
cannot share a checkout. Caddy's single reference config is split per host.
Docs, onboarding and the runbooks are updated throughout, including the security
rationales that co-residency makes false: separate SSH credentials no longer put
a host boundary between beta and prod, and the boundary that remains is the
database and the filesystem.
185 lines
12 KiB
HCL
185 lines
12 KiB
HCL
# ---------------------------------------------------------------------------------
|
|
# Hosts
|
|
# ---------------------------------------------------------------------------------
|
|
# Keyed by HOST (vps1, vps2), NOT by environment: vps2 alone runs TWO environments
|
|
# (prod AND beta) as separate Swarm stacks on the SAME box, so "one entry = one
|
|
# environment" broke the moment that became true — two entries that both defaulted
|
|
# `app_dir` to "/opt/thermograph" would let a prod apply and a beta apply stomp the
|
|
# identical checkout. Each host now carries its SSH identity ONCE (host IP,
|
|
# ssh_user, key) plus an `environments` map of everything that varies per
|
|
# environment running on it — checkout path, git branch, image tags, domain,
|
|
# sizing. `app_dir` is REQUIRED with no default for exactly this reason: two
|
|
# environments on one host must be given two explicitly DIFFERENT paths (see
|
|
# terraform.tfvars.example), so their checkouts — and a `git reset --hard` in one
|
|
# of them — can never collide.
|
|
#
|
|
# vps1 (75.119.132.91): Forgejo (git+CI+registry), Grafana/Loki/Alloy,
|
|
# emigriffith.dev, and the `dev` environment (its own Postgres, mesh-only, no
|
|
# public DNS/Caddy site). vps2 (169.58.46.181): `prod` (thermograph.org) and
|
|
# `beta` (beta.thermograph.org) — Centralis, Postfix and the backups also live
|
|
# here — sharing the ONE TimescaleDB instance both app environments use, on
|
|
# separate databases/roles (deploy/db/provision-env-db.sh); this variable's
|
|
# per-environment db_cpus/db_memory size only that environment's app-container
|
|
# caps, not a second database (see deploy/db/init/20-tuning.sh). The LAN dev
|
|
# branch (`make dev-up`, a laptop-local rehearsal) deploys nowhere via Terraform
|
|
# and is NOT managed here.
|
|
#
|
|
# An environment with `domain = ""` gets no Caddy/TLS: its app port is opened on
|
|
# the firewall and it's reached directly.
|
|
#
|
|
# Resource sizing (workers / app_cpus / db_cpus / db_memory), per environment,
|
|
# defaults to the historical 4 / 4 / 2 / 8g budget (right for beta). vps2 is 48 GB
|
|
# / 12 cores — raise these for prod (the example uses app_cpus 8, db_cpus 4,
|
|
# db_memory "16g", via the "large" size tier). The Postgres internal budget scales
|
|
# from db_memory automatically (deploy/db/init/20-tuning.sh), so no separate
|
|
# tuning edit is needed to exploit the RAM.
|
|
#
|
|
# TODO(cutover): main.tf's `module.host` for_each (via `local.all_hosts`) still
|
|
# expects ONE flat entry per key, reading `each.value.host`/`.role`/`.app_dir`/…
|
|
# directly. It needs a flattening step ahead of that for_each — one derived entry
|
|
# per (host, environment) pair, keyed e.g. "<host>-<environment>" ("vps2-prod" and
|
|
# "vps2-beta" as two separate module instances sharing vps2's host/ssh_user/
|
|
# ssh_private_key_path but each with its own app_dir/role/image tags/sizing).
|
|
# This variable's new shape is not yet wired into main.tf's module block.
|
|
variable "hosts" {
|
|
description = "Map of VPS boxes to manage, keyed by a short host name (\"vps1\", \"vps2\") — NOT by environment. SSH identity is per-host; everything environment-specific lives in that host's `environments` map."
|
|
type = map(object({
|
|
host = string # IP or hostname to SSH to
|
|
ssh_user = optional(string, "deploy") # SSH login user
|
|
ssh_private_key_path = string # path to the private key for that user
|
|
# One entry per environment running on this host, keyed by environment name
|
|
# ("prod" | "beta" | "dev"). A host normally carries one; vps2 carries two.
|
|
environments = map(object({
|
|
role = string # "prod" | "beta" | "dev" (informational + outputs; should match the map key above)
|
|
git_branch = string # this INFRA repo's branch the checkout is reset to
|
|
# Which app images to run, e.g. "sha-<12 hex>" (each matching build-push.yml's tag
|
|
# for the commit that app repo built) or a semver tag on a release push. The app is
|
|
# TWO separately-published images now — emi/thermograph-backend/app and
|
|
# emi/thermograph-frontend/app — pinned independently. Required, no default: the
|
|
# host's checkout is this infra repo, not the app repos, so there's no "current
|
|
# commit" to derive a tag from; both must be explicit. Bump these (via a normal
|
|
# tfvars edit + apply) whenever an app repo ships a commit you want this
|
|
# environment running; the infra repo's own git_branch is independent and rarely
|
|
# needs to change.
|
|
backend_image_tag = string
|
|
frontend_image_tag = string
|
|
domain = optional(string, "") # public domain for this environment; "" => no Caddy/TLS
|
|
compose_files = optional(list(string), ["docker-compose.yml"])
|
|
# REQUIRED, no default — see the file header: two environments on one host
|
|
# must never be able to default to the same checkout path.
|
|
app_dir = string # checkout path on the host
|
|
workers = optional(number, 4) # uvicorn workers (WORKERS)
|
|
app_cpus = optional(number, 4) # app container CPU cap (APP_CPUS)
|
|
db_cpus = optional(number, 2) # db container CPU cap (DB_CPUS)
|
|
db_memory = optional(string, "8g") # db container memory cap (DB_MEMORY)
|
|
# A named size tier (see locals.sizes in main.tf: nano/small/medium/large) —
|
|
# when set, overrides the four fields above with the tier's preset. Leave
|
|
# null (default) to keep hand-picking workers/app_cpus/db_cpus/db_memory,
|
|
# as prod/beta already do below.
|
|
size = optional(string, null)
|
|
# The floating tag matches today's behavior everywhere until you pin it. Pin to
|
|
# an exact minor (SELECT extversion FROM pg_extension WHERE extname='timescaledb'
|
|
# on the live DB) before any host of this stack could ever replicate with
|
|
# another — a floating tag risks mismatched extension minors, which blocks a
|
|
# physical replica (hop-1 cutover runbook hazard #7). Use the SAME tag everywhere.
|
|
timescaledb_tag = optional(string, "latest-pg18")
|
|
# Self-host the ERA5 archive (docker-compose.openmeteo.yml + a host rclone mount
|
|
# of the object-storage bucket). Only the self-hosting environment (prod) sets true.
|
|
openmeteo = optional(bool, false)
|
|
om_data_dir = optional(string, "/mnt/om-archive") # host rclone mount point (OM_DATA_DIR)
|
|
}))
|
|
}))
|
|
}
|
|
|
|
# GCP-created hosts, keyed the same way `hosts` used to be — one entry per
|
|
# environment, since these are hypothetical single-purpose boxes (e.g. a future
|
|
# UAT VM), not vps1/vps2, so they don't need the host/environment split above.
|
|
# Default {} => zero GCP resources planned and the google provider is never
|
|
# actually invoked (see versions.tf and modules/gcp-host). Populate an entry to
|
|
# have Terraform create the VM itself; its output IP then feeds into the SAME
|
|
# thermograph-host module every SSH-managed host uses (main.tf), so provisioning
|
|
# logic is never duplicated between providers.
|
|
# TODO(cutover): main.tf's `local.all_hosts = merge(var.hosts, ...)` currently
|
|
# merges this flat shape with `var.hosts` into one map for `module.host`'s
|
|
# for_each. Once `var.hosts` is nested (above), that merge needs to target the
|
|
# same flattened (host, environment) shape this variable already has — e.g. by
|
|
# treating each `gcp_hosts` entry as its own single-environment host.
|
|
variable "gcp_hosts" {
|
|
description = "Map of hosts for Terraform to CREATE on GCP (Compute Engine), keyed the same way as `hosts`. Empty by default -- no live GCP resources exist yet; this is a scaffold for future use. See modules/gcp-host."
|
|
type = map(object({
|
|
project = string # GCP project ID
|
|
zone = string # e.g. "us-west1-a"
|
|
machine_type = optional(string, "e2-medium")
|
|
ssh_user = optional(string, "deploy")
|
|
ssh_public_key_path = string # path to the PUBLIC key installed on the instance
|
|
ssh_private_key_path = string # path to the matching PRIVATE key (for the module's provisioner)
|
|
role = string
|
|
git_branch = string
|
|
backend_image_tag = string
|
|
frontend_image_tag = string
|
|
domain = optional(string, "")
|
|
compose_files = optional(list(string), ["docker-compose.yml"])
|
|
app_dir = optional(string, "/opt/thermograph")
|
|
size = optional(string, "small")
|
|
timescaledb_tag = optional(string, "latest-pg18")
|
|
}))
|
|
default = {}
|
|
}
|
|
|
|
variable "repo_url" {
|
|
description = "Git remote to clone from when a host has no checkout yet. Points at THIS repo (thermograph-infra) now, not the app repo -- the app's own source is never checked out on a host; only its published registry images are pulled (see var.hosts[*].backend_image_tag / frontend_image_tag). thermograph-infra is a private repo, so this typically needs embedded read credentials, e.g. a Forgejo deploy token: \"https://<token-name>:<token>@git.thermograph.org/emi/thermograph-infra.git\"."
|
|
type = string
|
|
default = "https://git.thermograph.org/emi/thermograph-infra.git"
|
|
sensitive = true
|
|
}
|
|
|
|
variable "app_port" {
|
|
description = "Port the app binds inside the container / is health-checked on."
|
|
type = number
|
|
default = 8137
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------------
|
|
# Self-hosted Open-Meteo (object storage) — consumed only by hosts with openmeteo=true
|
|
# ---------------------------------------------------------------------------------
|
|
# The ERA5 .om archive lives in an object-storage bucket, surfaced on the host by an
|
|
# rclone FUSE mount at each host's om_data_dir. These describe that bucket + mount.
|
|
# om_rclone_conf holds credentials, so it's sensitive and lands in state — keep the
|
|
# real value in terraform.tfvars (gitignored), never committed.
|
|
variable "om_bucket_remote" {
|
|
description = "rclone remote:path for the archive bucket, e.g. \"om-archive:thermograph-era5\" (matches a [remote] in om_rclone_conf)."
|
|
type = string
|
|
default = ""
|
|
}
|
|
|
|
variable "om_rclone_conf" {
|
|
description = "Full rclone.conf contents defining the archive remote (installed to /etc/rclone/rclone.conf, 0600). Sensitive."
|
|
type = string
|
|
default = ""
|
|
sensitive = true
|
|
}
|
|
|
|
variable "om_vfs_cache_max" {
|
|
description = "rclone --vfs-cache-max-size: bounds the on-disk hot cache for the mount (keep within the disk budget)."
|
|
type = string
|
|
default = "80G"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------------
|
|
# Secrets: owned by the SOPS+age vault now, NOT Terraform
|
|
# ---------------------------------------------------------------------------------
|
|
# POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_METRICS_TOKEN,
|
|
# THERMOGRAPH_INDEXNOW_KEY, THERMOGRAPH_VAPID_*, REGISTRY_TOKEN, Discord/SMTP
|
|
# credentials, and every other app secret are no longer Terraform variables --
|
|
# they're rendered at deploy time from deploy/secrets/*.yaml (SOPS-encrypted,
|
|
# committed) by deploy/render-secrets.sh, which deploy.sh calls before `docker
|
|
# compose up`. Terraform's job here shrank to topology/sizing only (see
|
|
# modules/thermograph-host/templates/thermograph-topology.env.tftpl) plus
|
|
# triggering the deploy. See deploy/secrets/README.md to rotate or add a secret.
|
|
#
|
|
# The four secrets Terraform used to generate itself (postgres_password,
|
|
# auth_secret, metrics_token, indexnow_key) still exist as values -- they just
|
|
# live in the vault now, seeded once from Terraform's own generated values via
|
|
# deploy/secrets/seed-from-live.sh so the handoff changed nothing in use. Rotate
|
|
# them the same way as any other vault secret from here on (sops edit + commit +
|
|
# deploy), not via a Terraform keeper change.
|