5 commits

Author	SHA1	Message	Date
Emi Griffith	7b14b9062c	openbao: make the parity gate actually work, and run it nightly ... Three fixes, all on the path between "the migration looks ready" and "the migration is ready". verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and render-secrets-openbao.sh fell through to the bare default -- prod's credential -- for every environment. On vps2 that made `--env beta` authenticate as tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta. `--all` failed the same way, taking dev with it. deploy.sh and deploy-stack.sh always sourced it; only the verifier did not, which is the worst place for the omission: the tool whose job is to notice divergence was itself diverging from the path it verifies. It now calls thermograph_topology per environment and takes TG_SKIP_COMMON from there rather than re-deriving it. infra-sync.yml renders all three env files on every push touching infra/**, and none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the selector would have changed deploy.sh and deploy-stack.sh but not this workflow, which would have kept re-rendering from SOPS -- silent while parity holds, and a hard failure the moment the SOPS files are retired. The one-line cutover the README describes was never sufficient on its own. ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from vps1, nightly, failing loudly on a mismatch. README.md called this the gate for a cutover; nothing implemented it. A gate that exists only in prose gets satisfied by assertion. It grants CI no vault access -- the host renders, CI only asks it to. First measured run, immediately after the verifier fix: dev 12 keys PASS, byte-identical beta 24 keys FAIL, 3 values differ prod 32 keys FAIL, 3 values differ The same three keys on both: THERMOGRAPH_S3_SECRET_KEY, THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed. dev passes because dev never layers common. One `seed-from-sops.sh --env common` fixes beta and prod together, and the 7-day clock starts after that.	2026-08-01 08:36:35 -07:00
Emi Griffith	001e6b1365	ci: add an always-on Actions runner on vps2 ... The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up.	2026-08-01 07:54:28 -07:00
emi	d138f00a20	infra: split the estate into vps1/vps2 — beta joins prod, dev gets a home (#103)	2026-07-26 06:56:38 +00:00
Emi Griffith	6086b40fe6	Add off-box encrypted backups to Contabo Object Storage ... Prod DB dumps and Forgejo (db + data volume) are now copied off-box to the era5-thermograph S3 bucket, age-encrypted to the vault recipient so the host age key at /etc/thermograph/age.key decrypts them for restore. Nothing plaintext leaves the box; no local intermediate (streamed via age | rclone rcat). ops-cron: the prod pg_dump job gains an off-box push; a new forgejo-backup job dumps Forgejo's Postgres + tars its data volume on beta (the git host had NO backup at all). 30-day off-box retention on both. S3 creds are Forgejo Actions secrets (S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY), mirrored into the SOPS vault (prod.yaml/beta.yaml) for host-side use. Contabo needs path-style addressing. Adds deploy/backup/README.md with the DR restore runbook.	2026-07-23 13:42:09 -07:00
Emi Griffith	7b2db07722	CI: port the split repos' workflows to per-domain path-filtered monorepo pipelines ... One root workflow set replaces the four repos' copies (deleted -- root-only is where Forgejo reads them, and dead copies are a trap): per-domain build-push with explicit image paths (emi/thermograph/backend|frontend; the old github.repository-derived path collides in a monorepo), path-filtered per-domain beta/prod/dev deploys, a domain-input reusable build check, a single always-reporting PR gate (path-filtered required checks deadlock auto-merge), a new infra-sync pipeline (host checkout + secrets render on infra/** pushes), and ports of secrets-guard / ops-cron / observability-validate to monorepo paths.	2026-07-22 22:11:33 -07:00

Renamed from infra/.forgejo/workflows/ops-cron.yml (Browse further)