43c9e2052a
Some checks failed
PR build (required check) / changes (pull_request) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Failing after 9s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
Brings the last two credential stores under SOPS, so all three environments and the control plane are managed the same way. dev.yaml — dev was not "intentionally not wired in", it was broken The vault README claimed dev had no real secrets. Both halves were false. Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls back to https://thermograph.org, so dev's IndexNow pings and verification links claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real base URL, and 12 values total. And dev already held production credentials: backend-deploy-dev.yml injects S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned keypair for that bucket is read-write — on the bucket holding prod's backups. Dev does not need them; without bucket creds the lake service 503s and history falls through to the archive. Those workflow lines still need removing. dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of common.yaml's sixteen values are live production credentials, the render is a plaintext concatenation consumed through env_file:, and dev is the operator's desktop AND the Forgejo runner executing unreviewed dev-branch code with the docker socket mounted. An override in dev.yaml would not help — last-wins governs consumers, but prod's value is still physically a line in the file. Verified: current renderer leaks 28 lines onto dev including both S3 keypairs and the VAPID private key; patched gives exactly 12 keys and no prod credential. Beta's render is byte-identical either way. centralis.prod.yaml — its own file, its own renderer /etc/centralis.env was hand-edited, which is how a JSON registry got written unquoted into a shell-sourced file tonight and silently collapsed three identities to one. The renderer now owns the file. It is service- AND environment-scoped, not folded into prod.yaml, because /etc/thermograph.env is loaded into every container in the app stack. Folding these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which authenticate an endpoint carrying run_on_host and sql_query(write) — into the web backend's environment, turning a read-anything bug in the app into a foothold on the control plane. Quoting is guaranteed three ways rather than assumed. Critically, `sops -d --output-type dotenv` emits values verbatim, so reusing the existing render path would have reproduced tonight's bug exactly. The new function decrypts to JSON and emits POSIX single-quoted assignments; it then sources its own output in a clean shell and compares every value before touching /etc; and it refuses to write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after sourcing. Tested against 16 hostile values including embedded quotes, newlines and ;rm -rf /. Value-identity proven, not asserted: rendered vs live compared as *effective* values on prod (a textual diff would report a false difference, and would report a false match if the vault had captured quote characters as part of the value), plus both env files producing a byte-identical `docker compose config` digest. 9 keys, both token subjects preserved. Nothing deployed. Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw dotenv only because none of its 32 current values contains a shell-special character. It is one quoted secret away from the same bug. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY |
||
|---|---|---|
| .. | ||
| .claude/skills/key-gaps | ||
| deploy | ||
| lake-iceberg | ||
| ops | ||
| terraform | ||
| .env.example | ||
| .gitignore | ||
| .sops.yaml | ||
| ACCESS.md | ||
| CLAUDE.md | ||
| DEPLOY-DEV.md | ||
| DEPLOY.md | ||
| docker-compose.dev.yml | ||
| docker-compose.openmeteo.yml | ||
| docker-compose.yml | ||
| docker-stack.yml | ||
| Makefile | ||
| README.md | ||
thermograph-infra
Infrastructure for Thermograph: Terraform host
provisioning, the SOPS+age secrets vault, Docker Swarm/WireGuard networking,
Forgejo, Caddy, and the deploy scripts that run the already-built app image on
each host. Extracted from the app monorepo (emi/thermograph) — the app repo
owns building and testing the app; this repo owns running it.
terraform/— provisions/configures hosts (SSH-driven by default; an optional GCP-creating module is scaffolded, no live resources yet) and triggers each deploy. Seeterraform/README.md.deploy/secrets/— the git-native SOPS+age secrets vault (every app secret, encrypted at rest, rendered at deploy time). Seedeploy/secrets/README.md.deploy/swarm/,deploy/forgejo/— the 3-node WireGuard/Swarm cluster that hosts Forgejo (git + CI + registry); does not run the app itself. SeeACCESS.mdand the READMEs under each directory.deploy/deploy.sh— pulls the pinned app image (IMAGE_TAG) and rolls the compose stack; invoked by Terraform and by the app repo's.forgejo/workflows/deploy.ymlover SSH.docker-compose*.yml,docker-stack.yml— how the app image runs (compose in production today;docker-stack.ymlis a design record for a possible future Swarm-based app deploy, not currently live).
The app's own source, Dockerfile, and build/test CI stay in the app repo —
this repo never checks out app source; hosts only pull tagged images from the
registry. See ACCESS.md for host access and the Swarm/Forgejo topology, and
terraform/README.md for the day-to-day plan/apply workflow.
Branches & how changes reach each environment
main— what prod and beta run: their/opt/thermographcheckoutsgit reset --hard origin/mainat the start of every deploy (deploy/deploy.sh). A merge tomainreaches those hosts on the next app deploy (or a by-handdeploy.shrun); there is no separate infra deploy trigger.dev— what LAN dev runs:~/thermograph-devresets to it viadeploy/deploy-dev.sh. Keep it fast-forwarded tomain(infra changes are not environment-staged today; the branches exist so LAN dev can trail or lead when needed).release— currently consumed by nothing (prod tracksmain, notrelease). It exists to mirror the app repos' dev→main→release promotion shape if per-environment infra staging is ever wanted; until then, treatmainas live-everywhere.
Note the asymmetry with the app repos: app code IS environment-staged (dev→main→release maps to LAN→beta→prod via image tags), infra is not.