9 commits

Author	SHA1	Message	Date
Emi Griffith	657e27ba9f	openbao: render Centralis from the vault ... Centralis could not migrate at all. `seed-from-sops.sh --all` refused centralis.prod with `CENTRALIS_TOKENS: contains a shell metacharacter`, and render-secrets-openbao.sh had no Centralis path whatsoever — so the seeded data would have had nothing to read it. Both halves come from applying the app stack's rules to a file that does not share its shape. /etc/thermograph.env is raw unquoted KEY=value. /etc/centralis.env is consumed by exactly one thing and that thing is a shell: sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up' CENTRALIS_TOKENS is JSON, so it can never satisfy a no-metacharacters rule, and rendering it unquoted is precisely the 2026-07-24 failure: bash strips the quotes, Centralis fails closed on the malformed registry and serves a single `shared` identity, indistinguishable from the file never being written. So: - seed-from-sops.sh validates against the renderer that will consume the path. The dotenv rules still apply to common and env/*; centralis/* is exempt because it is single-quoted downstream. - render-secrets-openbao.sh gains thermograph_render_openbao_centralis, mirroring render_centralis_secrets: fetch as JSON, emit POSIX single-quoted assignments, then PROVE the file by sourcing it under `env -i` with `set -a` and comparing every value back before anything touches the destination. Mirrored rather than shared, per the existing note above thermograph_render_openbao — consolidate when the SOPS path is deleted. - render_centralis_secrets gains the same early-return backend dispatch the app path already uses, so the new function is reachable. Without it the SOPS path was the only path regardless of TG_SECRETS_BACKEND. Verified against a synthetic registry containing a JSON token map, an apostrophe, and `$(...)`/backtick/backslash/double-quote payloads: 5 keys round-trip byte-for-byte through `set -a; . file`, CENTRALIS_TOKENS parses to the right subjects, and the command substitution does not execute. The negative case is covered too — fed the unquoted 2026-07-24 shape, the round-trip check reports the value as changed (by length, never content) and refuses to write. Also corrects the claim in render-secrets-openbao.sh that the SOPS path never tripped the metacharacter hazard "because every current value happens to be alphanumeric". CENTRALIS_TOKENS is the counter-example; it never tripped those paths because it was never in them.	2026-07-30 19:55:17 -07:00
Emi Griffith	6bd07d33cb	openbao: add a dormant OpenBao backend alongside SOPS ... Phase 1 of moving the secret store to OpenBao. Nothing is cut over: TG_SECRETS_BACKEND defaults to sops for all three environments, so SOPS remains authoritative until an environment is flipped in a reviewed PR. The reason for the migration is one specific failure class. infra/CLAUDE.md requires that vps1 never hold prod credentials, but that rule is currently enforced by a caller-side shell flag (THERMOGRAPH_SECRETS_SKIP_COMMON) at three call sites. A fourth call site, or one by-hand render, writes both S3 keypairs, the VAPID private key, REGISTRY_TOKEN and the IndexNow/metrics tokens onto the Forgejo CI-runner box and exits 0. policies/tg-host-dev.hcl makes that a 403 instead: dev's identity cannot read the common path at all. Shape: - vps2 only, native systemd (not a container: a Swarm service is circular, and a plain docker run is one `prune -a` from gone), raft storage (2.6.0 deprecated the file backend), static auto-unseal so a reboot cannot block deploys, mesh-only self-signed TLS (not ACME — that would put DNS in the boot chain of the secret store). - AppRole per environment, CIDR-bound, 5-minute tokens. Each environment's secret-id is owned by its own deploy user, which is a boundary that does not exist today since prod and beta both reach the same root-owned age key. - Render still produces the same raw unquoted dotenv: Centralis compares /etc/thermograph.env textually and the Swarm entrypoint parses it by line. The backend dispatch in render-secrets.sh returns early rather than refactoring the shared write path, so the SOPS path stays byte-identical. Verified by rendering dev (12 keys) and prod (32 keys) through it, matching the live hosts. The duplicated write path is noted for consolidation once prod has been on OpenBao for a release cycle. The OpenBao renderer rejects values containing a newline or a shell metacharacter. /etc/thermograph.env is sourced by bash in six places, so such a value is a command-execution path on the deploy host; the SOPS path has always had this hazard and avoids it only because every current value happens to be alphanumeric. Also records that the age keypair is NOT retired by this migration: it is the backup encryption key for every off-box dump (ops-cron.yml:142,197, 30-day retention), so deleting it with the vault files would silently destroy a month of database recoverability. verify-parity.sh is the cutover gate — it renders both backends and diffs them, reporting key names and counts only, never values.	2026-07-29 21:35:20 -07:00
emi	d138f00a20	infra: split the estate into vps1/vps2 — beta joins prod, dev gets a home (#103)	2026-07-26 06:56:38 +00:00
emi	9b6b12a5da	infra: mirror LAN dev secrets under $HOME for snap-confined Docker (#85)	2026-07-25 07:16:08 +00:00
Emi Griffith	33cd69e44e	secrets: silence two SC2016s that are the intended behaviour ... shell-lint failed on two single-quoted strings. Both are deliberate and double-quoting either one would break it silently rather than loudly. In render_centralis_secrets, the verification step runs `bash -c 'set -a; . "$1"'` with the rendered file passed as an argument. The "$1" must be expanded by that inner shell. Double-quoted, it would interpolate the OUTER $1 — the function's own first argument, the repo root — so the check would source the wrong path and still report success. That check exists precisely to catch a bad render, so a false pass there is worse than no check. In verify-centralis-render.sh, the string is executed by the REMOTE shell over ssh. Double-quoted, mktemp would run locally and the script would send the host a path that does not exist there. Suppressed with the reason at each site rather than loosened globally. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY	2026-07-24 18:13:08 -07:00
Emi Griffith	43c9e2052a	secrets: vault dev and Centralis; render dev without common.yaml ... Brings the last two credential stores under SOPS, so all three environments and the control plane are managed the same way. dev.yaml — dev was not "intentionally not wired in", it was broken The vault README claimed dev had no real secrets. Both halves were false. Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls back to https://thermograph.org, so dev's IndexNow pings and verification links claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real base URL, and 12 values total. And dev already held production credentials: backend-deploy-dev.yml injects S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned keypair for that bucket is read-write — on the bucket holding prod's backups. Dev does not need them; without bucket creds the lake service 503s and history falls through to the archive. Those workflow lines still need removing. dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of common.yaml's sixteen values are live production credentials, the render is a plaintext concatenation consumed through env_file:, and dev is the operator's desktop AND the Forgejo runner executing unreviewed dev-branch code with the docker socket mounted. An override in dev.yaml would not help — last-wins governs consumers, but prod's value is still physically a line in the file. Verified: current renderer leaks 28 lines onto dev including both S3 keypairs and the VAPID private key; patched gives exactly 12 keys and no prod credential. Beta's render is byte-identical either way. centralis.prod.yaml — its own file, its own renderer /etc/centralis.env was hand-edited, which is how a JSON registry got written unquoted into a shell-sourced file tonight and silently collapsed three identities to one. The renderer now owns the file. It is service- AND environment-scoped, not folded into prod.yaml, because /etc/thermograph.env is loaded into every container in the app stack. Folding these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which authenticate an endpoint carrying run_on_host and sql_query(write) — into the web backend's environment, turning a read-anything bug in the app into a foothold on the control plane. Quoting is guaranteed three ways rather than assumed. Critically, `sops -d --output-type dotenv` emits values verbatim, so reusing the existing render path would have reproduced tonight's bug exactly. The new function decrypts to JSON and emits POSIX single-quoted assignments; it then sources its own output in a clean shell and compares every value before touching /etc; and it refuses to write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after sourcing. Tested against 16 hostile values including embedded quotes, newlines and ;rm -rf /. Value-identity proven, not asserted: rendered vs live compared as *effective* values on prod (a textual diff would report a false difference, and would report a false match if the vault had captured quote characters as part of the value), plus both env files producing a byte-identical `docker compose config` digest. 9 keys, both token subjects preserved. Nothing deployed. Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw dotenv only because none of its 32 current values contains a shell-special character. It is one quoted secret away from the same bug. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY	2026-07-24 18:10:51 -07:00
Emi Griffith	cb69a22c0f	secrets: factor shared values into common.yaml; stop the renderer failing open ... Two changes to how credentials are stored and distributed. 1. common.yaml now exists. The renderer has always concatenated common.yaml then <env>.yaml, host winning, and the README has always documented common.yaml with a checkmark. The file was never created. So every value shared by prod and beta was duplicated in both vaults, free to drift apart with nothing to detect it. 16 values move to common.yaml: VAPID keypair, metrics token, IndexNow key, REGISTRY_TOKEN, the S3 endpoint/bucket and both S3 keypairs, plus shared config. 5 stay per-host because their values genuinely differ (APP_CPUS, DB_CPUS, DB_MEMORY, WORKERS, THERMOGRAPH_BASE_URL). 8 exist only on prod — the Discord and mail credentials, which beta does not have at all. Three more are held back deliberately despite being identical today: POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_DATABASE_URL. These are the credentials that let one environment act as another, and beta is the more exposed box — it serves public Forgejo and Grafana. They match only because beta was seeded from prod. Keeping them per-host costs one line each and preserves the ability to diverge; putting them in common.yaml would encode the equivalence as intentional and make breaking it a migration rather than an edit. Reasoning recorded in the vault README, since a future reader will otherwise "fix" it. Done without any plaintext leaving prod: ciphertext was shipped up, decrypted against the host's age key, recombined, re-encrypted, and shipped back. The consolidation refuses to write unless it has proved merged(common + env) equals the original env vault exactly — same keys, same values — and re-verifies from the written files afterwards. Both checks passed for prod and beta. 2. render_thermograph_secrets no longer fails open. One `return 0` covered two different situations: "this host is not configured for SOPS" (true of the LAN dev box, and correct to succeed) and "this host IS configured but its vault is not where we looked". The second is a failure, and returning 0 made it a silent no-op wearing a success code — a deploy would report success having rendered nothing, and the host would keep serving whatever /etc/thermograph.env already held, including after a rotation. The likely cause is passing the wrong root: the function wants the directory containing deploy/secrets, which on the hosts is /opt/thermograph/infra, not /opt/thermograph. That mistake looked exactly like "not configured here", which is why it went unnoticed. It now exits 1 and names the probable cause. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY	2026-07-24 17:40:26 -07:00
emi	de8e847f9f	shell: add shellcheck CI guard and drive the tree to zero findings (#19) ... Adds .forgejo/workflows/shell-lint.yml (pinned shellcheck v0.11.0 + sha256, -x, default severity, fail on any finding, not path-filtered) and drives all 26 scripts to zero findings. Two defects shellcheck cannot see: render-secrets.sh left DECRYPTED vault contents in /tmp whenever a sops decrypt failed -- the caller's set -e aborted the function before either cleanup ran. Now removed on every exit path, with `|| return 1` on both sops calls so a decrypt failure can never write a partial /etc/thermograph.env regardless of the caller's shell options. Explicitly not a `trap ... RETURN`: such a trap set in a sourced function persists into the caller's shell and re-fires when the caller's next `.`/source completes, where the function-local tmp is unset -- fatal and silent under deploy.sh's set -u. The file now records that reasoning. autoscale.sh ran `set -eu` without pipefail while piping docker stats into awk, so a failed left side was swallowed and the loop autoscaled on empty input. Promoted to pipefail with a missed sample treated as a skip; verified busybox ash in docker:27-cli supports it. Also: capture-fixtures.sh's `jq . || cat` ran cat after jq had consumed stdin, silently writing truncated fixtures; deploy.sh/deploy-stack.sh `# shellcheck source=` paths corrected for the monorepo layout.	2026-07-23 22:26:05 +00:00
Emi Griffith	ae1d9bb534	Subtree-merge thermograph-infra (origin/main) into infra/ ... git-subtree-dir: infra git-subtree-mainline: d6df04eab2 git-subtree-split: 99b4b3f78d	2026-07-22 22:01:11 -07:00