diff --git a/infra/deploy/secrets/README.md b/infra/deploy/secrets/README.md index 4b85e59..7ec4bb3 100644 --- a/infra/deploy/secrets/README.md +++ b/infra/deploy/secrets/README.md @@ -251,17 +251,44 @@ files), so layering would only push the VAPID private key, both S3 keypairs and - `/etc/thermograph/age.key` (`0400`) on each VPS that renders at deploy time. - **Back it up** in the password manager. The public recipient is in `.sops.yaml`. -On **vps1** (dev) those two can end up being the same file. `render-secrets.sh` -falls back to `sudo cat` for a root-owned key, and `deploy-dev.sh` still assumes -the account driving a CI-triggered dev deploy has no passwordless sudo and no -tty to answer a `sudo` prompt (a `systemd --user` Forgejo-runner service) — -inherited from the old desktop setup, where that was true of the whole box. -Whether it's still true of vps1's CI-runner account specifically (as opposed -to the `agent` login, which does have passwordless sudo there per `ACCESS.md`) -wasn't re-verified for this pass; `deploy-dev.sh` points `THERMOGRAPH_AGE_KEY` -at the operator's keyring as a fallback either way, so the box keeps **one** -copy of the recovery root rather than two regardless of which account ends up -mattering. +### The two on-host copies are a deliberate quorum + +`/etc/thermograph/age.key` on **vps1** and **vps2** is not provisioning residue and +must not be tidied away. Together they are the recovery quorum for the entire +estate: five SOPS vaults, and every off-box backup, since `ops-cron.yml:142,197` +pipe those dumps through `age -r` to this same recipient. + +That is not theoretical. On **2026-07-30** the operator's copy was shredded from the +desktop *before* it had been copied to its replacement machine, leaving zero +operator-side copies. These two were the only surviving material, and the key was +restored from vps2. The ordering rule below exists because of that hour. + +- **Never remove the last operator-side copy** until its replacement has been + verified against all five vaults (`sops -d … | grep -c '='` → 16 / 12 / 8 / 16 / 9). + Deleting first and verifying second is unrecoverable if the copy is bad. +- **Both hosts must hold byte-identical copies.** Divergence is as bad as absence: + two different keys means one host renders secrets the other cannot read, and a + restore silently picks the wrong one. +- **Verify it, don't assume it** — `deploy/secrets/verify-age-quorum.sh` asserts + presence, permissions and hash equality across both hosts. It prints SHA-256 + digests and modes only, never the key, so it is safe in a CI log and suitable as + a cron gate. Deletion and divergence are both silent failures; nothing else in the + estate would notice either until a restore. + +Verified 2026-07-30: vps1 `0440 root:deploy`, vps2 `0400 root:root`, digests equal. +Note the asymmetry — on vps1 the group `deploy` can read the key that decrypts +`common.yaml` and `prod.yaml`, on the box that also runs Forgejo, its CI runner and +dev's unreviewed branches. The dev render does not need that group bit: it runs as +`agent`, which is not in group `deploy` and reaches the key through its passwordless +`sudo` instead (`/opt/thermograph-dev` is `agent:agent`). Tightening vps1 to `0400 +root:root` is therefore expected to be safe and is the obvious follow-up; it is +called out rather than done here because it changes a live host on the deploy path. + +This weakens — in practice, not in intent — the rule in `infra/CLAUDE.md` that "vps1 +must never hold prod credentials". Withholding `common.yaml` from dev's *render* does +not withhold the key that decrypts it from the *box*, because every vault is +encrypted to a single recipient. Fixing that properly means a second age identity for +dev, not a policy line. ## Prerequisites (once per machine) diff --git a/infra/deploy/secrets/verify-age-quorum.sh b/infra/deploy/secrets/verify-age-quorum.sh new file mode 100755 index 0000000..8bb36ba --- /dev/null +++ b/infra/deploy/secrets/verify-age-quorum.sh @@ -0,0 +1,117 @@ +#!/usr/bin/env bash +# Assert that the age recovery quorum is intact. +# +# infra/deploy/secrets/verify-age-quorum.sh +# +# The age private key is the single recovery root for all five SOPS vaults AND for +# every off-box backup (ops-cron.yml:142,197 pipe dumps through `age -r` to the same +# recipient). The copies at /etc/thermograph/age.key on vps1 and vps2 are a DELIBERATE +# two-host quorum, not provisioning residue — see the "age key" section of +# deploy/secrets/README.md. This script is what makes that policy checkable instead of +# merely stated. +# +# It fails if the key is missing on either host, if the two copies have diverged, or +# if either is more permissive than 0440. Divergence matters as much as absence: two +# hosts holding different keys means one of them renders secrets nobody else can +# decrypt, and a restore from backup silently picks the wrong one. +# +# OUTPUT DISCIPLINE: prints SHA-256 digests and file modes only. A digest of a +# 32-byte random key is not reversible, so it is safe in a CI log; the key itself is +# never read, echoed or transferred. Nothing here copies the key anywhere. +# +# Exit status: 0 = quorum intact; 1 = something needs a human. Suitable as a cron gate. +set -euo pipefail + +VPS1="${TG_VPS1_SSH:-vps1}" +VPS2="${TG_VPS2_SSH:-vps2}" +KEY_PATH="${TG_AGE_KEY_PATH:-/etc/thermograph/age.key}" +SSH_OPTS="${TG_SSH_OPTS:--o BatchMode=yes -o ConnectTimeout=10}" + +# Reads the digest and mode of the key on one host. Uses sudo when the key is not +# directly readable, mirroring how render-secrets.sh lifts it. +probe() { + local target="$1" out + # The remote script is a QUOTED heredoc and the key path is passed as $1 to + # `bash -s`, so nothing expands on this side. Interpolating the path into the + # command string would work too, but it is the shape that quietly breaks the day + # a path contains a space, and shellcheck is right to flag it (SC2029). + # shellcheck disable=SC2086 # SSH_OPTS is a deliberate word-split option list + out=$(ssh $SSH_OPTS "$target" bash -s -- "$KEY_PATH" 2>/dev/null <<'REMOTE' +key="$1" +if [ ! -e "$key" ]; then echo MISSING; exit 0; fi +mode=$(stat -c %a "$key") +owner=$(stat -c %U:%G "$key") +if [ -r "$key" ]; then + digest=$(sha256sum "$key" | cut -d' ' -f1) +else + digest=$(sudo sha256sum "$key" 2>/dev/null | cut -d' ' -f1) +fi +[ -n "$digest" ] || { echo UNREADABLE; exit 0; } +echo "$digest $mode $owner" +REMOTE + ) || { echo UNREACHABLE; return 0; } + printf '%s\n' "$out" +} + +rc=0 +declare -A DIGEST + +for pair in "vps1:$VPS1" "vps2:$VPS2"; do + name="${pair%%:*}" + target="${pair#*:}" + result="$(probe "$target")" + case "$result" in + MISSING) + echo "!! ${name}: no key at ${KEY_PATH} — the quorum is DOWN TO ONE COPY" >&2 + rc=1 + ;; + UNREADABLE) + echo "!! ${name}: key present but unreadable even via sudo" >&2 + rc=1 + ;; + UNREACHABLE) + echo "!! ${name}: host unreachable — quorum NOT verified (this is not a pass)" >&2 + rc=1 + ;; + *) + digest="${result%% *}" + rest="${result#* }" + DIGEST["$name"]="$digest" + printf ' %-5s %s mode=%s\n' "$name" "${digest:0:16}…" "$rest" + mode="${rest%% *}" + # 0400 or 0440 only. Anything wider means a non-root account on that box can + # read the key that decrypts prod — and vps1 also runs Forgejo and its CI runner. + case "$mode" in + 400|440) ;; + *) echo "!! ${name}: mode ${mode} is wider than 0440" >&2; rc=1 ;; + esac + # Advisory, not a failure: 0440 with a non-root group means that group can read + # the key that decrypts prod. On vps1 that is doubly worth knowing, because the + # same box runs Forgejo, its CI runner and dev's unreviewed branches. Left + # non-fatal because it is the current provisioned state — a check that fails on + # day one is a check that gets ignored by day three. See README.md. + group="${rest#* }"; group="${group#*:}" + if [ "$mode" = 440 ] && [ "$group" != root ]; then + printf ' note: group %s can read this key\n' "$group" + fi + ;; + esac +done + +if [ -n "${DIGEST[vps1]:-}" ] && [ -n "${DIGEST[vps2]:-}" ]; then + if [ "${DIGEST[vps1]}" = "${DIGEST[vps2]}" ]; then + echo " quorum: 2 copies, identical" + else + echo "!! the two copies have DIVERGED — they are different keys" >&2 + echo "!! do not 'fix' this by overwriting one. Establish which decrypts the" >&2 + echo "!! vaults (sops -d on any deploy/secrets/*.yaml) before touching either." >&2 + rc=1 + fi +fi + +if [ "$rc" -eq 0 ]; then + echo " OK — recovery quorum intact" +else + echo "!! recovery quorum degraded; see deploy/secrets/README.md" >&2 +fi +exit "$rc"