Merge pull request 'secrets: make the two-host age key quorum a checked policy' (#137) from feat/age-recovery-quorum into dev

infra/deploy/secrets/README.md

@@ -251,17 +251,44 @@ files), so layering would only push the VAPID private key, both S3 keypairs and
   - `/etc/thermograph/age.key` (`0400`) on each VPS that renders at deploy time.
 - **Back it up** in the password manager. The public recipient is in `.sops.yaml`.
 
-On **vps1** (dev) those two can end up being the same file. `render-secrets.sh`
-falls back to `sudo cat` for a root-owned key, and `deploy-dev.sh` still assumes
-the account driving a CI-triggered dev deploy has no passwordless sudo and no
-tty to answer a `sudo` prompt (a `systemd --user` Forgejo-runner service) —
-inherited from the old desktop setup, where that was true of the whole box.
-Whether it's still true of vps1's CI-runner account specifically (as opposed
-to the `agent` login, which does have passwordless sudo there per `ACCESS.md`)
-wasn't re-verified for this pass; `deploy-dev.sh` points `THERMOGRAPH_AGE_KEY`
-at the operator's keyring as a fallback either way, so the box keeps **one**
-copy of the recovery root rather than two regardless of which account ends up
-mattering.
+### The two on-host copies are a deliberate quorum
+
+`/etc/thermograph/age.key` on **vps1** and **vps2** is not provisioning residue and
+must not be tidied away. Together they are the recovery quorum for the entire
+estate: five SOPS vaults, and every off-box backup, since `ops-cron.yml:142,197`
+pipe those dumps through `age -r` to this same recipient.
+
+That is not theoretical. On **2026-07-30** the operator's copy was shredded from the
+desktop *before* it had been copied to its replacement machine, leaving zero
+operator-side copies. These two were the only surviving material, and the key was
+restored from vps2. The ordering rule below exists because of that hour.
+
+- **Never remove the last operator-side copy** until its replacement has been
+  verified against all five vaults (`sops -d … | grep -c '='` → 16 / 12 / 8 / 16 / 9).
+  Deleting first and verifying second is unrecoverable if the copy is bad.
+- **Both hosts must hold byte-identical copies.** Divergence is as bad as absence:
+  two different keys means one host renders secrets the other cannot read, and a
+  restore silently picks the wrong one.
+- **Verify it, don't assume it** — `deploy/secrets/verify-age-quorum.sh` asserts
+  presence, permissions and hash equality across both hosts. It prints SHA-256
+  digests and modes only, never the key, so it is safe in a CI log and suitable as
+  a cron gate. Deletion and divergence are both silent failures; nothing else in the
+  estate would notice either until a restore.
+
+Verified 2026-07-30: vps1 `0440 root:deploy`, vps2 `0400 root:root`, digests equal.
+Note the asymmetry — on vps1 the group `deploy` can read the key that decrypts
+`common.yaml` and `prod.yaml`, on the box that also runs Forgejo, its CI runner and
+dev's unreviewed branches. The dev render does not need that group bit: it runs as
+`agent`, which is not in group `deploy` and reaches the key through its passwordless
+`sudo` instead (`/opt/thermograph-dev` is `agent:agent`). Tightening vps1 to `0400
+root:root` is therefore expected to be safe and is the obvious follow-up; it is
+called out rather than done here because it changes a live host on the deploy path.
+
+This weakens — in practice, not in intent — the rule in `infra/CLAUDE.md` that "vps1
+must never hold prod credentials". Withholding `common.yaml` from dev's *render* does
+not withhold the key that decrypts it from the *box*, because every vault is
+encrypted to a single recipient. Fixing that properly means a second age identity for
+dev, not a policy line.
 
 ## Prerequisites (once per machine)
 

infra/deploy/secrets/verify-age-quorum.sh

@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# Assert that the age recovery quorum is intact.
+#
+#   infra/deploy/secrets/verify-age-quorum.sh
+#
+# The age private key is the single recovery root for all five SOPS vaults AND for
+# every off-box backup (ops-cron.yml:142,197 pipe dumps through `age -r` to the same
+# recipient). The copies at /etc/thermograph/age.key on vps1 and vps2 are a DELIBERATE
+# two-host quorum, not provisioning residue — see the "age key" section of
+# deploy/secrets/README.md. This script is what makes that policy checkable instead of
+# merely stated.
+#
+# It fails if the key is missing on either host, if the two copies have diverged, or
+# if either is more permissive than 0440. Divergence matters as much as absence: two
+# hosts holding different keys means one of them renders secrets nobody else can
+# decrypt, and a restore from backup silently picks the wrong one.
+#
+# OUTPUT DISCIPLINE: prints SHA-256 digests and file modes only. A digest of a
+# 32-byte random key is not reversible, so it is safe in a CI log; the key itself is
+# never read, echoed or transferred. Nothing here copies the key anywhere.
+#
+# Exit status: 0 = quorum intact; 1 = something needs a human. Suitable as a cron gate.
+set -euo pipefail
+
+VPS1="${TG_VPS1_SSH:-vps1}"
+VPS2="${TG_VPS2_SSH:-vps2}"
+KEY_PATH="${TG_AGE_KEY_PATH:-/etc/thermograph/age.key}"
+SSH_OPTS="${TG_SSH_OPTS:--o BatchMode=yes -o ConnectTimeout=10}"
+
+# Reads the digest and mode of the key on one host. Uses sudo when the key is not
+# directly readable, mirroring how render-secrets.sh lifts it.
+probe() {
+  local target="$1" out
+  # The remote script is a QUOTED heredoc and the key path is passed as $1 to
+  # `bash -s`, so nothing expands on this side. Interpolating the path into the
+  # command string would work too, but it is the shape that quietly breaks the day
+  # a path contains a space, and shellcheck is right to flag it (SC2029).
+  # shellcheck disable=SC2086  # SSH_OPTS is a deliberate word-split option list
+  out=$(ssh $SSH_OPTS "$target" bash -s -- "$KEY_PATH" 2>/dev/null <<'REMOTE'
+key="$1"
+if [ ! -e "$key" ]; then echo MISSING; exit 0; fi
+mode=$(stat -c %a "$key")
+owner=$(stat -c %U:%G "$key")
+if [ -r "$key" ]; then
+  digest=$(sha256sum "$key" | cut -d' ' -f1)
+else
+  digest=$(sudo sha256sum "$key" 2>/dev/null | cut -d' ' -f1)
+fi
+[ -n "$digest" ] || { echo UNREADABLE; exit 0; }
+echo "$digest $mode $owner"
+REMOTE
+  ) || { echo UNREACHABLE; return 0; }
+  printf '%s\n' "$out"
+}
+
+rc=0
+declare -A DIGEST
+
+for pair in "vps1:$VPS1" "vps2:$VPS2"; do
+  name="${pair%%:*}"
+  target="${pair#*:}"
+  result="$(probe "$target")"
+  case "$result" in
+    MISSING)
+      echo "!! ${name}: no key at ${KEY_PATH} — the quorum is DOWN TO ONE COPY" >&2
+      rc=1
+      ;;
+    UNREADABLE)
+      echo "!! ${name}: key present but unreadable even via sudo" >&2
+      rc=1
+      ;;
+    UNREACHABLE)
+      echo "!! ${name}: host unreachable — quorum NOT verified (this is not a pass)" >&2
+      rc=1
+      ;;
+    *)
+      digest="${result%% *}"
+      rest="${result#* }"
+      DIGEST["$name"]="$digest"
+      printf '   %-5s %s  mode=%s\n' "$name" "${digest:0:16}…" "$rest"
+      mode="${rest%% *}"
+      # 0400 or 0440 only. Anything wider means a non-root account on that box can
+      # read the key that decrypts prod — and vps1 also runs Forgejo and its CI runner.
+      case "$mode" in
+        400|440) ;;
+        *) echo "!! ${name}: mode ${mode} is wider than 0440" >&2; rc=1 ;;
+      esac
+      # Advisory, not a failure: 0440 with a non-root group means that group can read
+      # the key that decrypts prod. On vps1 that is doubly worth knowing, because the
+      # same box runs Forgejo, its CI runner and dev's unreviewed branches. Left
+      # non-fatal because it is the current provisioned state — a check that fails on
+      # day one is a check that gets ignored by day three. See README.md.
+      group="${rest#* }"; group="${group#*:}"
+      if [ "$mode" = 440 ] && [ "$group" != root ]; then
+        printf '         note: group %s can read this key\n' "$group"
+      fi
+      ;;
+  esac
+done
+
+if [ -n "${DIGEST[vps1]:-}" ] && [ -n "${DIGEST[vps2]:-}" ]; then
+  if [ "${DIGEST[vps1]}" = "${DIGEST[vps2]}" ]; then
+    echo "   quorum: 2 copies, identical"
+  else
+    echo "!! the two copies have DIVERGED — they are different keys" >&2
+    echo "!! do not 'fix' this by overwriting one. Establish which decrypts the" >&2
+    echo "!! vaults (sops -d on any deploy/secrets/*.yaml) before touching either." >&2
+    rc=1
+  fi
+fi
+
+if [ "$rc" -eq 0 ]; then
+  echo "   OK — recovery quorum intact"
+else
+  echo "!! recovery quorum degraded; see deploy/secrets/README.md" >&2
+fi
+exit "$rc"