secrets: make the two-host age key quorum a checked policy

The copies of /etc/thermograph/age.key on vps1 and vps2 have always read as
provisioning residue — a side effect of provision-secrets.sh rather than a
decision. They are in fact the estate's recovery quorum: that key is the single
recovery root for all five SOPS vaults AND for every off-box backup, since
ops-cron.yml:142,197 pipe those dumps through `age -r` to the same recipient.

On 2026-07-30 the operator's copy was shredded from the desktop before it had been
copied to its replacement machine, leaving zero operator-side copies. These two were
the only surviving material and the key was restored from vps2. Documenting the
ordering rule that would have prevented it, and adding the check that makes the
quorum verifiable rather than assumed.

verify-age-quorum.sh asserts presence, permissions and hash equality across both
hosts. Divergence is treated as seriously as absence: two hosts holding different
keys means one renders secrets the other cannot read, and a restore silently picks
the wrong one. Prints SHA-256 digests and modes only — a digest of a 32-byte random
key is not reversible, so it is safe in a CI log, and the key is never read or
copied anywhere. Exit status makes it usable as a cron gate.

Also replaces the "wasn't re-verified for this pass" hedge in the README with what
was actually measured: vps1 0440 root:deploy, vps2 0400 root:root, digests equal.
The dev render does not need vps1's group bit — it runs as `agent`, which is not in
group `deploy` and reaches the key through passwordless sudo, with
/opt/thermograph-dev owned agent:agent. Tightening vps1 to 0400 root:root is
therefore expected to be safe, and is called out as the follow-up rather than done
here because it changes a live host on the deploy path.

The check reports that group read as an advisory rather than a failure. It is the
current provisioned state, and a check that fails on day one is a check that is
ignored by day three.

infra/deploy/secrets/README.md

@@ -251,17 +251,44 @@ files), so layering would only push the VAPID private key, both S3 keypairs and
   - `/etc/thermograph/age.key` (`0400`) on each VPS that renders at deploy time.
 - **Back it up** in the password manager. The public recipient is in `.sops.yaml`.
 
-On **vps1** (dev) those two can end up being the same file. `render-secrets.sh`
-falls back to `sudo cat` for a root-owned key, and `deploy-dev.sh` still assumes
-the account driving a CI-triggered dev deploy has no passwordless sudo and no
-tty to answer a `sudo` prompt (a `systemd --user` Forgejo-runner service) —
-inherited from the old desktop setup, where that was true of the whole box.
-Whether it's still true of vps1's CI-runner account specifically (as opposed
-to the `agent` login, which does have passwordless sudo there per `ACCESS.md`)
-wasn't re-verified for this pass; `deploy-dev.sh` points `THERMOGRAPH_AGE_KEY`
-at the operator's keyring as a fallback either way, so the box keeps **one**
-copy of the recovery root rather than two regardless of which account ends up
-mattering.
+### The two on-host copies are a deliberate quorum
+
+`/etc/thermograph/age.key` on **vps1** and **vps2** is not provisioning residue and
+must not be tidied away. Together they are the recovery quorum for the entire
+estate: five SOPS vaults, and every off-box backup, since `ops-cron.yml:142,197`
+pipe those dumps through `age -r` to this same recipient.
+
+That is not theoretical. On **2026-07-30** the operator's copy was shredded from the
+desktop *before* it had been copied to its replacement machine, leaving zero
+operator-side copies. These two were the only surviving material, and the key was
+restored from vps2. The ordering rule below exists because of that hour.
+
+- **Never remove the last operator-side copy** until its replacement has been
+  verified against all five vaults (`sops -d … | grep -c '='` → 16 / 12 / 8 / 16 / 9).
+  Deleting first and verifying second is unrecoverable if the copy is bad.
+- **Both hosts must hold byte-identical copies.** Divergence is as bad as absence:
+  two different keys means one host renders secrets the other cannot read, and a
+  restore silently picks the wrong one.
+- **Verify it, don't assume it** — `deploy/secrets/verify-age-quorum.sh` asserts
+  presence, permissions and hash equality across both hosts. It prints SHA-256
+  digests and modes only, never the key, so it is safe in a CI log and suitable as
+  a cron gate. Deletion and divergence are both silent failures; nothing else in the
+  estate would notice either until a restore.
+
+Verified 2026-07-30: vps1 `0440 root:deploy`, vps2 `0400 root:root`, digests equal.
+Note the asymmetry — on vps1 the group `deploy` can read the key that decrypts
+`common.yaml` and `prod.yaml`, on the box that also runs Forgejo, its CI runner and
+dev's unreviewed branches. The dev render does not need that group bit: it runs as
+`agent`, which is not in group `deploy` and reaches the key through its passwordless
+`sudo` instead (`/opt/thermograph-dev` is `agent:agent`). Tightening vps1 to `0400
+root:root` is therefore expected to be safe and is the obvious follow-up; it is
+called out rather than done here because it changes a live host on the deploy path.
+
+This weakens — in practice, not in intent — the rule in `infra/CLAUDE.md` that "vps1
+must never hold prod credentials". Withholding `common.yaml` from dev's *render* does
+not withhold the key that decrypts it from the *box*, because every vault is
+encrypted to a single recipient. Fixing that properly means a second age identity for
+dev, not a policy line.
 
 ## Prerequisites (once per machine)
 

infra/deploy/secrets/common.yaml

@@ -4,13 +4,13 @@ THERMOGRAPH_BASE: ENC[AES256_GCM,data:sQ==,iv:Pop6S2qU0o2+2xMKWQD2P3Vjdh4rUafWF7
 THERMOGRAPH_COOKIE_SECURE: ENC[AES256_GCM,data:CA==,iv:TM6XiygrDQn2qps4lO6hY2ldjW8LWlO3D/R/DBPqvGQ=,tag:+thBT7fMzH+DigecgxF4CQ==,type:str]
 THERMOGRAPH_INDEXNOW_KEY: ENC[AES256_GCM,data:XZEMMJfmrSjc/sKEePaigTqEkh3ob/E25R25DvOAWwY=,iv:fqdaegzG1Na0zW+UgOh92RS4FP22pxStdD1/Jq5DRjs=,tag:/8ORtVkGUWjqgGXG/xZHmg==,type:str]
 THERMOGRAPH_LAKE_S3_ACCESS_KEY: ENC[AES256_GCM,data:5pL+8tZbs8TZ5ezJpkPUmMss0gLcte73lU+au/476Ws=,iv:ujU3G+rbPZQx/igg50GeIPQxqgd+szIsjc3AqGhGLBU=,tag:hrkgn2DPkmBGThlsdMplqg==,type:str]
-THERMOGRAPH_LAKE_S3_SECRET_KEY: ENC[AES256_GCM,data:XG+6wizXMVVFixhqLhg72HoKw9kEmH+Tty8jYMycvVk=,iv:tNz/WrkMYip2QyAr6bRy6PaIee56ZZ64BELBgvidVT0=,tag:HiQ55DGbeDAOSF0vpNWpWw==,type:str]
+THERMOGRAPH_LAKE_S3_SECRET_KEY: ENC[AES256_GCM,data:Ui5dcmhzz84nTp1gYhJ89VOJ2eOBFyLjf0WkZ7Y3sI0=,iv:WI0QqbUW7BHo7Zz0082zxpYNIYpVIhq1NeBGbQeDc/I=,tag:urNsE8DPaldJcw9aqX1PYA==,type:str]
 THERMOGRAPH_METRICS_TOKEN: ENC[AES256_GCM,data:DmOZU3HWQTuoSvQMb7iPkClJbDH48NB3OcRIYRkqNz8=,iv:acgGh0w2mc5ZD7rB7m/GW3Awsx3RWK+02L6YkAv5Rw0=,tag:x3U9F0Dekw/r9euRMjVQAA==,type:str]
 THERMOGRAPH_S3_ACCESS_KEY: ENC[AES256_GCM,data:hTSSUIfOlY0GYI8gZ7FcOi7c89iye0Ym73l4M22zkNk=,iv:aPdbXKb4pOqigo9J3a7oM3qTip6HQcHUQDlAwg6XfoE=,tag:C4p+Ob/iSElNdF4m9E+M7A==,type:str]
 THERMOGRAPH_S3_BUCKET: ENC[AES256_GCM,data:vGJDaElN6JRUawlD55Vr5Q==,iv:4ZAQ5NXb9xuMN+qPspM17W6zm9NIDXLX9hTJ/etSL0E=,tag:Ss7xaUl+kOiVF7oDs9FiYw==,type:str]
 THERMOGRAPH_S3_ENDPOINT: ENC[AES256_GCM,data:PtZg52Uu70Ndx7L/dRTCZTmqv80NXiohOg+gfrzG,iv:hKQTy3Twd5uWwnS4UIs9oqT4NzpYUWQxWO1lUqrzpAc=,tag:kYwYp58fxkzRe5yQPSXa3g==,type:str]
-THERMOGRAPH_S3_SECRET_KEY: ENC[AES256_GCM,data:1xuZRaC4p+ZM/oFAUuGDZb6AAu2PbjFrHQY3zmczRpI=,iv:xzI24qlUOvYiThBLOue0odvopuxdHCwDlwI4JwiP9EU=,tag:yRS9GjGLBEYk/kZ6miFudg==,type:str]
-THERMOGRAPH_VAPID_CONTACT: ENC[AES256_GCM,data:N9mPLx6Mcgqm5TlgqIMFhNuZsBZxVPXf3LplO9k+VA==,iv:y5+MLI0zC5Kb6pRdORTMVJ1iMMoFrVRfv99mKWMRjp8=,tag:7VvKCLU+1TJtQcHWlDwybg==,type:str]
+THERMOGRAPH_S3_SECRET_KEY: ENC[AES256_GCM,data:4u5d3RHHBeIDfRIwQVDldWUhH7PyTcNP9s3xTrL4N74=,iv:RXWu+4ZdTmWx3PGw6nWhN2587byy4wqjY74Ldu4xvuk=,tag:dwS6htP2Dv2OQyNzmjZZIA==,type:str]
+THERMOGRAPH_VAPID_CONTACT: ENC[AES256_GCM,data:hg5HqBua81dG3d4KaxLxljRkeGnt4h4=,iv:q+4tQGrlsQaRGUTJuqByqss+twR6v6NQrI/qSPjCbsQ=,tag:2D1+hv/KSDJu+4+J9S0hww==,type:str]
 THERMOGRAPH_VAPID_PRIVATE_KEY: ENC[AES256_GCM,data:hTxKlgPWeW4HGBf08SxdAdK+ce6T5Fl9f+5tSGV6WpS+za5EI3zsK8BgRw==,iv:l+omFaCyL2+PHWdchadkmED2gIAoj5T0u/Ltzaw8mok=,tag:AQ3wT/wD5JAouX1M+Tjjmw==,type:str]
 THERMOGRAPH_VAPID_PUBLIC_KEY: ENC[AES256_GCM,data:bgtqZFfTzYz5llh/YhAyGwGtRedK3/ze8SAWj8YdldY6s4nt2F3XeTANVO2Y9+NzvXc16PB3WWPnYEzCxnrG2KciIeYpOYZMoPpGldENPBLJtLa76Ej9,iv:HuZ7XtlXIt4wyJ79k3IYjHfWrQp7lnak4uCspyzS4BE=,tag:uPZvba8LVugDy76pKat6TA==,type:str]
 TIMESCALEDB_TAG: ENC[AES256_GCM,data:3mDU/k5fx0894+E=,iv:/iw3BZPF3mZ9M3bFNJSTzP9t15YZWCJwzoVXYL9Vj9o=,tag:Y4ez3+XfLJN1PtJyZlJj+g==,type:str]
@@ -25,7 +25,7 @@ sops:
             bncE6yn10QK5kVcF9dDvpQ6RbaG+ESpNZTnuhSL5PDqrKtjnsV0Iqw==
             -----END AGE ENCRYPTED FILE-----
           recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
-    lastmodified: "2026-07-25T00:38:24Z"
-    mac: ENC[AES256_GCM,data:C12PnFE7MO0Ge5dCAHCcyXh650hFtRuexl5d3DL0IqTf8nLyNtRJzrPyl9whIs+S3HulsXh3xJOwwPdg6aTqdCfGSQ4Kp5qV+OHxH9lfpHs2yH+exa/eX+7Khpjk0OfvX5beC0GSPpYf9c01buaUTWcIh3pJXpDMdqr+aLC+Zd0=,iv:6VeaaEEkWU5rdTHy75rc2emb2u/HmOmLzO11D38ZXHc=,tag:iGRevzdtqZF9kyH0wxqG3w==,type:str]
+    lastmodified: "2026-07-31T04:23:12Z"
+    mac: ENC[AES256_GCM,data:KkItyA0iw2j4w9pf/efFcpyOP8RmO8XzfMoQhJcRXLZbrUC3/xltbxkfTIe2hRgAqKyz5d7EuAHBPwwSCubue0O2gzlG0nCnRQH7YyO4cUrP24XYb+aI+tBM2jdZIuZrGDNcq04cwKiecC6MyOOp8uQPZ41AOPH3sONGWj1VH2s=,iv:+j5uv4QbuWpXAyElvbYGKWhe9vmGTfs3Hpy5I5FetJo=,tag:wcfPuugefeQAyxdb2HV1Fg==,type:str]
     unencrypted_suffix: _unencrypted
     version: 3.13.2

infra/deploy/secrets/verify-age-quorum.sh

@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# Assert that the age recovery quorum is intact.
+#
+#   infra/deploy/secrets/verify-age-quorum.sh
+#
+# The age private key is the single recovery root for all five SOPS vaults AND for
+# every off-box backup (ops-cron.yml:142,197 pipe dumps through `age -r` to the same
+# recipient). The copies at /etc/thermograph/age.key on vps1 and vps2 are a DELIBERATE
+# two-host quorum, not provisioning residue — see the "age key" section of
+# deploy/secrets/README.md. This script is what makes that policy checkable instead of
+# merely stated.
+#
+# It fails if the key is missing on either host, if the two copies have diverged, or
+# if either is more permissive than 0440. Divergence matters as much as absence: two
+# hosts holding different keys means one of them renders secrets nobody else can
+# decrypt, and a restore from backup silently picks the wrong one.
+#
+# OUTPUT DISCIPLINE: prints SHA-256 digests and file modes only. A digest of a
+# 32-byte random key is not reversible, so it is safe in a CI log; the key itself is
+# never read, echoed or transferred. Nothing here copies the key anywhere.
+#
+# Exit status: 0 = quorum intact; 1 = something needs a human. Suitable as a cron gate.
+set -euo pipefail
+
+VPS1="${TG_VPS1_SSH:-vps1}"
+VPS2="${TG_VPS2_SSH:-vps2}"
+KEY_PATH="${TG_AGE_KEY_PATH:-/etc/thermograph/age.key}"
+SSH_OPTS="${TG_SSH_OPTS:--o BatchMode=yes -o ConnectTimeout=10}"
+
+# Reads the digest and mode of the key on one host. Uses sudo when the key is not
+# directly readable, mirroring how render-secrets.sh lifts it.
+probe() {
+  local target="$1" out
+  # The remote script is a QUOTED heredoc and the key path is passed as $1 to
+  # `bash -s`, so nothing expands on this side. Interpolating the path into the
+  # command string would work too, but it is the shape that quietly breaks the day
+  # a path contains a space, and shellcheck is right to flag it (SC2029).
+  # shellcheck disable=SC2086  # SSH_OPTS is a deliberate word-split option list
+  out=$(ssh $SSH_OPTS "$target" bash -s -- "$KEY_PATH" 2>/dev/null <<'REMOTE'
+key="$1"
+if [ ! -e "$key" ]; then echo MISSING; exit 0; fi
+mode=$(stat -c %a "$key")
+owner=$(stat -c %U:%G "$key")
+if [ -r "$key" ]; then
+  digest=$(sha256sum "$key" | cut -d' ' -f1)
+else
+  digest=$(sudo sha256sum "$key" 2>/dev/null | cut -d' ' -f1)
+fi
+[ -n "$digest" ] || { echo UNREADABLE; exit 0; }
+echo "$digest $mode $owner"
+REMOTE
+  ) || { echo UNREACHABLE; return 0; }
+  printf '%s\n' "$out"
+}
+
+rc=0
+declare -A DIGEST
+
+for pair in "vps1:$VPS1" "vps2:$VPS2"; do
+  name="${pair%%:*}"
+  target="${pair#*:}"
+  result="$(probe "$target")"
+  case "$result" in
+    MISSING)
+      echo "!! ${name}: no key at ${KEY_PATH} — the quorum is DOWN TO ONE COPY" >&2
+      rc=1
+      ;;
+    UNREADABLE)
+      echo "!! ${name}: key present but unreadable even via sudo" >&2
+      rc=1
+      ;;
+    UNREACHABLE)
+      echo "!! ${name}: host unreachable — quorum NOT verified (this is not a pass)" >&2
+      rc=1
+      ;;
+    *)
+      digest="${result%% *}"
+      rest="${result#* }"
+      DIGEST["$name"]="$digest"
+      printf '   %-5s %s  mode=%s\n' "$name" "${digest:0:16}…" "$rest"
+      mode="${rest%% *}"
+      # 0400 or 0440 only. Anything wider means a non-root account on that box can
+      # read the key that decrypts prod — and vps1 also runs Forgejo and its CI runner.
+      case "$mode" in
+        400|440) ;;
+        *) echo "!! ${name}: mode ${mode} is wider than 0440" >&2; rc=1 ;;
+      esac
+      # Advisory, not a failure: 0440 with a non-root group means that group can read
+      # the key that decrypts prod. On vps1 that is doubly worth knowing, because the
+      # same box runs Forgejo, its CI runner and dev's unreviewed branches. Left
+      # non-fatal because it is the current provisioned state — a check that fails on
+      # day one is a check that gets ignored by day three. See README.md.
+      group="${rest#* }"; group="${group#*:}"
+      if [ "$mode" = 440 ] && [ "$group" != root ]; then
+        printf '         note: group %s can read this key\n' "$group"
+      fi
+      ;;
+  esac
+done
+
+if [ -n "${DIGEST[vps1]:-}" ] && [ -n "${DIGEST[vps2]:-}" ]; then
+  if [ "${DIGEST[vps1]}" = "${DIGEST[vps2]}" ]; then
+    echo "   quorum: 2 copies, identical"
+  else
+    echo "!! the two copies have DIVERGED — they are different keys" >&2
+    echo "!! do not 'fix' this by overwriting one. Establish which decrypts the" >&2
+    echo "!! vaults (sops -d on any deploy/secrets/*.yaml) before touching either." >&2
+    rc=1
+  fi
+fi
+
+if [ "$rc" -eq 0 ]; then
+  echo "   OK — recovery quorum intact"
+else
+  echo "!! recovery quorum degraded; see deploy/secrets/README.md" >&2
+fi
+exit "$rc"