--- name: key-gaps description: Audit which secret keys are missing, partially configured, or drifting across Thermograph environments (prod, beta, dev) — comparing the live /etc/thermograph.env on each box and/or the SOPS vault files (deploy/secrets/*.yaml). Use when asked about "key gaps", "missing keys/secrets", "what's not set", cross-environment secret drift, or as a pre-deploy / pre-cutover safety check. Reads key NAMES only, never values. --- # Key-gap audit Reports, per environment and across them, where secret **keys** are missing or inconsistent — with special weight on the failure modes that bite silently here. Run the deterministic script; don't eyeball it: ``` python3 .claude/skills/key-gaps/key_gaps.py = [= ...] ``` `` is either a **SOPS file** (`deploy/secrets/prod.yaml`) or a **key-list file** (one `KEY` per line, or `KEY=`/`KEY:` lines). Keys are readable in a SOPS file even encrypted, so **no age key or decryption is needed**. Exit status is non-zero if any CRITICAL or required gap is found (so it also works as a CI/pre-deploy gate). ## Audit the SOPS vault (offline, no SSH) ``` python3 .claude/skills/key-gaps/key_gaps.py \ prod=deploy/secrets/prod.yaml beta=deploy/secrets/beta.yaml ``` ## Audit the LIVE boxes (read-only, key names only) Gather the key names over SSH (never the values), then audit. Hosts per the root `CLAUDE.md` topology: - **vps2** (`169.58.46.181`) hosts BOTH prod and beta as separate Swarm stacks — TWO live env files on the same box: `/etc/thermograph.env` (prod) and `/etc/thermograph-beta.env` (beta). "The live env file per environment" is no longer one path per host; it's one path per environment, and both environments' files live on this one host. Get both in the same SSH round trip, or two separate commands against the same host — never assume one host means one file here. - **vps1** (`75.119.132.91`) hosts dev, at the same `/etc/thermograph.env` path (it's a different host, so no collision with prod's file of the same name). ```sh K=~/.ssh/thermograph_agent_ed25519 # sudo: the env files are root-owned (0640). On these boxes `agent` can often # read them directly too, but sudo works uniformly regardless. # # The character class must allow DIGITS. This was `^[A-Z_]+=` until 2026-07-24, # which silently skipped every key whose name contains a digit — in this estate # that is all six *_S3_* keys. Prod audited as 26 keys against 32 real ones, and # the audit reported the S3 credentials as missing from both hosts. They were # present the whole time, and the phantom was independently "confirmed" twice by # people re-running the same pattern. An audit that under-reports is worse than # no audit: the invented finding sends someone to provision a credential that # already exists, and it was briefly recorded as a root cause of a real bug. # The match still stops at the `=`, so no value is ever read. ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/prod.keys # vps2: prod ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph-beta.env' > /tmp/beta.keys # vps2: beta ssh -i $K agent@75.119.132.91 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/dev.keys # vps1: dev python3 .claude/skills/key-gaps/key_gaps.py prod=/tmp/prod.keys beta=/tmp/beta.keys dev=/tmp/dev.keys ``` ## Verify a SOPS cutover matches live Before flipping a host onto the vault, confirm the rendered set equals the live set — mix the two source types for the same box: ``` python3 .claude/skills/key-gaps/key_gaps.py live=/tmp/prod.keys vault=deploy/secrets/prod.yaml ``` The `Cross-environment drift` section should report **none** for tracked keys. ## Reading the output - **CRITICAL** — a required, *self-generating* secret is missing (`THERMOGRAPH_AUTH_SECRET`, `THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY`). Missing here is the worst case: the app silently mints a new value on boot, invalidating every login session and push subscription. Fix before deploying. - **MISSING** — a required secret is absent (`POSTGRES_PASSWORD`, `REGISTRY_TOKEN`). - **Feature GAP** — a feature is *partially* configured (e.g. `discord-account-linking` has the app id but not the client secret), so it's silently broken. A fully-unset feature is "off", not a gap. - **Dependent-key GAP** — an optional key is set but the *other* key it needs to do anything isn't (e.g. `THERMOGRAPH_DISCORD_WEATHER_CHANNEL` set with no `THERMOGRAPH_DISCORD_BOT_TOKEN` — discord.py gates the post on both). Asymmetric, unlike a feature group: the prerequisite key is fine set alone (the bot token alone already enables DMs). - **drift** — a tracked key is in some environments but not others; often intentional, but worth a glance. ## Keeping it current The secret manifest (which keys are required / self-generating / grouped into features / dependent on another key) lives at the top of `key_gaps.py` in `REQUIRED`, `OPTIONAL_SELF_GEN`, `FEATURE_GROUPS`, `DEPENDENT_OPTIONAL`, and `STANDALONE_OPTIONAL`. When a new `THERMOGRAPH_*` credential is added to `deploy/thermograph.env.example` / `deploy/secrets/`, add it there too — check the code path that reads it (`os.environ.get` call site) to see whether it's truly required, self-generating, part of an all-or-nothing feature, or only meaningful alongside another key, rather than guessing from the name.