d138f00a20
5.4 KiB
| name | description |
|---|---|
| key-gaps | Audit which secret keys are missing, partially configured, or drifting across Thermograph environments (prod, beta, dev) — comparing the live /etc/thermograph.env on each box and/or the SOPS vault files (deploy/secrets/*.yaml). Use when asked about "key gaps", "missing keys/secrets", "what's not set", cross-environment secret drift, or as a pre-deploy / pre-cutover safety check. Reads key NAMES only, never values. |
Key-gap audit
Reports, per environment and across them, where secret keys are missing or inconsistent — with special weight on the failure modes that bite silently here.
Run the deterministic script; don't eyeball it:
python3 .claude/skills/key-gaps/key_gaps.py <env>=<source> [<env>=<source> ...]
<source> is either a SOPS file (deploy/secrets/prod.yaml) or a key-list
file (one KEY per line, or KEY=/KEY: lines). Keys are readable in a SOPS file
even encrypted, so no age key or decryption is needed. Exit status is non-zero if
any CRITICAL or required gap is found (so it also works as a CI/pre-deploy gate).
Audit the SOPS vault (offline, no SSH)
python3 .claude/skills/key-gaps/key_gaps.py \
prod=deploy/secrets/prod.yaml beta=deploy/secrets/beta.yaml
Audit the LIVE boxes (read-only, key names only)
Gather the key names over SSH (never the values), then audit. Hosts per the
root CLAUDE.md topology:
- vps2 (
169.58.46.181) hosts BOTH prod and beta as separate Swarm stacks — TWO live env files on the same box:/etc/thermograph.env(prod) and/etc/thermograph-beta.env(beta). "The live env file per environment" is no longer one path per host; it's one path per environment, and both environments' files live on this one host. Get both in the same SSH round trip, or two separate commands against the same host — never assume one host means one file here. - vps1 (
75.119.132.91) hosts dev, at the same/etc/thermograph.envpath (it's a different host, so no collision with prod's file of the same name).
K=~/.ssh/thermograph_agent_ed25519
# sudo: the env files are root-owned (0640). On these boxes `agent` can often
# read them directly too, but sudo works uniformly regardless.
#
# The character class must allow DIGITS. This was `^[A-Z_]+=` until 2026-07-24,
# which silently skipped every key whose name contains a digit — in this estate
# that is all six *_S3_* keys. Prod audited as 26 keys against 32 real ones, and
# the audit reported the S3 credentials as missing from both hosts. They were
# present the whole time, and the phantom was independently "confirmed" twice by
# people re-running the same pattern. An audit that under-reports is worse than
# no audit: the invented finding sends someone to provision a credential that
# already exists, and it was briefly recorded as a root cause of a real bug.
# The match still stops at the `=`, so no value is ever read.
ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/prod.keys # vps2: prod
ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph-beta.env' > /tmp/beta.keys # vps2: beta
ssh -i $K agent@75.119.132.91 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/dev.keys # vps1: dev
python3 .claude/skills/key-gaps/key_gaps.py prod=/tmp/prod.keys beta=/tmp/beta.keys dev=/tmp/dev.keys
Verify a SOPS cutover matches live
Before flipping a host onto the vault, confirm the rendered set equals the live set — mix the two source types for the same box:
python3 .claude/skills/key-gaps/key_gaps.py live=/tmp/prod.keys vault=deploy/secrets/prod.yaml
The Cross-environment drift section should report none for tracked keys.
Reading the output
- CRITICAL — a required, self-generating secret is missing (
THERMOGRAPH_AUTH_SECRET,THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY). Missing here is the worst case: the app silently mints a new value on boot, invalidating every login session and push subscription. Fix before deploying. - MISSING — a required secret is absent (
POSTGRES_PASSWORD,REGISTRY_TOKEN). - Feature GAP — a feature is partially configured (e.g.
discord-account-linkinghas the app id but not the client secret), so it's silently broken. A fully-unset feature is "off", not a gap. - Dependent-key GAP — an optional key is set but the other key it needs to do
anything isn't (e.g.
THERMOGRAPH_DISCORD_WEATHER_CHANNELset with noTHERMOGRAPH_DISCORD_BOT_TOKEN— discord.py gates the post on both). Asymmetric, unlike a feature group: the prerequisite key is fine set alone (the bot token alone already enables DMs). - drift — a tracked key is in some environments but not others; often intentional, but worth a glance.
Keeping it current
The secret manifest (which keys are required / self-generating / grouped into features /
dependent on another key) lives at the top of key_gaps.py in REQUIRED,
OPTIONAL_SELF_GEN, FEATURE_GROUPS, DEPENDENT_OPTIONAL, and STANDALONE_OPTIONAL.
When a new THERMOGRAPH_* credential is added to deploy/thermograph.env.example /
deploy/secrets/, add it there too — check the code path that reads it (os.environ.get
call site) to see whether it's truly required, self-generating, part of an all-or-nothing
feature, or only meaningful alongside another key, rather than guessing from the name.