7ed375f32e
`grep -oE "^[A-Z_]+="` cannot match a key name containing a digit. This estate has exactly six such names — all the *_S3_* keys — so a prod audit returned 26 keys against 32 real ones, and reported the S3 and lake credentials as missing from both live hosts. They were present the whole time. The phantom was independently reproduced twice by re-running the same pattern, which is what made it convincing, and it was briefly recorded as one of two root causes of the lake being unqueryable (#56). That issue has one cause: the missing duckdb-lake image. An audit that under-reports is worse than no audit. A missing-secret finding sends someone to provision a credential that already exists, and in a rotation tool it would justify writing over one. Fixed to `^[A-Z][A-Z0-9_]*=` in both the skill and key_gaps.py's docstring. The match still stops at the `=`, so no value is read — that property is the reason this grep exists rather than a parser. Centralis's secrets_gaps carried the same bug and is fixed separately, with a regression test. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
4.6 KiB
| name | description |
|---|---|
| key-gaps | Audit which secret keys are missing, partially configured, or drifting across Thermograph environments (prod, beta, dev) — comparing the live /etc/thermograph.env on each box and/or the SOPS vault files (deploy/secrets/*.yaml). Use when asked about "key gaps", "missing keys/secrets", "what's not set", cross-environment secret drift, or as a pre-deploy / pre-cutover safety check. Reads key NAMES only, never values. |
Key-gap audit
Reports, per environment and across them, where secret keys are missing or inconsistent — with special weight on the failure modes that bite silently here.
Run the deterministic script; don't eyeball it:
python3 .claude/skills/key-gaps/key_gaps.py <env>=<source> [<env>=<source> ...]
<source> is either a SOPS file (deploy/secrets/prod.yaml) or a key-list
file (one KEY per line, or KEY=/KEY: lines). Keys are readable in a SOPS file
even encrypted, so no age key or decryption is needed. Exit status is non-zero if
any CRITICAL or required gap is found (so it also works as a CI/pre-deploy gate).
Audit the SOPS vault (offline, no SSH)
python3 .claude/skills/key-gaps/key_gaps.py \
prod=deploy/secrets/prod.yaml beta=deploy/secrets/beta.yaml
Audit the LIVE boxes (read-only, key names only)
Gather the key names over SSH (never the values), then audit. Hosts/keys per INFRA.md:
K=~/.ssh/thermograph_agent_ed25519
# sudo: /etc/thermograph.env is root-owned (0640). On prod `agent` can read it
# directly too, but sudo works uniformly on both boxes.
#
# The character class must allow DIGITS. This was `^[A-Z_]+=` until 2026-07-24,
# which silently skipped every key whose name contains a digit — in this estate
# that is all six *_S3_* keys. Prod audited as 26 keys against 32 real ones, and
# the audit reported the S3 credentials as missing from both hosts. They were
# present the whole time, and the phantom was independently "confirmed" twice by
# people re-running the same pattern. An audit that under-reports is worse than
# no audit: the invented finding sends someone to provision a credential that
# already exists, and it was briefly recorded as a root cause of a real bug.
# The match still stops at the `=`, so no value is ever read.
ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/prod.keys # prod
ssh -i $K agent@75.119.132.91 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/beta.keys # beta
python3 .claude/skills/key-gaps/key_gaps.py prod=/tmp/prod.keys beta=/tmp/beta.keys
Verify a SOPS cutover matches live
Before flipping a host onto the vault, confirm the rendered set equals the live set — mix the two source types for the same box:
python3 .claude/skills/key-gaps/key_gaps.py live=/tmp/prod.keys vault=deploy/secrets/prod.yaml
The Cross-environment drift section should report none for tracked keys.
Reading the output
- CRITICAL — a required, self-generating secret is missing (
THERMOGRAPH_AUTH_SECRET,THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY). Missing here is the worst case: the app silently mints a new value on boot, invalidating every login session and push subscription. Fix before deploying. - MISSING — a required secret is absent (
POSTGRES_PASSWORD,REGISTRY_TOKEN). - Feature GAP — a feature is partially configured (e.g.
discord-account-linkinghas the app id but not the client secret), so it's silently broken. A fully-unset feature is "off", not a gap. - Dependent-key GAP — an optional key is set but the other key it needs to do
anything isn't (e.g.
THERMOGRAPH_DISCORD_WEATHER_CHANNELset with noTHERMOGRAPH_DISCORD_BOT_TOKEN— discord.py gates the post on both). Asymmetric, unlike a feature group: the prerequisite key is fine set alone (the bot token alone already enables DMs). - drift — a tracked key is in some environments but not others; often intentional, but worth a glance.
Keeping it current
The secret manifest (which keys are required / self-generating / grouped into features /
dependent on another key) lives at the top of key_gaps.py in REQUIRED,
OPTIONAL_SELF_GEN, FEATURE_GROUPS, DEPENDENT_OPTIONAL, and STANDALONE_OPTIONAL.
When a new THERMOGRAPH_* credential is added to deploy/thermograph.env.example /
deploy/secrets/, add it there too — check the code path that reads it (os.environ.get
call site) to see whether it's truly required, self-generating, part of an all-or-nothing
feature, or only meaningful alongside another key, rather than guessing from the name.