7ed375f32e
`grep -oE "^[A-Z_]+="` cannot match a key name containing a digit. This estate has exactly six such names — all the *_S3_* keys — so a prod audit returned 26 keys against 32 real ones, and reported the S3 and lake credentials as missing from both live hosts. They were present the whole time. The phantom was independently reproduced twice by re-running the same pattern, which is what made it convincing, and it was briefly recorded as one of two root causes of the lake being unqueryable (#56). That issue has one cause: the missing duckdb-lake image. An audit that under-reports is worse than no audit. A missing-secret finding sends someone to provision a credential that already exists, and in a rotation tool it would justify writing over one. Fixed to `^[A-Z][A-Z0-9_]*=` in both the skill and key_gaps.py's docstring. The match still stops at the `=`, so no value is read — that property is the reason this grep exists rather than a parser. Centralis's secrets_gaps carried the same bug and is fixed separately, with a regression test. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
88 lines
4.6 KiB
Markdown
88 lines
4.6 KiB
Markdown
---
|
|
name: key-gaps
|
|
description: Audit which secret keys are missing, partially configured, or drifting across Thermograph environments (prod, beta, dev) — comparing the live /etc/thermograph.env on each box and/or the SOPS vault files (deploy/secrets/*.yaml). Use when asked about "key gaps", "missing keys/secrets", "what's not set", cross-environment secret drift, or as a pre-deploy / pre-cutover safety check. Reads key NAMES only, never values.
|
|
---
|
|
|
|
# Key-gap audit
|
|
|
|
Reports, per environment and across them, where secret **keys** are missing or
|
|
inconsistent — with special weight on the failure modes that bite silently here.
|
|
|
|
Run the deterministic script; don't eyeball it:
|
|
|
|
```
|
|
python3 .claude/skills/key-gaps/key_gaps.py <env>=<source> [<env>=<source> ...]
|
|
```
|
|
|
|
`<source>` is either a **SOPS file** (`deploy/secrets/prod.yaml`) or a **key-list
|
|
file** (one `KEY` per line, or `KEY=`/`KEY:` lines). Keys are readable in a SOPS file
|
|
even encrypted, so **no age key or decryption is needed**. Exit status is non-zero if
|
|
any CRITICAL or required gap is found (so it also works as a CI/pre-deploy gate).
|
|
|
|
## Audit the SOPS vault (offline, no SSH)
|
|
|
|
```
|
|
python3 .claude/skills/key-gaps/key_gaps.py \
|
|
prod=deploy/secrets/prod.yaml beta=deploy/secrets/beta.yaml
|
|
```
|
|
|
|
## Audit the LIVE boxes (read-only, key names only)
|
|
|
|
Gather the key names over SSH (never the values), then audit. Hosts/keys per INFRA.md:
|
|
|
|
```sh
|
|
K=~/.ssh/thermograph_agent_ed25519
|
|
# sudo: /etc/thermograph.env is root-owned (0640). On prod `agent` can read it
|
|
# directly too, but sudo works uniformly on both boxes.
|
|
#
|
|
# The character class must allow DIGITS. This was `^[A-Z_]+=` until 2026-07-24,
|
|
# which silently skipped every key whose name contains a digit — in this estate
|
|
# that is all six *_S3_* keys. Prod audited as 26 keys against 32 real ones, and
|
|
# the audit reported the S3 credentials as missing from both hosts. They were
|
|
# present the whole time, and the phantom was independently "confirmed" twice by
|
|
# people re-running the same pattern. An audit that under-reports is worse than
|
|
# no audit: the invented finding sends someone to provision a credential that
|
|
# already exists, and it was briefly recorded as a root cause of a real bug.
|
|
# The match still stops at the `=`, so no value is ever read.
|
|
ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/prod.keys # prod
|
|
ssh -i $K agent@75.119.132.91 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/beta.keys # beta
|
|
python3 .claude/skills/key-gaps/key_gaps.py prod=/tmp/prod.keys beta=/tmp/beta.keys
|
|
```
|
|
|
|
## Verify a SOPS cutover matches live
|
|
|
|
Before flipping a host onto the vault, confirm the rendered set equals the live set —
|
|
mix the two source types for the same box:
|
|
|
|
```
|
|
python3 .claude/skills/key-gaps/key_gaps.py live=/tmp/prod.keys vault=deploy/secrets/prod.yaml
|
|
```
|
|
The `Cross-environment drift` section should report **none** for tracked keys.
|
|
|
|
## Reading the output
|
|
|
|
- **CRITICAL** — a required, *self-generating* secret is missing (`THERMOGRAPH_AUTH_SECRET`,
|
|
`THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY`). Missing here is the worst case: the app
|
|
silently mints a new value on boot, invalidating every login session and push
|
|
subscription. Fix before deploying.
|
|
- **MISSING** — a required secret is absent (`POSTGRES_PASSWORD`, `REGISTRY_TOKEN`).
|
|
- **Feature GAP** — a feature is *partially* configured (e.g. `discord-account-linking`
|
|
has the app id but not the client secret), so it's silently broken. A fully-unset
|
|
feature is "off", not a gap.
|
|
- **Dependent-key GAP** — an optional key is set but the *other* key it needs to do
|
|
anything isn't (e.g. `THERMOGRAPH_DISCORD_WEATHER_CHANNEL` set with no
|
|
`THERMOGRAPH_DISCORD_BOT_TOKEN` — discord.py gates the post on both). Asymmetric,
|
|
unlike a feature group: the prerequisite key is fine set alone (the bot token alone
|
|
already enables DMs).
|
|
- **drift** — a tracked key is in some environments but not others; often intentional,
|
|
but worth a glance.
|
|
|
|
## Keeping it current
|
|
|
|
The secret manifest (which keys are required / self-generating / grouped into features /
|
|
dependent on another key) lives at the top of `key_gaps.py` in `REQUIRED`,
|
|
`OPTIONAL_SELF_GEN`, `FEATURE_GROUPS`, `DEPENDENT_OPTIONAL`, and `STANDALONE_OPTIONAL`.
|
|
When a new `THERMOGRAPH_*` credential is added to `deploy/thermograph.env.example` /
|
|
`deploy/secrets/`, add it there too — check the code path that reads it (`os.environ.get`
|
|
call site) to see whether it's truly required, self-generating, part of an all-or-nothing
|
|
feature, or only meaningful alongside another key, rather than guessing from the name.
|