thermograph/infra/.claude/skills/key-gaps/SKILL.md

---
name: key-gaps
description: Audit which secret keys are missing, partially configured, or drifting across Thermograph environments (prod, beta, dev) — comparing the live /etc/thermograph.env on each box and/or the SOPS vault files (deploy/secrets/*.yaml). Use when asked about "key gaps", "missing keys/secrets", "what's not set", cross-environment secret drift, or as a pre-deploy / pre-cutover safety check. Reads key NAMES only, never values.
---

# Key-gap audit

Reports, per environment and across them, where secret **keys** are missing or
inconsistent — with special weight on the failure modes that bite silently here.

Run the deterministic script; don't eyeball it:

```
python3 .claude/skills/key-gaps/key_gaps.py <env>=<source> [<env>=<source> ...]
```

`<source>` is either a **SOPS file** (`deploy/secrets/prod.yaml`) or a **key-list
file** (one `KEY` per line, or `KEY=`/`KEY:` lines). Keys are readable in a SOPS file
even encrypted, so **no age key or decryption is needed**. Exit status is non-zero if
any CRITICAL or required gap is found (so it also works as a CI/pre-deploy gate).

## Audit the SOPS vault (offline, no SSH)

```
python3 .claude/skills/key-gaps/key_gaps.py \
  prod=deploy/secrets/prod.yaml beta=deploy/secrets/beta.yaml
```

## Audit the LIVE boxes (read-only, key names only)

Gather the key names over SSH (never the values), then audit. Hosts per the
root `CLAUDE.md` topology:

- **vps2** (`169.58.46.181`) hosts BOTH prod and beta as separate Swarm
  stacks — TWO live env files on the same box: `/etc/thermograph.env` (prod)
  and `/etc/thermograph-beta.env` (beta). "The live env file per environment"
  is no longer one path per host; it's one path per environment, and both
  environments' files live on this one host. Get both in the same SSH round
  trip, or two separate commands against the same host — never assume one
  host means one file here.
- **vps1** (`75.119.132.91`) hosts dev, at the same `/etc/thermograph.env`
  path (it's a different host, so no collision with prod's file of the same
  name).

```sh
K=~/.ssh/thermograph_agent_ed25519
# sudo: the env files are root-owned (0640). On these boxes `agent` can often
# read them directly too, but sudo works uniformly regardless.
#
# The character class must allow DIGITS. This was `^[A-Z_]+=` until 2026-07-24,
# which silently skipped every key whose name contains a digit — in this estate
# that is all six *_S3_* keys. Prod audited as 26 keys against 32 real ones, and
# the audit reported the S3 credentials as missing from both hosts. They were
# present the whole time, and the phantom was independently "confirmed" twice by
# people re-running the same pattern. An audit that under-reports is worse than
# no audit: the invented finding sends someone to provision a credential that
# already exists, and it was briefly recorded as a root cause of a real bug.
# The match still stops at the `=`, so no value is ever read.
ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/prod.keys       # vps2: prod
ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph-beta.env' > /tmp/beta.keys  # vps2: beta
ssh -i $K agent@75.119.132.91  'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/dev.keys       # vps1: dev
python3 .claude/skills/key-gaps/key_gaps.py prod=/tmp/prod.keys beta=/tmp/beta.keys dev=/tmp/dev.keys
```

## Verify a SOPS cutover matches live

Before flipping a host onto the vault, confirm the rendered set equals the live set —
mix the two source types for the same box:

```
python3 .claude/skills/key-gaps/key_gaps.py live=/tmp/prod.keys vault=deploy/secrets/prod.yaml
```
The `Cross-environment drift` section should report **none** for tracked keys.

## Reading the output

- **CRITICAL** — a required, *self-generating* secret is missing (`THERMOGRAPH_AUTH_SECRET`,
  `THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY`). Missing here is the worst case: the app
  silently mints a new value on boot, invalidating every login session and push
  subscription. Fix before deploying.
- **MISSING** — a required secret is absent (`POSTGRES_PASSWORD`, `REGISTRY_TOKEN`).
- **Feature GAP** — a feature is *partially* configured (e.g. `discord-account-linking`
  has the app id but not the client secret), so it's silently broken. A fully-unset
  feature is "off", not a gap.
- **Dependent-key GAP** — an optional key is set but the *other* key it needs to do
  anything isn't (e.g. `THERMOGRAPH_DISCORD_WEATHER_CHANNEL` set with no
  `THERMOGRAPH_DISCORD_BOT_TOKEN` — discord.py gates the post on both). Asymmetric,
  unlike a feature group: the prerequisite key is fine set alone (the bot token alone
  already enables DMs).
- **drift** — a tracked key is in some environments but not others; often intentional,
  but worth a glance.

## Keeping it current

The secret manifest (which keys are required / self-generating / grouped into features /
dependent on another key) lives at the top of `key_gaps.py` in `REQUIRED`,
`OPTIONAL_SELF_GEN`, `FEATURE_GROUPS`, `DEPENDENT_OPTIONAL`, and `STANDALONE_OPTIONAL`.
When a new `THERMOGRAPH_*` credential is added to `deploy/thermograph.env.example` /
`deploy/secrets/`, add it there too — check the code path that reads it (`os.environ.get`
call site) to see whether it's truly required, self-generating, part of an all-or-nothing
feature, or only meaningful alongside another key, rather than guessing from the name.