43c9e2052a
Some checks failed
PR build (required check) / changes (pull_request) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Failing after 9s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
Brings the last two credential stores under SOPS, so all three environments and the control plane are managed the same way. dev.yaml — dev was not "intentionally not wired in", it was broken The vault README claimed dev had no real secrets. Both halves were false. Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls back to https://thermograph.org, so dev's IndexNow pings and verification links claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real base URL, and 12 values total. And dev already held production credentials: backend-deploy-dev.yml injects S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned keypair for that bucket is read-write — on the bucket holding prod's backups. Dev does not need them; without bucket creds the lake service 503s and history falls through to the archive. Those workflow lines still need removing. dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of common.yaml's sixteen values are live production credentials, the render is a plaintext concatenation consumed through env_file:, and dev is the operator's desktop AND the Forgejo runner executing unreviewed dev-branch code with the docker socket mounted. An override in dev.yaml would not help — last-wins governs consumers, but prod's value is still physically a line in the file. Verified: current renderer leaks 28 lines onto dev including both S3 keypairs and the VAPID private key; patched gives exactly 12 keys and no prod credential. Beta's render is byte-identical either way. centralis.prod.yaml — its own file, its own renderer /etc/centralis.env was hand-edited, which is how a JSON registry got written unquoted into a shell-sourced file tonight and silently collapsed three identities to one. The renderer now owns the file. It is service- AND environment-scoped, not folded into prod.yaml, because /etc/thermograph.env is loaded into every container in the app stack. Folding these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which authenticate an endpoint carrying run_on_host and sql_query(write) — into the web backend's environment, turning a read-anything bug in the app into a foothold on the control plane. Quoting is guaranteed three ways rather than assumed. Critically, `sops -d --output-type dotenv` emits values verbatim, so reusing the existing render path would have reproduced tonight's bug exactly. The new function decrypts to JSON and emits POSIX single-quoted assignments; it then sources its own output in a clean shell and compares every value before touching /etc; and it refuses to write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after sourcing. Tested against 16 hostile values including embedded quotes, newlines and ;rm -rf /. Value-identity proven, not asserted: rendered vs live compared as *effective* values on prod (a textual diff would report a false difference, and would report a false match if the vault had captured quote characters as part of the value), plus both env files producing a byte-identical `docker compose config` digest. 9 keys, both token subjects preserved. Nothing deployed. Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw dotenv only because none of its 32 current values contains a shell-special character. It is one quoted secret away from the same bug. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
99 lines
5 KiB
Bash
Executable file
99 lines
5 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Seed deploy/secrets/centralis.<env>.yaml by encrypting prod's live
|
|
# /etc/centralis.env ENTIRELY ON THE BOX. The plaintext never leaves the box;
|
|
# only already-encrypted ciphertext is written back here. Encryption uses the
|
|
# age PUBLIC key (safe to hardcode), so the machine you run this from needs
|
|
# nothing but SSH access — no sops, no age, no private key.
|
|
#
|
|
# deploy/secrets/seed-centralis-on-host.sh prod agent@169.58.46.181
|
|
#
|
|
# Then commit the resulting deploy/secrets/centralis.<env>.yaml.
|
|
#
|
|
# ---------------------------------------------------------------------------
|
|
# WHY THIS IS NOT seed-encrypt-on-host.sh WITH A DIFFERENT PATH
|
|
# ---------------------------------------------------------------------------
|
|
# `seed-encrypt-on-host.sh` captures the RAW TEXT to the right of the `=`. That
|
|
# is correct for /etc/thermograph.env, which is machine-rendered and whose 32
|
|
# values happen to be bare tokens, but it is WRONG for a hand-maintained,
|
|
# shell-SOURCED file. /etc/centralis.env is consumed as
|
|
#
|
|
# set -a && . /etc/centralis.env && set +a && docker compose up
|
|
#
|
|
# so the value the program actually receives is the value AFTER bash's quote
|
|
# removal. `CENTRALIS_TOKENS` is stored there single-quoted, because it is JSON
|
|
# and contains double quotes. Capturing the raw text would put the surrounding
|
|
# single quotes INSIDE the vault; the renderer would then quote them again, and
|
|
# Centralis would receive `'{"emi":...}'` — literal quotes and all — and fail
|
|
# its JSON parse. Fails closed, so it would look like the token registry had
|
|
# been deleted. That is the same class of bug as the one this whole migration
|
|
# exists to remove, arriving from the other direction.
|
|
#
|
|
# So this script asks BASH what the values are, by sourcing the file in a clean
|
|
# environment (`env -i`) and reading the resulting variables. What goes into the
|
|
# vault is exactly what Centralis receives today, by construction.
|
|
set -euo pipefail
|
|
cd "$(dirname "$(readlink -f "$0")")/../.."
|
|
|
|
ENV_NAME="${1:?usage: seed-centralis-on-host.sh <env> <ssh-target> [ssh-key]}"
|
|
SSH_TARGET="${2:?ssh target, e.g. agent@169.58.46.181}"
|
|
SSH_KEY="${3:-$HOME/.ssh/thermograph_agent_ed25519}"
|
|
OUT="deploy/secrets/centralis.${ENV_NAME}.yaml"
|
|
|
|
# Public age recipient — NOT a secret (matches .sops.yaml). The private key never
|
|
# leaves the operator's machine / the hosts' /etc/thermograph/age.key.
|
|
AGE_PUB="age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2"
|
|
SOPS_URL="https://github.com/getsops/sops/releases/download/v3.13.2/sops-v3.13.2.linux.amd64"
|
|
|
|
echo "==> Encrypting ${SSH_TARGET}:/etc/centralis.env on-host -> ${OUT}"
|
|
ssh -i "$SSH_KEY" "$SSH_TARGET" "AGE_PUB='$AGE_PUB' SOPS_URL='$SOPS_URL' bash -s" > "$OUT" <<'REMOTE'
|
|
set -euo pipefail
|
|
exec 3>&1 # real stdout carries ONLY the ciphertext
|
|
{ # setup noise -> stderr, so it can't corrupt the file
|
|
if ! command -v sops >/dev/null 2>&1; then
|
|
sudo curl -fsSL "$SOPS_URL" -o /usr/local/bin/sops
|
|
sudo chmod +x /usr/local/bin/sops
|
|
fi
|
|
} >&2
|
|
umask 077
|
|
tmp="$(mktemp)"; tmy="$(mktemp)"; tpy="$(mktemp)"
|
|
trap 'rm -f "$tmp" "$tmy" "$tpy"' EXIT
|
|
sudo cat /etc/centralis.env > "$tmp" # plaintext stays on this box only
|
|
|
|
# The declared assignment targets, in file order. `source` gives last-wins for a
|
|
# repeated key and we read the post-source environment, so the de-dup here only
|
|
# controls the ORDER of the emitted YAML, never which value is taken.
|
|
names="$(grep -oE '^[[:space:]]*(export[[:space:]]+)?[A-Za-z_][A-Za-z0-9_]*=' "$tmp" \
|
|
| sed -E 's/^[[:space:]]*(export[[:space:]]+)?//; s/=$//' | awk '!seen[$0]++')"
|
|
[ -n "$names" ] || { echo "!! no KEY=VALUE lines in /etc/centralis.env" >&2; exit 1; }
|
|
echo " capturing: $(echo "$names" | tr '\n' ' ')" >&2
|
|
|
|
cat > "$tpy" <<'PY'
|
|
import json, os, sys
|
|
for n in sys.argv[1:]:
|
|
v = os.environ.get(n)
|
|
if v is None:
|
|
sys.stderr.write("!! %s vanished after sourcing\n" % n)
|
|
sys.exit(1)
|
|
print("%s: %s" % (n, json.dumps(v))) # JSON-escaped scalar = valid YAML
|
|
PY
|
|
|
|
# Source in a CLEAN environment and read the values back, so what is stored is
|
|
# exactly what `set -a && . /etc/centralis.env` hands to docker compose. PATH is
|
|
# passed through only so python3 is findable. $names is deliberately unquoted:
|
|
# it is a whitespace-separated list of identifiers by construction.
|
|
# shellcheck disable=SC2086
|
|
env -i PATH="$PATH" bash -c \
|
|
'set -a; . "$1"; set +a; shift; exec python3 "$@"' _ "$tmp" "$tpy" $names > "$tmy"
|
|
[ -s "$tmy" ] || { echo "!! captured nothing" >&2; exit 1; }
|
|
|
|
cd /tmp # no .sops.yaml here, so --age is honored
|
|
sops -e --age "$AGE_PUB" --input-type yaml --output-type yaml "$tmy" >&3
|
|
REMOTE
|
|
|
|
if grep -q 'ENC\[AES256_GCM' "$OUT"; then
|
|
echo "==> OK: ${OUT} written and encrypted ($(grep -cE '^[A-Za-z_][A-Za-z0-9_]*: ' "$OUT") keys)"
|
|
echo " Next: infra/deploy/secrets/verify-centralis-render.sh ${ENV_NAME} ${SSH_TARGET}"
|
|
else
|
|
echo "!! ${OUT} is not encrypted (the on-host step failed) — removing" >&2
|
|
rm -f "$OUT"; exit 1
|
|
fi
|