2026-07-22 19:20:05 +00:00
|
|
|
#!/usr/bin/env bash
|
2026-07-26 06:56:38 +00:00
|
|
|
# Dev deploy: roll the per-service registry-pull stack (see deploy.sh) onto the
|
|
|
|
|
# dev compose overlay instead of beta/prod's Swarm stacks.
|
2026-07-22 19:20:05 +00:00
|
|
|
#
|
2026-07-26 06:56:38 +00:00
|
|
|
# Dev lives on vps1 now, at /opt/thermograph-dev — not on the operator's desktop.
|
|
|
|
|
# The desktop hosts no Thermograph environment any more; it runs the AI models
|
|
|
|
|
# and offers flex Swarm capacity. What moved is only WHERE and HOW dev is
|
|
|
|
|
# reached: it is a normal fleet host now (SSH deploy from CI, SOPS render at
|
|
|
|
|
# /etc/thermograph.env, mesh-only exposure on 10.10.0.2:8137), rather than a
|
|
|
|
|
# sudo-free systemd --user stack on someone's Wi-Fi.
|
|
|
|
|
#
|
|
|
|
|
# $APP_DIR is a checkout of the monorepo (deploy assets under infra/), the same
|
|
|
|
|
# way /opt/thermograph is for prod. A laptop can still point APP_DIR at a
|
|
|
|
|
# checkout under $HOME and get the old local behaviour.
|
2026-07-22 19:20:05 +00:00
|
|
|
#
|
|
|
|
|
# Design: a thin wrapper around deploy.sh, not a duplicate. deploy.sh already owns
|
|
|
|
|
# the entire per-service registry-pull mechanism -- secrets sourcing, docker login,
|
|
|
|
|
# the retry-pull loop, --no-deps single-service rolls vs. --remove-orphans `all`,
|
|
|
|
|
# .image-tags.env persistence, and the 8137/8080 health checks. None of that is
|
2026-07-26 06:56:38 +00:00
|
|
|
# dev-specific; the only things dev actually changes are WHERE it deploys (a
|
|
|
|
|
# separate checkout + branch), WHICH compose files are in play (the base file
|
|
|
|
|
# plus docker-compose.dev.yml's uncapped/mesh-published overrides), and its
|
|
|
|
|
# secrets policy (below). All are expressible as environment (APP_DIR/BRANCH
|
|
|
|
|
# that deploy.sh already reads, and docker compose's own COMPOSE_FILE
|
|
|
|
|
# variable), so re-exec'ing deploy.sh with that environment set covers it with
|
|
|
|
|
# no forked copy of the pull/roll/health logic to drift out of sync. If dev ever
|
|
|
|
|
# needs deploy logic that genuinely diverges from beta/prod (not just "different
|
|
|
|
|
# files/directory"), promote this to a standalone script at that point rather
|
|
|
|
|
# than growing special cases into deploy.sh.
|
2026-07-22 19:20:05 +00:00
|
|
|
#
|
2026-07-22 23:36:21 +00:00
|
|
|
# Usage, mirrors deploy.sh directly:
|
2026-07-22 19:20:05 +00:00
|
|
|
# # roll just the backend onto a dev-tagged image:
|
2026-07-26 06:56:38 +00:00
|
|
|
# ssh vps1 'SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> /opt/thermograph-dev/infra/deploy/deploy-dev.sh'
|
2026-07-22 19:20:05 +00:00
|
|
|
# # bring the whole dev stack up (both tags required, same as deploy.sh):
|
2026-07-26 06:56:38 +00:00
|
|
|
# ssh vps1 'SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> /opt/thermograph-dev/infra/deploy/deploy-dev.sh'
|
2026-07-22 19:20:05 +00:00
|
|
|
set -euo pipefail
|
|
|
|
|
|
2026-07-26 06:56:38 +00:00
|
|
|
# Dev context: its own checkout and its own branch, on its own host (vps1).
|
|
|
|
|
# Both come from deploy/env-topology.sh so there is one place that says where
|
|
|
|
|
# dev lives. $APP_DIR is still honoured when set explicitly — that is how a
|
|
|
|
|
# laptop points this at a checkout under $HOME.
|
|
|
|
|
_dev_self="$(cd "$(dirname "$0")" && pwd)"
|
|
|
|
|
# shellcheck source=infra/deploy/env-topology.sh
|
|
|
|
|
. "$_dev_self/env-topology.sh"
|
|
|
|
|
thermograph_topology dev
|
|
|
|
|
APP_DIR="${APP_DIR:-$TG_APP_DIR}"
|
|
|
|
|
BRANCH="${BRANCH:-$TG_BRANCH}"
|
2026-07-22 19:20:05 +00:00
|
|
|
export APP_DIR BRANCH
|
2026-07-26 06:56:38 +00:00
|
|
|
# deploy.sh resolves the environment itself; say so explicitly rather than
|
|
|
|
|
# letting it fall back to a host marker that, on vps1, names dev anyway.
|
|
|
|
|
export THERMOGRAPH_ENV=dev
|
2026-07-22 19:20:05 +00:00
|
|
|
|
|
|
|
|
# Point every `docker compose` invocation inside deploy.sh at the LAN-dev overlay
|
|
|
|
|
# (uncapped CPU, backend published on 0.0.0.0:8137, frontend unpublished -- see
|
|
|
|
|
# docker-compose.dev.yml) without deploy.sh needing to know dev exists at all.
|
|
|
|
|
# COMPOSE_FILE is docker compose's own env var for this; ':' is the Linux/macOS
|
|
|
|
|
# path-list separator it expects (';' only on Windows).
|
|
|
|
|
export COMPOSE_FILE="docker-compose.yml:docker-compose.dev.yml"
|
|
|
|
|
|
2026-07-23 05:11:33 +00:00
|
|
|
# Pin the dev project name: docker-compose.yml pins `name: thermograph` for
|
|
|
|
|
# prod/beta, but dev's stack has always been the separate "thermograph-dev"
|
|
|
|
|
# project (originally derived from this checkout's directory name). The env
|
|
|
|
|
# var wins over the compose-file `name:` key, preserving that split.
|
|
|
|
|
export COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-thermograph-dev}"
|
|
|
|
|
|
secrets: vault dev and Centralis; render dev without common.yaml
Brings the last two credential stores under SOPS, so all three environments and
the control plane are managed the same way.
dev.yaml — dev was not "intentionally not wired in", it was broken
The vault README claimed dev had no real secrets. Both halves were false.
Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash
loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor
THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord
gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls
back to https://thermograph.org, so dev's IndexNow pings and verification links
claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real
base URL, and 12 values total.
And dev already held production credentials: backend-deploy-dev.yml injects
S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned
keypair for that bucket is read-write — on the bucket holding prod's backups.
Dev does not need them; without bucket creds the lake service 503s and history
falls through to the archive. Those workflow lines still need removing.
dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of
common.yaml's sixteen values are live production credentials, the render is a
plaintext concatenation consumed through env_file:, and dev is the operator's
desktop AND the Forgejo runner executing unreviewed dev-branch code with the
docker socket mounted. An override in dev.yaml would not help — last-wins
governs consumers, but prod's value is still physically a line in the file.
Verified: current renderer leaks 28 lines onto dev including both S3 keypairs
and the VAPID private key; patched gives exactly 12 keys and no prod credential.
Beta's render is byte-identical either way.
centralis.prod.yaml — its own file, its own renderer
/etc/centralis.env was hand-edited, which is how a JSON registry got written
unquoted into a shell-sourced file tonight and silently collapsed three
identities to one. The renderer now owns the file.
It is service- AND environment-scoped, not folded into prod.yaml, because
/etc/thermograph.env is loaded into every container in the app stack. Folding
these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which
authenticate an endpoint carrying run_on_host and sql_query(write) — into the
web backend's environment, turning a read-anything bug in the app into a
foothold on the control plane.
Quoting is guaranteed three ways rather than assumed. Critically, `sops -d
--output-type dotenv` emits values verbatim, so reusing the existing render path
would have reproduced tonight's bug exactly. The new function decrypts to JSON
and emits POSIX single-quoted assignments; it then sources its own output in a
clean shell and compares every value before touching /etc; and it refuses to
write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after
sourcing. Tested against 16 hostile values including embedded quotes, newlines
and ;rm -rf /.
Value-identity proven, not asserted: rendered vs live compared as *effective*
values on prod (a textual diff would report a false difference, and would report
a false match if the vault had captured quote characters as part of the value),
plus both env files producing a byte-identical `docker compose config` digest.
9 keys, both token subjects preserved. Nothing deployed.
Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw
dotenv only because none of its 32 current values contains a shell-special
character. It is one quoted secret away from the same bug.
Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-25 01:10:51 +00:00
|
|
|
# --- Secrets -----------------------------------------------------------------
|
|
|
|
|
# dev renders from the SOPS vault the same way prod and beta do -- same files, same
|
|
|
|
|
# tool, same edit/commit/deploy loop -- with two dev-only adjustments. Both are
|
|
|
|
|
# expressed as environment that render-secrets.sh already parameterises, so nothing
|
|
|
|
|
# about the rendering logic is forked. See deploy/secrets/README.md.
|
|
|
|
|
|
|
|
|
|
# (1) dev renders deploy/secrets/dev.yaml ALONE; it does NOT layer common.yaml.
|
|
|
|
|
#
|
|
|
|
|
# common.yaml is the internet-facing fleet's SHARED PRODUCTION credential set: the
|
|
|
|
|
# registry token, both S3 keypairs (the bucket keypair is read-write, and that
|
|
|
|
|
# bucket also holds prod's database backups), the VAPID private key that signs push
|
|
|
|
|
# to real subscribers, the IndexNow key, the metrics token. Layering it here would
|
|
|
|
|
# write all of them in plaintext to /etc/thermograph.env on the box that is also the
|
|
|
|
|
# Forgejo CI runner, and hand them to every container in the dev stack through
|
|
|
|
|
# `env_file:` -- that is, to whatever unreviewed branch dev happens to be running.
|
|
|
|
|
# It would also silently apply THERMOGRAPH_COOKIE_SECURE=1, which is correct for the
|
|
|
|
|
# TLS hosts and breaks login on dev's plain-HTTP LAN URL.
|
|
|
|
|
#
|
|
|
|
|
# dev.yaml is self-contained instead: it carries dev's own copy of everything dev
|
|
|
|
|
# genuinely needs, and nothing else.
|
|
|
|
|
export THERMOGRAPH_SECRETS_SKIP_COMMON=1
|
|
|
|
|
|
2026-07-26 06:56:38 +00:00
|
|
|
# (1b) Snap-packaged Docker is confined to $HOME (plus a short allowlist) and
|
|
|
|
|
# silently can't see /etc/thermograph.env at all, so env_file: /etc/thermograph.env
|
|
|
|
|
# resolves to nothing for every container no matter how correctly it's rendered.
|
|
|
|
|
# Where that is the case, mirror the render to a path under $APP_DIR;
|
|
|
|
|
# docker-compose.dev.yml carries a matching optional env_file entry.
|
|
|
|
|
#
|
|
|
|
|
# CONDITIONAL, not unconditional: this writes a plaintext copy of the render
|
|
|
|
|
# into the checkout, and on vps1 (ordinary apt Docker, which reads /etc fine)
|
|
|
|
|
# that copy would be pure liability. Detect the confinement rather than add a
|
|
|
|
|
# knob someone has to remember — the docker binary resolving under /snap is
|
|
|
|
|
# exactly the condition that breaks the /etc path.
|
|
|
|
|
_docker_bin="$(command -v docker || true)"
|
|
|
|
|
if [ -n "$_docker_bin" ] && case "$(readlink -f "$_docker_bin")" in /snap/*) true ;; *) false ;; esac; then
|
|
|
|
|
export THERMOGRAPH_SECRETS_ENV_FILE_MIRROR="$APP_DIR/infra/deploy/dev-secrets.env"
|
|
|
|
|
fi
|
2026-07-25 07:16:08 +00:00
|
|
|
|
secrets: vault dev and Centralis; render dev without common.yaml
Brings the last two credential stores under SOPS, so all three environments and
the control plane are managed the same way.
dev.yaml — dev was not "intentionally not wired in", it was broken
The vault README claimed dev had no real secrets. Both halves were false.
Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash
loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor
THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord
gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls
back to https://thermograph.org, so dev's IndexNow pings and verification links
claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real
base URL, and 12 values total.
And dev already held production credentials: backend-deploy-dev.yml injects
S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned
keypair for that bucket is read-write — on the bucket holding prod's backups.
Dev does not need them; without bucket creds the lake service 503s and history
falls through to the archive. Those workflow lines still need removing.
dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of
common.yaml's sixteen values are live production credentials, the render is a
plaintext concatenation consumed through env_file:, and dev is the operator's
desktop AND the Forgejo runner executing unreviewed dev-branch code with the
docker socket mounted. An override in dev.yaml would not help — last-wins
governs consumers, but prod's value is still physically a line in the file.
Verified: current renderer leaks 28 lines onto dev including both S3 keypairs
and the VAPID private key; patched gives exactly 12 keys and no prod credential.
Beta's render is byte-identical either way.
centralis.prod.yaml — its own file, its own renderer
/etc/centralis.env was hand-edited, which is how a JSON registry got written
unquoted into a shell-sourced file tonight and silently collapsed three
identities to one. The renderer now owns the file.
It is service- AND environment-scoped, not folded into prod.yaml, because
/etc/thermograph.env is loaded into every container in the app stack. Folding
these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which
authenticate an endpoint carrying run_on_host and sql_query(write) — into the
web backend's environment, turning a read-anything bug in the app into a
foothold on the control plane.
Quoting is guaranteed three ways rather than assumed. Critically, `sops -d
--output-type dotenv` emits values verbatim, so reusing the existing render path
would have reproduced tonight's bug exactly. The new function decrypts to JSON
and emits POSIX single-quoted assignments; it then sources its own output in a
clean shell and compares every value before touching /etc; and it refuses to
write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after
sourcing. Tested against 16 hostile values including embedded quotes, newlines
and ;rm -rf /.
Value-identity proven, not asserted: rendered vs live compared as *effective*
values on prod (a textual diff would report a false difference, and would report
a false match if the vault had captured quote characters as part of the value),
plus both env files producing a byte-identical `docker compose config` digest.
9 keys, both token subjects preserved. Nothing deployed.
Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw
dotenv only because none of its 32 current values contains a shell-special
character. It is one quoted secret away from the same bug.
Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-25 01:10:51 +00:00
|
|
|
# (2) The age key is read from wherever it is READABLE, not necessarily
|
|
|
|
|
# /etc/thermograph/age.key. render-secrets.sh falls back to `sudo cat` for a
|
2026-07-26 06:56:38 +00:00
|
|
|
# root-owned 0400 key, which needs passwordless sudo.
|
|
|
|
|
#
|
|
|
|
|
# On vps1 that is the normal case: the `agent` user has passwordless sudo and
|
|
|
|
|
# the key sits at the fleet-standard path, so this block does nothing. The
|
|
|
|
|
# fallback exists for the laptop case — a checkout where the key lives only in
|
|
|
|
|
# the operator's own keyring (that machine is where `sops` is run), and where a
|
|
|
|
|
# non-interactive shell could not prompt for sudo even if the key were in /etc.
|
|
|
|
|
# Preferring the standard path when readable keeps vps1 on the fleet convention
|
|
|
|
|
# and means provisioning dev needs no second copy of the estate's recovery root.
|
secrets: vault dev and Centralis; render dev without common.yaml
Brings the last two credential stores under SOPS, so all three environments and
the control plane are managed the same way.
dev.yaml — dev was not "intentionally not wired in", it was broken
The vault README claimed dev had no real secrets. Both halves were false.
Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash
loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor
THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord
gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls
back to https://thermograph.org, so dev's IndexNow pings and verification links
claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real
base URL, and 12 values total.
And dev already held production credentials: backend-deploy-dev.yml injects
S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned
keypair for that bucket is read-write — on the bucket holding prod's backups.
Dev does not need them; without bucket creds the lake service 503s and history
falls through to the archive. Those workflow lines still need removing.
dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of
common.yaml's sixteen values are live production credentials, the render is a
plaintext concatenation consumed through env_file:, and dev is the operator's
desktop AND the Forgejo runner executing unreviewed dev-branch code with the
docker socket mounted. An override in dev.yaml would not help — last-wins
governs consumers, but prod's value is still physically a line in the file.
Verified: current renderer leaks 28 lines onto dev including both S3 keypairs
and the VAPID private key; patched gives exactly 12 keys and no prod credential.
Beta's render is byte-identical either way.
centralis.prod.yaml — its own file, its own renderer
/etc/centralis.env was hand-edited, which is how a JSON registry got written
unquoted into a shell-sourced file tonight and silently collapsed three
identities to one. The renderer now owns the file.
It is service- AND environment-scoped, not folded into prod.yaml, because
/etc/thermograph.env is loaded into every container in the app stack. Folding
these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which
authenticate an endpoint carrying run_on_host and sql_query(write) — into the
web backend's environment, turning a read-anything bug in the app into a
foothold on the control plane.
Quoting is guaranteed three ways rather than assumed. Critically, `sops -d
--output-type dotenv` emits values verbatim, so reusing the existing render path
would have reproduced tonight's bug exactly. The new function decrypts to JSON
and emits POSIX single-quoted assignments; it then sources its own output in a
clean shell and compares every value before touching /etc; and it refuses to
write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after
sourcing. Tested against 16 hostile values including embedded quotes, newlines
and ;rm -rf /.
Value-identity proven, not asserted: rendered vs live compared as *effective*
values on prod (a textual diff would report a false difference, and would report
a false match if the vault had captured quote characters as part of the value),
plus both env files producing a byte-identical `docker compose config` digest.
9 keys, both token subjects preserved. Nothing deployed.
Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw
dotenv only because none of its 32 current values contains a shell-special
character. It is one quoted secret away from the same bug.
Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-25 01:10:51 +00:00
|
|
|
if [ -z "${THERMOGRAPH_AGE_KEY:-}" ] && [ ! -r /etc/thermograph/age.key ] \
|
|
|
|
|
&& [ -r "$HOME/.config/sops/age/keys.txt" ]; then
|
|
|
|
|
export THERMOGRAPH_AGE_KEY="$HOME/.config/sops/age/keys.txt"
|
|
|
|
|
fi
|
|
|
|
|
|
|
|
|
|
# Fail closed rather than leak. If this box is provisioned to render (an env marker
|
|
|
|
|
# AND a key) but the render-secrets.sh in this checkout predates
|
|
|
|
|
# THERMOGRAPH_SECRETS_SKIP_COMMON, the render would quietly concatenate common.yaml
|
|
|
|
|
# into dev's /etc/thermograph.env -- exactly the leak (1) exists to prevent, with a
|
|
|
|
|
# success exit code. Checking a sibling script for the flag is blunt, but the two
|
|
|
|
|
# files are versioned together in this same checkout, and the alternative failure
|
|
|
|
|
# mode is unreviewed code running with the estate's object-storage keys. Remove this
|
|
|
|
|
# once the flag is unconditionally present in render-secrets.sh.
|
|
|
|
|
_render_sh="$(dirname "$0")/render-secrets.sh"
|
|
|
|
|
_secrets_marker="${THERMOGRAPH_SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}"
|
|
|
|
|
_age_key="${THERMOGRAPH_AGE_KEY:-/etc/thermograph/age.key}"
|
|
|
|
|
if [ -s "$_secrets_marker" ] && [ -f "$_age_key" ] && [ -f "$_render_sh" ] \
|
|
|
|
|
&& ! grep -q 'THERMOGRAPH_SECRETS_SKIP_COMMON' "$_render_sh"; then
|
|
|
|
|
echo "!! This box is provisioned to render SOPS secrets (env marker + age key)," >&2
|
|
|
|
|
echo "!! but $_render_sh does not honour" >&2
|
|
|
|
|
echo "!! THERMOGRAPH_SECRETS_SKIP_COMMON, so it would render common.yaml -- the" >&2
|
|
|
|
|
echo "!! fleet's shared PRODUCTION credentials -- into /etc/thermograph.env on the" >&2
|
|
|
|
|
echo "!! CI-runner box. Refusing to deploy." >&2
|
|
|
|
|
echo "!! Fix: update deploy/render-secrets.sh, or remove $_secrets_marker to fall" >&2
|
|
|
|
|
echo "!! back to the un-vaulted dev path." >&2
|
|
|
|
|
exit 1
|
|
|
|
|
fi
|
|
|
|
|
|
|
|
|
|
# deploy.sh needs POSTGRES_PASSWORD to interpolate the compose file (both to init the
|
|
|
|
|
# db container and to build backend's THERMOGRAPH_DATABASE_URL). On a provisioned dev
|
|
|
|
|
# box it arrives from dev.yaml via the render above -- deploy.sh sources
|
|
|
|
|
# /etc/thermograph.env AFTER calling the renderer, so the vault value wins over this
|
|
|
|
|
# line. This default is the fallback for the un-vaulted paths that remain: a fresh
|
|
|
|
|
# box before provisioning, and a bare `make dev-up`. It is the same value dev.yaml
|
|
|
|
|
# holds and the value dev's existing pgdata volume was initialized with -- keep the
|
|
|
|
|
# two in step, and note that changing it needs an ALTER ROLE against the running
|
|
|
|
|
# database as well (see deploy/secrets/README.md), not just an edit.
|
2026-07-22 19:20:05 +00:00
|
|
|
export POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-thermograph-dev}"
|
|
|
|
|
|
secrets: vault dev and Centralis; render dev without common.yaml
Brings the last two credential stores under SOPS, so all three environments and
the control plane are managed the same way.
dev.yaml — dev was not "intentionally not wired in", it was broken
The vault README claimed dev had no real secrets. Both halves were false.
Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash
loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor
THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord
gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls
back to https://thermograph.org, so dev's IndexNow pings and verification links
claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real
base URL, and 12 values total.
And dev already held production credentials: backend-deploy-dev.yml injects
S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned
keypair for that bucket is read-write — on the bucket holding prod's backups.
Dev does not need them; without bucket creds the lake service 503s and history
falls through to the archive. Those workflow lines still need removing.
dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of
common.yaml's sixteen values are live production credentials, the render is a
plaintext concatenation consumed through env_file:, and dev is the operator's
desktop AND the Forgejo runner executing unreviewed dev-branch code with the
docker socket mounted. An override in dev.yaml would not help — last-wins
governs consumers, but prod's value is still physically a line in the file.
Verified: current renderer leaks 28 lines onto dev including both S3 keypairs
and the VAPID private key; patched gives exactly 12 keys and no prod credential.
Beta's render is byte-identical either way.
centralis.prod.yaml — its own file, its own renderer
/etc/centralis.env was hand-edited, which is how a JSON registry got written
unquoted into a shell-sourced file tonight and silently collapsed three
identities to one. The renderer now owns the file.
It is service- AND environment-scoped, not folded into prod.yaml, because
/etc/thermograph.env is loaded into every container in the app stack. Folding
these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which
authenticate an endpoint carrying run_on_host and sql_query(write) — into the
web backend's environment, turning a read-anything bug in the app into a
foothold on the control plane.
Quoting is guaranteed three ways rather than assumed. Critically, `sops -d
--output-type dotenv` emits values verbatim, so reusing the existing render path
would have reproduced tonight's bug exactly. The new function decrypts to JSON
and emits POSIX single-quoted assignments; it then sources its own output in a
clean shell and compares every value before touching /etc; and it refuses to
write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after
sourcing. Tested against 16 hostile values including embedded quotes, newlines
and ;rm -rf /.
Value-identity proven, not asserted: rendered vs live compared as *effective*
values on prod (a textual diff would report a false difference, and would report
a false match if the vault had captured quote characters as part of the value),
plus both env files producing a byte-identical `docker compose config` digest.
9 keys, both token subjects preserved. Nothing deployed.
Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw
dotenv only because none of its 32 current values contains a shell-special
character. It is one quoted secret away from the same bug.
Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-25 01:10:51 +00:00
|
|
|
# REGISTRY_TOKEN (for `docker login` in deploy.sh) is deliberately NOT in dev.yaml
|
|
|
|
|
# and has no default here. dev pulls with the persistent `docker login` credential
|
|
|
|
|
# already in this host's docker config -- the same arrangement the prod/beta CI
|
|
|
|
|
# deploy paths use, and the reason deploy.sh's login step is conditional. Putting a
|
|
|
|
|
# registry token in dev's vault would only add a second copy of a credential the box
|
|
|
|
|
# already has, in a file more processes can read. Export it by hand for a one-off run
|
|
|
|
|
# on a box that has never logged in; a genuine auth problem fails loudly at `pull`.
|
2026-07-22 19:20:05 +00:00
|
|
|
|
|
|
|
|
echo "==> LAN-dev deploy: APP_DIR=$APP_DIR BRANCH=$BRANCH COMPOSE_FILE=$COMPOSE_FILE"
|
|
|
|
|
exec "$(dirname "$0")/deploy.sh" "$@"
|