thermograph/infra/deploy/deploy-dev.sh

164 lines
9.9 KiB
Bash
Raw Normal View History

#!/usr/bin/env bash
infra: split the estate into vps1/vps2, moving beta next to prod and dev onto vps1 Environments stop being machines. Until now each box WAS an environment -- "beta" named both a deploy target and a host, "the desktop" named both the operator's computer and the dev server -- so every path could assume one environment per host and hardcode /opt/thermograph, /etc/thermograph.env and ports 8137/8080. That assumption ends here: vps1 75.119.132.91 Forgejo, Grafana/Loki, the portfolio site, and DEV (own Postgres, mesh-only on 10.10.0.2:8137) vps2 169.58.46.181 PROD and BETA as two Swarm stacks sharing one TimescaleDB instance, plus Centralis, Postfix, backups desktop AI model hosting + flex Swarm capacity, no environment Nothing live has moved; the ordered cutover is in infra/deploy/RUNBOOK-vps1-vps2-cutover.md. deploy/env-topology.sh is the single source of truth: env -> host, checkout, branch, deploy mode, stack name, env file, LB ports, DB role/database, service prefix. THERMOGRAPH_ENV is the input, and on vps2 it is the only thing distinguishing a beta deploy from a prod one -- deploy.sh refuses to run when it disagrees with the checkout it was invoked from. The host-wide secrets-env and deploy-mode markers survive only as a fallback for a single-environment box. Beta gets its own stack file rather than an overlay (a merge cannot REMOVE the db service, and beta having no database of its own is the point). Its services are prefixed beta-*: Swarm registers a service's short name as a DNS alias on every network it joins, so two stacks both calling a service `web` on the shared data network would let prod's frontend resolve a beta task. Prod's stack, env and LB config are untouched. One Postgres, two databases with two roles: deploy/db/provision-env-db.sh creates thermograph_beta (NOSUPERUSER, owns only its own database, CONNECT revoked from PUBLIC) plus a <role>_ro for ad-hoc queries, and refuses to run for the environment that owns the instance -- doing so would demote prod's bootstrap superuser. Fixes that co-residency would otherwise have broken silently: - CI secrets are keyed by host (VPS1_SSH_*, VPS2_SSH_*). SSH_* meant "beta" and also "the Forgejo box" because those shared a machine; that conflation is what once had the prod backup dumping beta. - The nightly backup dumps BOTH application databases to separate off-box prefixes, and fails loudly on a missing one instead of skipping it. - Alloy stopped deriving the `host` label from the node -- on vps2 that would have filed every beta line as prod, feeding prod's alert rules with beta's traffic. It is now derived per source, with a new `node` label for the machine, and beta's log volume is mounted via a vps2-only overlay. - ops/dbq.sh, ops/iceberg.sh and the secrets seed scripts derived their SSH target and env-file path from the topology instead of hardcoding beta to 75.119.132.91 -- which is vps1's address now. - Dev's overlay binds DEV_BIND_ADDR (loopback by default, the mesh address on vps1) instead of 0.0.0.0: correct for a home LAN box, a public exposure of unreviewed branches on a VPS. Terraform's hosts variable is keyed by machine with a nested environments map, and main.tf flattens (host, environment) pairs so two environments on one box cannot share a checkout. Caddy's single reference config is split per host. Docs, onboarding and the runbooks are updated throughout, including the security rationales that co-residency makes false: separate SSH credentials no longer put a host boundary between beta and prod, and the boundary that remains is the database and the filesystem.
2026-07-25 22:01:29 +00:00
# Dev deploy: roll the per-service registry-pull stack (see deploy.sh) onto the
# dev compose overlay instead of beta/prod's Swarm stacks.
#
infra: split the estate into vps1/vps2, moving beta next to prod and dev onto vps1 Environments stop being machines. Until now each box WAS an environment -- "beta" named both a deploy target and a host, "the desktop" named both the operator's computer and the dev server -- so every path could assume one environment per host and hardcode /opt/thermograph, /etc/thermograph.env and ports 8137/8080. That assumption ends here: vps1 75.119.132.91 Forgejo, Grafana/Loki, the portfolio site, and DEV (own Postgres, mesh-only on 10.10.0.2:8137) vps2 169.58.46.181 PROD and BETA as two Swarm stacks sharing one TimescaleDB instance, plus Centralis, Postfix, backups desktop AI model hosting + flex Swarm capacity, no environment Nothing live has moved; the ordered cutover is in infra/deploy/RUNBOOK-vps1-vps2-cutover.md. deploy/env-topology.sh is the single source of truth: env -> host, checkout, branch, deploy mode, stack name, env file, LB ports, DB role/database, service prefix. THERMOGRAPH_ENV is the input, and on vps2 it is the only thing distinguishing a beta deploy from a prod one -- deploy.sh refuses to run when it disagrees with the checkout it was invoked from. The host-wide secrets-env and deploy-mode markers survive only as a fallback for a single-environment box. Beta gets its own stack file rather than an overlay (a merge cannot REMOVE the db service, and beta having no database of its own is the point). Its services are prefixed beta-*: Swarm registers a service's short name as a DNS alias on every network it joins, so two stacks both calling a service `web` on the shared data network would let prod's frontend resolve a beta task. Prod's stack, env and LB config are untouched. One Postgres, two databases with two roles: deploy/db/provision-env-db.sh creates thermograph_beta (NOSUPERUSER, owns only its own database, CONNECT revoked from PUBLIC) plus a <role>_ro for ad-hoc queries, and refuses to run for the environment that owns the instance -- doing so would demote prod's bootstrap superuser. Fixes that co-residency would otherwise have broken silently: - CI secrets are keyed by host (VPS1_SSH_*, VPS2_SSH_*). SSH_* meant "beta" and also "the Forgejo box" because those shared a machine; that conflation is what once had the prod backup dumping beta. - The nightly backup dumps BOTH application databases to separate off-box prefixes, and fails loudly on a missing one instead of skipping it. - Alloy stopped deriving the `host` label from the node -- on vps2 that would have filed every beta line as prod, feeding prod's alert rules with beta's traffic. It is now derived per source, with a new `node` label for the machine, and beta's log volume is mounted via a vps2-only overlay. - ops/dbq.sh, ops/iceberg.sh and the secrets seed scripts derived their SSH target and env-file path from the topology instead of hardcoding beta to 75.119.132.91 -- which is vps1's address now. - Dev's overlay binds DEV_BIND_ADDR (loopback by default, the mesh address on vps1) instead of 0.0.0.0: correct for a home LAN box, a public exposure of unreviewed branches on a VPS. Terraform's hosts variable is keyed by machine with a nested environments map, and main.tf flattens (host, environment) pairs so two environments on one box cannot share a checkout. Caddy's single reference config is split per host. Docs, onboarding and the runbooks are updated throughout, including the security rationales that co-residency makes false: separate SSH credentials no longer put a host boundary between beta and prod, and the boundary that remains is the database and the filesystem.
2026-07-25 22:01:29 +00:00
# Dev lives on vps1 now, at /opt/thermograph-dev — not on the operator's desktop.
# The desktop hosts no Thermograph environment any more; it runs the AI models
# and offers flex Swarm capacity. What moved is only WHERE and HOW dev is
# reached: it is a normal fleet host now (SSH deploy from CI, SOPS render at
# /etc/thermograph.env, mesh-only exposure on 10.10.0.2:8137), rather than a
# sudo-free systemd --user stack on someone's Wi-Fi.
#
# $APP_DIR is a checkout of the monorepo (deploy assets under infra/), the same
# way /opt/thermograph is for prod. A laptop can still point APP_DIR at a
# checkout under $HOME and get the old local behaviour.
#
# Design: a thin wrapper around deploy.sh, not a duplicate. deploy.sh already owns
# the entire per-service registry-pull mechanism -- secrets sourcing, docker login,
# the retry-pull loop, --no-deps single-service rolls vs. --remove-orphans `all`,
# .image-tags.env persistence, and the 8137/8080 health checks. None of that is
infra: split the estate into vps1/vps2, moving beta next to prod and dev onto vps1 Environments stop being machines. Until now each box WAS an environment -- "beta" named both a deploy target and a host, "the desktop" named both the operator's computer and the dev server -- so every path could assume one environment per host and hardcode /opt/thermograph, /etc/thermograph.env and ports 8137/8080. That assumption ends here: vps1 75.119.132.91 Forgejo, Grafana/Loki, the portfolio site, and DEV (own Postgres, mesh-only on 10.10.0.2:8137) vps2 169.58.46.181 PROD and BETA as two Swarm stacks sharing one TimescaleDB instance, plus Centralis, Postfix, backups desktop AI model hosting + flex Swarm capacity, no environment Nothing live has moved; the ordered cutover is in infra/deploy/RUNBOOK-vps1-vps2-cutover.md. deploy/env-topology.sh is the single source of truth: env -> host, checkout, branch, deploy mode, stack name, env file, LB ports, DB role/database, service prefix. THERMOGRAPH_ENV is the input, and on vps2 it is the only thing distinguishing a beta deploy from a prod one -- deploy.sh refuses to run when it disagrees with the checkout it was invoked from. The host-wide secrets-env and deploy-mode markers survive only as a fallback for a single-environment box. Beta gets its own stack file rather than an overlay (a merge cannot REMOVE the db service, and beta having no database of its own is the point). Its services are prefixed beta-*: Swarm registers a service's short name as a DNS alias on every network it joins, so two stacks both calling a service `web` on the shared data network would let prod's frontend resolve a beta task. Prod's stack, env and LB config are untouched. One Postgres, two databases with two roles: deploy/db/provision-env-db.sh creates thermograph_beta (NOSUPERUSER, owns only its own database, CONNECT revoked from PUBLIC) plus a <role>_ro for ad-hoc queries, and refuses to run for the environment that owns the instance -- doing so would demote prod's bootstrap superuser. Fixes that co-residency would otherwise have broken silently: - CI secrets are keyed by host (VPS1_SSH_*, VPS2_SSH_*). SSH_* meant "beta" and also "the Forgejo box" because those shared a machine; that conflation is what once had the prod backup dumping beta. - The nightly backup dumps BOTH application databases to separate off-box prefixes, and fails loudly on a missing one instead of skipping it. - Alloy stopped deriving the `host` label from the node -- on vps2 that would have filed every beta line as prod, feeding prod's alert rules with beta's traffic. It is now derived per source, with a new `node` label for the machine, and beta's log volume is mounted via a vps2-only overlay. - ops/dbq.sh, ops/iceberg.sh and the secrets seed scripts derived their SSH target and env-file path from the topology instead of hardcoding beta to 75.119.132.91 -- which is vps1's address now. - Dev's overlay binds DEV_BIND_ADDR (loopback by default, the mesh address on vps1) instead of 0.0.0.0: correct for a home LAN box, a public exposure of unreviewed branches on a VPS. Terraform's hosts variable is keyed by machine with a nested environments map, and main.tf flattens (host, environment) pairs so two environments on one box cannot share a checkout. Caddy's single reference config is split per host. Docs, onboarding and the runbooks are updated throughout, including the security rationales that co-residency makes false: separate SSH credentials no longer put a host boundary between beta and prod, and the boundary that remains is the database and the filesystem.
2026-07-25 22:01:29 +00:00
# dev-specific; the only things dev actually changes are WHERE it deploys (a
# separate checkout + branch), WHICH compose files are in play (the base file
# plus docker-compose.dev.yml's uncapped/mesh-published overrides), and its
# secrets policy (below). All are expressible as environment (APP_DIR/BRANCH
# that deploy.sh already reads, and docker compose's own COMPOSE_FILE
# variable), so re-exec'ing deploy.sh with that environment set covers it with
# no forked copy of the pull/roll/health logic to drift out of sync. If dev ever
# needs deploy logic that genuinely diverges from beta/prod (not just "different
# files/directory"), promote this to a standalone script at that point rather
# than growing special cases into deploy.sh.
#
# Usage, mirrors deploy.sh directly:
# # roll just the backend onto a dev-tagged image:
infra: split the estate into vps1/vps2, moving beta next to prod and dev onto vps1 Environments stop being machines. Until now each box WAS an environment -- "beta" named both a deploy target and a host, "the desktop" named both the operator's computer and the dev server -- so every path could assume one environment per host and hardcode /opt/thermograph, /etc/thermograph.env and ports 8137/8080. That assumption ends here: vps1 75.119.132.91 Forgejo, Grafana/Loki, the portfolio site, and DEV (own Postgres, mesh-only on 10.10.0.2:8137) vps2 169.58.46.181 PROD and BETA as two Swarm stacks sharing one TimescaleDB instance, plus Centralis, Postfix, backups desktop AI model hosting + flex Swarm capacity, no environment Nothing live has moved; the ordered cutover is in infra/deploy/RUNBOOK-vps1-vps2-cutover.md. deploy/env-topology.sh is the single source of truth: env -> host, checkout, branch, deploy mode, stack name, env file, LB ports, DB role/database, service prefix. THERMOGRAPH_ENV is the input, and on vps2 it is the only thing distinguishing a beta deploy from a prod one -- deploy.sh refuses to run when it disagrees with the checkout it was invoked from. The host-wide secrets-env and deploy-mode markers survive only as a fallback for a single-environment box. Beta gets its own stack file rather than an overlay (a merge cannot REMOVE the db service, and beta having no database of its own is the point). Its services are prefixed beta-*: Swarm registers a service's short name as a DNS alias on every network it joins, so two stacks both calling a service `web` on the shared data network would let prod's frontend resolve a beta task. Prod's stack, env and LB config are untouched. One Postgres, two databases with two roles: deploy/db/provision-env-db.sh creates thermograph_beta (NOSUPERUSER, owns only its own database, CONNECT revoked from PUBLIC) plus a <role>_ro for ad-hoc queries, and refuses to run for the environment that owns the instance -- doing so would demote prod's bootstrap superuser. Fixes that co-residency would otherwise have broken silently: - CI secrets are keyed by host (VPS1_SSH_*, VPS2_SSH_*). SSH_* meant "beta" and also "the Forgejo box" because those shared a machine; that conflation is what once had the prod backup dumping beta. - The nightly backup dumps BOTH application databases to separate off-box prefixes, and fails loudly on a missing one instead of skipping it. - Alloy stopped deriving the `host` label from the node -- on vps2 that would have filed every beta line as prod, feeding prod's alert rules with beta's traffic. It is now derived per source, with a new `node` label for the machine, and beta's log volume is mounted via a vps2-only overlay. - ops/dbq.sh, ops/iceberg.sh and the secrets seed scripts derived their SSH target and env-file path from the topology instead of hardcoding beta to 75.119.132.91 -- which is vps1's address now. - Dev's overlay binds DEV_BIND_ADDR (loopback by default, the mesh address on vps1) instead of 0.0.0.0: correct for a home LAN box, a public exposure of unreviewed branches on a VPS. Terraform's hosts variable is keyed by machine with a nested environments map, and main.tf flattens (host, environment) pairs so two environments on one box cannot share a checkout. Caddy's single reference config is split per host. Docs, onboarding and the runbooks are updated throughout, including the security rationales that co-residency makes false: separate SSH credentials no longer put a host boundary between beta and prod, and the boundary that remains is the database and the filesystem.
2026-07-25 22:01:29 +00:00
# ssh vps1 'SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> /opt/thermograph-dev/infra/deploy/deploy-dev.sh'
# # bring the whole dev stack up (both tags required, same as deploy.sh):
infra: split the estate into vps1/vps2, moving beta next to prod and dev onto vps1 Environments stop being machines. Until now each box WAS an environment -- "beta" named both a deploy target and a host, "the desktop" named both the operator's computer and the dev server -- so every path could assume one environment per host and hardcode /opt/thermograph, /etc/thermograph.env and ports 8137/8080. That assumption ends here: vps1 75.119.132.91 Forgejo, Grafana/Loki, the portfolio site, and DEV (own Postgres, mesh-only on 10.10.0.2:8137) vps2 169.58.46.181 PROD and BETA as two Swarm stacks sharing one TimescaleDB instance, plus Centralis, Postfix, backups desktop AI model hosting + flex Swarm capacity, no environment Nothing live has moved; the ordered cutover is in infra/deploy/RUNBOOK-vps1-vps2-cutover.md. deploy/env-topology.sh is the single source of truth: env -> host, checkout, branch, deploy mode, stack name, env file, LB ports, DB role/database, service prefix. THERMOGRAPH_ENV is the input, and on vps2 it is the only thing distinguishing a beta deploy from a prod one -- deploy.sh refuses to run when it disagrees with the checkout it was invoked from. The host-wide secrets-env and deploy-mode markers survive only as a fallback for a single-environment box. Beta gets its own stack file rather than an overlay (a merge cannot REMOVE the db service, and beta having no database of its own is the point). Its services are prefixed beta-*: Swarm registers a service's short name as a DNS alias on every network it joins, so two stacks both calling a service `web` on the shared data network would let prod's frontend resolve a beta task. Prod's stack, env and LB config are untouched. One Postgres, two databases with two roles: deploy/db/provision-env-db.sh creates thermograph_beta (NOSUPERUSER, owns only its own database, CONNECT revoked from PUBLIC) plus a <role>_ro for ad-hoc queries, and refuses to run for the environment that owns the instance -- doing so would demote prod's bootstrap superuser. Fixes that co-residency would otherwise have broken silently: - CI secrets are keyed by host (VPS1_SSH_*, VPS2_SSH_*). SSH_* meant "beta" and also "the Forgejo box" because those shared a machine; that conflation is what once had the prod backup dumping beta. - The nightly backup dumps BOTH application databases to separate off-box prefixes, and fails loudly on a missing one instead of skipping it. - Alloy stopped deriving the `host` label from the node -- on vps2 that would have filed every beta line as prod, feeding prod's alert rules with beta's traffic. It is now derived per source, with a new `node` label for the machine, and beta's log volume is mounted via a vps2-only overlay. - ops/dbq.sh, ops/iceberg.sh and the secrets seed scripts derived their SSH target and env-file path from the topology instead of hardcoding beta to 75.119.132.91 -- which is vps1's address now. - Dev's overlay binds DEV_BIND_ADDR (loopback by default, the mesh address on vps1) instead of 0.0.0.0: correct for a home LAN box, a public exposure of unreviewed branches on a VPS. Terraform's hosts variable is keyed by machine with a nested environments map, and main.tf flattens (host, environment) pairs so two environments on one box cannot share a checkout. Caddy's single reference config is split per host. Docs, onboarding and the runbooks are updated throughout, including the security rationales that co-residency makes false: separate SSH credentials no longer put a host boundary between beta and prod, and the boundary that remains is the database and the filesystem.
2026-07-25 22:01:29 +00:00
# ssh vps1 'SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> /opt/thermograph-dev/infra/deploy/deploy-dev.sh'
set -euo pipefail
infra: split the estate into vps1/vps2, moving beta next to prod and dev onto vps1 Environments stop being machines. Until now each box WAS an environment -- "beta" named both a deploy target and a host, "the desktop" named both the operator's computer and the dev server -- so every path could assume one environment per host and hardcode /opt/thermograph, /etc/thermograph.env and ports 8137/8080. That assumption ends here: vps1 75.119.132.91 Forgejo, Grafana/Loki, the portfolio site, and DEV (own Postgres, mesh-only on 10.10.0.2:8137) vps2 169.58.46.181 PROD and BETA as two Swarm stacks sharing one TimescaleDB instance, plus Centralis, Postfix, backups desktop AI model hosting + flex Swarm capacity, no environment Nothing live has moved; the ordered cutover is in infra/deploy/RUNBOOK-vps1-vps2-cutover.md. deploy/env-topology.sh is the single source of truth: env -> host, checkout, branch, deploy mode, stack name, env file, LB ports, DB role/database, service prefix. THERMOGRAPH_ENV is the input, and on vps2 it is the only thing distinguishing a beta deploy from a prod one -- deploy.sh refuses to run when it disagrees with the checkout it was invoked from. The host-wide secrets-env and deploy-mode markers survive only as a fallback for a single-environment box. Beta gets its own stack file rather than an overlay (a merge cannot REMOVE the db service, and beta having no database of its own is the point). Its services are prefixed beta-*: Swarm registers a service's short name as a DNS alias on every network it joins, so two stacks both calling a service `web` on the shared data network would let prod's frontend resolve a beta task. Prod's stack, env and LB config are untouched. One Postgres, two databases with two roles: deploy/db/provision-env-db.sh creates thermograph_beta (NOSUPERUSER, owns only its own database, CONNECT revoked from PUBLIC) plus a <role>_ro for ad-hoc queries, and refuses to run for the environment that owns the instance -- doing so would demote prod's bootstrap superuser. Fixes that co-residency would otherwise have broken silently: - CI secrets are keyed by host (VPS1_SSH_*, VPS2_SSH_*). SSH_* meant "beta" and also "the Forgejo box" because those shared a machine; that conflation is what once had the prod backup dumping beta. - The nightly backup dumps BOTH application databases to separate off-box prefixes, and fails loudly on a missing one instead of skipping it. - Alloy stopped deriving the `host` label from the node -- on vps2 that would have filed every beta line as prod, feeding prod's alert rules with beta's traffic. It is now derived per source, with a new `node` label for the machine, and beta's log volume is mounted via a vps2-only overlay. - ops/dbq.sh, ops/iceberg.sh and the secrets seed scripts derived their SSH target and env-file path from the topology instead of hardcoding beta to 75.119.132.91 -- which is vps1's address now. - Dev's overlay binds DEV_BIND_ADDR (loopback by default, the mesh address on vps1) instead of 0.0.0.0: correct for a home LAN box, a public exposure of unreviewed branches on a VPS. Terraform's hosts variable is keyed by machine with a nested environments map, and main.tf flattens (host, environment) pairs so two environments on one box cannot share a checkout. Caddy's single reference config is split per host. Docs, onboarding and the runbooks are updated throughout, including the security rationales that co-residency makes false: separate SSH credentials no longer put a host boundary between beta and prod, and the boundary that remains is the database and the filesystem.
2026-07-25 22:01:29 +00:00
# Dev context: its own checkout and its own branch, on its own host (vps1).
# Both come from deploy/env-topology.sh so there is one place that says where
# dev lives. $APP_DIR is still honoured when set explicitly — that is how a
# laptop points this at a checkout under $HOME.
_dev_self="$(cd "$(dirname "$0")" && pwd)"
# shellcheck source=infra/deploy/env-topology.sh
. "$_dev_self/env-topology.sh"
thermograph_topology dev
APP_DIR="${APP_DIR:-$TG_APP_DIR}"
BRANCH="${BRANCH:-$TG_BRANCH}"
export APP_DIR BRANCH
infra: split the estate into vps1/vps2, moving beta next to prod and dev onto vps1 Environments stop being machines. Until now each box WAS an environment -- "beta" named both a deploy target and a host, "the desktop" named both the operator's computer and the dev server -- so every path could assume one environment per host and hardcode /opt/thermograph, /etc/thermograph.env and ports 8137/8080. That assumption ends here: vps1 75.119.132.91 Forgejo, Grafana/Loki, the portfolio site, and DEV (own Postgres, mesh-only on 10.10.0.2:8137) vps2 169.58.46.181 PROD and BETA as two Swarm stacks sharing one TimescaleDB instance, plus Centralis, Postfix, backups desktop AI model hosting + flex Swarm capacity, no environment Nothing live has moved; the ordered cutover is in infra/deploy/RUNBOOK-vps1-vps2-cutover.md. deploy/env-topology.sh is the single source of truth: env -> host, checkout, branch, deploy mode, stack name, env file, LB ports, DB role/database, service prefix. THERMOGRAPH_ENV is the input, and on vps2 it is the only thing distinguishing a beta deploy from a prod one -- deploy.sh refuses to run when it disagrees with the checkout it was invoked from. The host-wide secrets-env and deploy-mode markers survive only as a fallback for a single-environment box. Beta gets its own stack file rather than an overlay (a merge cannot REMOVE the db service, and beta having no database of its own is the point). Its services are prefixed beta-*: Swarm registers a service's short name as a DNS alias on every network it joins, so two stacks both calling a service `web` on the shared data network would let prod's frontend resolve a beta task. Prod's stack, env and LB config are untouched. One Postgres, two databases with two roles: deploy/db/provision-env-db.sh creates thermograph_beta (NOSUPERUSER, owns only its own database, CONNECT revoked from PUBLIC) plus a <role>_ro for ad-hoc queries, and refuses to run for the environment that owns the instance -- doing so would demote prod's bootstrap superuser. Fixes that co-residency would otherwise have broken silently: - CI secrets are keyed by host (VPS1_SSH_*, VPS2_SSH_*). SSH_* meant "beta" and also "the Forgejo box" because those shared a machine; that conflation is what once had the prod backup dumping beta. - The nightly backup dumps BOTH application databases to separate off-box prefixes, and fails loudly on a missing one instead of skipping it. - Alloy stopped deriving the `host` label from the node -- on vps2 that would have filed every beta line as prod, feeding prod's alert rules with beta's traffic. It is now derived per source, with a new `node` label for the machine, and beta's log volume is mounted via a vps2-only overlay. - ops/dbq.sh, ops/iceberg.sh and the secrets seed scripts derived their SSH target and env-file path from the topology instead of hardcoding beta to 75.119.132.91 -- which is vps1's address now. - Dev's overlay binds DEV_BIND_ADDR (loopback by default, the mesh address on vps1) instead of 0.0.0.0: correct for a home LAN box, a public exposure of unreviewed branches on a VPS. Terraform's hosts variable is keyed by machine with a nested environments map, and main.tf flattens (host, environment) pairs so two environments on one box cannot share a checkout. Caddy's single reference config is split per host. Docs, onboarding and the runbooks are updated throughout, including the security rationales that co-residency makes false: separate SSH credentials no longer put a host boundary between beta and prod, and the boundary that remains is the database and the filesystem.
2026-07-25 22:01:29 +00:00
# deploy.sh resolves the environment itself; say so explicitly rather than
# letting it fall back to a host marker that, on vps1, names dev anyway.
export THERMOGRAPH_ENV=dev
# Point every `docker compose` invocation inside deploy.sh at the LAN-dev overlay
# (uncapped CPU, backend published on 0.0.0.0:8137, frontend unpublished -- see
# docker-compose.dev.yml) without deploy.sh needing to know dev exists at all.
# COMPOSE_FILE is docker compose's own env var for this; ':' is the Linux/macOS
# path-list separator it expects (';' only on Windows).
export COMPOSE_FILE="docker-compose.yml:docker-compose.dev.yml"
# Pin the dev project name: docker-compose.yml pins `name: thermograph` for
# prod/beta, but dev's stack has always been the separate "thermograph-dev"
# project (originally derived from this checkout's directory name). The env
# var wins over the compose-file `name:` key, preserving that split.
export COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-thermograph-dev}"
secrets: vault dev and Centralis; render dev without common.yaml Brings the last two credential stores under SOPS, so all three environments and the control plane are managed the same way. dev.yaml — dev was not "intentionally not wired in", it was broken The vault README claimed dev had no real secrets. Both halves were false. Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls back to https://thermograph.org, so dev's IndexNow pings and verification links claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real base URL, and 12 values total. And dev already held production credentials: backend-deploy-dev.yml injects S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned keypair for that bucket is read-write — on the bucket holding prod's backups. Dev does not need them; without bucket creds the lake service 503s and history falls through to the archive. Those workflow lines still need removing. dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of common.yaml's sixteen values are live production credentials, the render is a plaintext concatenation consumed through env_file:, and dev is the operator's desktop AND the Forgejo runner executing unreviewed dev-branch code with the docker socket mounted. An override in dev.yaml would not help — last-wins governs consumers, but prod's value is still physically a line in the file. Verified: current renderer leaks 28 lines onto dev including both S3 keypairs and the VAPID private key; patched gives exactly 12 keys and no prod credential. Beta's render is byte-identical either way. centralis.prod.yaml — its own file, its own renderer /etc/centralis.env was hand-edited, which is how a JSON registry got written unquoted into a shell-sourced file tonight and silently collapsed three identities to one. The renderer now owns the file. It is service- AND environment-scoped, not folded into prod.yaml, because /etc/thermograph.env is loaded into every container in the app stack. Folding these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which authenticate an endpoint carrying run_on_host and sql_query(write) — into the web backend's environment, turning a read-anything bug in the app into a foothold on the control plane. Quoting is guaranteed three ways rather than assumed. Critically, `sops -d --output-type dotenv` emits values verbatim, so reusing the existing render path would have reproduced tonight's bug exactly. The new function decrypts to JSON and emits POSIX single-quoted assignments; it then sources its own output in a clean shell and compares every value before touching /etc; and it refuses to write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after sourcing. Tested against 16 hostile values including embedded quotes, newlines and ;rm -rf /. Value-identity proven, not asserted: rendered vs live compared as *effective* values on prod (a textual diff would report a false difference, and would report a false match if the vault had captured quote characters as part of the value), plus both env files producing a byte-identical `docker compose config` digest. 9 keys, both token subjects preserved. Nothing deployed. Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw dotenv only because none of its 32 current values contains a shell-special character. It is one quoted secret away from the same bug. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-25 01:10:51 +00:00
# --- Secrets -----------------------------------------------------------------
# dev renders from the SOPS vault the same way prod and beta do -- same files, same
# tool, same edit/commit/deploy loop -- with two dev-only adjustments. Both are
# expressed as environment that render-secrets.sh already parameterises, so nothing
# about the rendering logic is forked. See deploy/secrets/README.md.
# (1) dev renders deploy/secrets/dev.yaml ALONE; it does NOT layer common.yaml.
#
# common.yaml is the internet-facing fleet's SHARED PRODUCTION credential set: the
# registry token, both S3 keypairs (the bucket keypair is read-write, and that
# bucket also holds prod's database backups), the VAPID private key that signs push
# to real subscribers, the IndexNow key, the metrics token. Layering it here would
# write all of them in plaintext to /etc/thermograph.env on the box that is also the
# Forgejo CI runner, and hand them to every container in the dev stack through
# `env_file:` -- that is, to whatever unreviewed branch dev happens to be running.
# It would also silently apply THERMOGRAPH_COOKIE_SECURE=1, which is correct for the
# TLS hosts and breaks login on dev's plain-HTTP LAN URL.
#
# dev.yaml is self-contained instead: it carries dev's own copy of everything dev
# genuinely needs, and nothing else.
export THERMOGRAPH_SECRETS_SKIP_COMMON=1
infra: split the estate into vps1/vps2, moving beta next to prod and dev onto vps1 Environments stop being machines. Until now each box WAS an environment -- "beta" named both a deploy target and a host, "the desktop" named both the operator's computer and the dev server -- so every path could assume one environment per host and hardcode /opt/thermograph, /etc/thermograph.env and ports 8137/8080. That assumption ends here: vps1 75.119.132.91 Forgejo, Grafana/Loki, the portfolio site, and DEV (own Postgres, mesh-only on 10.10.0.2:8137) vps2 169.58.46.181 PROD and BETA as two Swarm stacks sharing one TimescaleDB instance, plus Centralis, Postfix, backups desktop AI model hosting + flex Swarm capacity, no environment Nothing live has moved; the ordered cutover is in infra/deploy/RUNBOOK-vps1-vps2-cutover.md. deploy/env-topology.sh is the single source of truth: env -> host, checkout, branch, deploy mode, stack name, env file, LB ports, DB role/database, service prefix. THERMOGRAPH_ENV is the input, and on vps2 it is the only thing distinguishing a beta deploy from a prod one -- deploy.sh refuses to run when it disagrees with the checkout it was invoked from. The host-wide secrets-env and deploy-mode markers survive only as a fallback for a single-environment box. Beta gets its own stack file rather than an overlay (a merge cannot REMOVE the db service, and beta having no database of its own is the point). Its services are prefixed beta-*: Swarm registers a service's short name as a DNS alias on every network it joins, so two stacks both calling a service `web` on the shared data network would let prod's frontend resolve a beta task. Prod's stack, env and LB config are untouched. One Postgres, two databases with two roles: deploy/db/provision-env-db.sh creates thermograph_beta (NOSUPERUSER, owns only its own database, CONNECT revoked from PUBLIC) plus a <role>_ro for ad-hoc queries, and refuses to run for the environment that owns the instance -- doing so would demote prod's bootstrap superuser. Fixes that co-residency would otherwise have broken silently: - CI secrets are keyed by host (VPS1_SSH_*, VPS2_SSH_*). SSH_* meant "beta" and also "the Forgejo box" because those shared a machine; that conflation is what once had the prod backup dumping beta. - The nightly backup dumps BOTH application databases to separate off-box prefixes, and fails loudly on a missing one instead of skipping it. - Alloy stopped deriving the `host` label from the node -- on vps2 that would have filed every beta line as prod, feeding prod's alert rules with beta's traffic. It is now derived per source, with a new `node` label for the machine, and beta's log volume is mounted via a vps2-only overlay. - ops/dbq.sh, ops/iceberg.sh and the secrets seed scripts derived their SSH target and env-file path from the topology instead of hardcoding beta to 75.119.132.91 -- which is vps1's address now. - Dev's overlay binds DEV_BIND_ADDR (loopback by default, the mesh address on vps1) instead of 0.0.0.0: correct for a home LAN box, a public exposure of unreviewed branches on a VPS. Terraform's hosts variable is keyed by machine with a nested environments map, and main.tf flattens (host, environment) pairs so two environments on one box cannot share a checkout. Caddy's single reference config is split per host. Docs, onboarding and the runbooks are updated throughout, including the security rationales that co-residency makes false: separate SSH credentials no longer put a host boundary between beta and prod, and the boundary that remains is the database and the filesystem.
2026-07-25 22:01:29 +00:00
# (1b) Snap-packaged Docker is confined to $HOME (plus a short allowlist) and
# silently can't see /etc/thermograph.env at all, so env_file: /etc/thermograph.env
# resolves to nothing for every container no matter how correctly it's rendered.
# Where that is the case, mirror the render to a path under $APP_DIR;
# docker-compose.dev.yml carries a matching optional env_file entry.
#
# CONDITIONAL, not unconditional: this writes a plaintext copy of the render
# into the checkout, and on vps1 (ordinary apt Docker, which reads /etc fine)
# that copy would be pure liability. Detect the confinement rather than add a
# knob someone has to remember — the docker binary resolving under /snap is
# exactly the condition that breaks the /etc path.
_docker_bin="$(command -v docker || true)"
if [ -n "$_docker_bin" ] && case "$(readlink -f "$_docker_bin")" in /snap/*) true ;; *) false ;; esac; then
export THERMOGRAPH_SECRETS_ENV_FILE_MIRROR="$APP_DIR/infra/deploy/dev-secrets.env"
fi
secrets: vault dev and Centralis; render dev without common.yaml Brings the last two credential stores under SOPS, so all three environments and the control plane are managed the same way. dev.yaml — dev was not "intentionally not wired in", it was broken The vault README claimed dev had no real secrets. Both halves were false. Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls back to https://thermograph.org, so dev's IndexNow pings and verification links claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real base URL, and 12 values total. And dev already held production credentials: backend-deploy-dev.yml injects S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned keypair for that bucket is read-write — on the bucket holding prod's backups. Dev does not need them; without bucket creds the lake service 503s and history falls through to the archive. Those workflow lines still need removing. dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of common.yaml's sixteen values are live production credentials, the render is a plaintext concatenation consumed through env_file:, and dev is the operator's desktop AND the Forgejo runner executing unreviewed dev-branch code with the docker socket mounted. An override in dev.yaml would not help — last-wins governs consumers, but prod's value is still physically a line in the file. Verified: current renderer leaks 28 lines onto dev including both S3 keypairs and the VAPID private key; patched gives exactly 12 keys and no prod credential. Beta's render is byte-identical either way. centralis.prod.yaml — its own file, its own renderer /etc/centralis.env was hand-edited, which is how a JSON registry got written unquoted into a shell-sourced file tonight and silently collapsed three identities to one. The renderer now owns the file. It is service- AND environment-scoped, not folded into prod.yaml, because /etc/thermograph.env is loaded into every container in the app stack. Folding these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which authenticate an endpoint carrying run_on_host and sql_query(write) — into the web backend's environment, turning a read-anything bug in the app into a foothold on the control plane. Quoting is guaranteed three ways rather than assumed. Critically, `sops -d --output-type dotenv` emits values verbatim, so reusing the existing render path would have reproduced tonight's bug exactly. The new function decrypts to JSON and emits POSIX single-quoted assignments; it then sources its own output in a clean shell and compares every value before touching /etc; and it refuses to write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after sourcing. Tested against 16 hostile values including embedded quotes, newlines and ;rm -rf /. Value-identity proven, not asserted: rendered vs live compared as *effective* values on prod (a textual diff would report a false difference, and would report a false match if the vault had captured quote characters as part of the value), plus both env files producing a byte-identical `docker compose config` digest. 9 keys, both token subjects preserved. Nothing deployed. Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw dotenv only because none of its 32 current values contains a shell-special character. It is one quoted secret away from the same bug. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-25 01:10:51 +00:00
# (2) The age key is read from wherever it is READABLE, not necessarily
# /etc/thermograph/age.key. render-secrets.sh falls back to `sudo cat` for a
infra: split the estate into vps1/vps2, moving beta next to prod and dev onto vps1 Environments stop being machines. Until now each box WAS an environment -- "beta" named both a deploy target and a host, "the desktop" named both the operator's computer and the dev server -- so every path could assume one environment per host and hardcode /opt/thermograph, /etc/thermograph.env and ports 8137/8080. That assumption ends here: vps1 75.119.132.91 Forgejo, Grafana/Loki, the portfolio site, and DEV (own Postgres, mesh-only on 10.10.0.2:8137) vps2 169.58.46.181 PROD and BETA as two Swarm stacks sharing one TimescaleDB instance, plus Centralis, Postfix, backups desktop AI model hosting + flex Swarm capacity, no environment Nothing live has moved; the ordered cutover is in infra/deploy/RUNBOOK-vps1-vps2-cutover.md. deploy/env-topology.sh is the single source of truth: env -> host, checkout, branch, deploy mode, stack name, env file, LB ports, DB role/database, service prefix. THERMOGRAPH_ENV is the input, and on vps2 it is the only thing distinguishing a beta deploy from a prod one -- deploy.sh refuses to run when it disagrees with the checkout it was invoked from. The host-wide secrets-env and deploy-mode markers survive only as a fallback for a single-environment box. Beta gets its own stack file rather than an overlay (a merge cannot REMOVE the db service, and beta having no database of its own is the point). Its services are prefixed beta-*: Swarm registers a service's short name as a DNS alias on every network it joins, so two stacks both calling a service `web` on the shared data network would let prod's frontend resolve a beta task. Prod's stack, env and LB config are untouched. One Postgres, two databases with two roles: deploy/db/provision-env-db.sh creates thermograph_beta (NOSUPERUSER, owns only its own database, CONNECT revoked from PUBLIC) plus a <role>_ro for ad-hoc queries, and refuses to run for the environment that owns the instance -- doing so would demote prod's bootstrap superuser. Fixes that co-residency would otherwise have broken silently: - CI secrets are keyed by host (VPS1_SSH_*, VPS2_SSH_*). SSH_* meant "beta" and also "the Forgejo box" because those shared a machine; that conflation is what once had the prod backup dumping beta. - The nightly backup dumps BOTH application databases to separate off-box prefixes, and fails loudly on a missing one instead of skipping it. - Alloy stopped deriving the `host` label from the node -- on vps2 that would have filed every beta line as prod, feeding prod's alert rules with beta's traffic. It is now derived per source, with a new `node` label for the machine, and beta's log volume is mounted via a vps2-only overlay. - ops/dbq.sh, ops/iceberg.sh and the secrets seed scripts derived their SSH target and env-file path from the topology instead of hardcoding beta to 75.119.132.91 -- which is vps1's address now. - Dev's overlay binds DEV_BIND_ADDR (loopback by default, the mesh address on vps1) instead of 0.0.0.0: correct for a home LAN box, a public exposure of unreviewed branches on a VPS. Terraform's hosts variable is keyed by machine with a nested environments map, and main.tf flattens (host, environment) pairs so two environments on one box cannot share a checkout. Caddy's single reference config is split per host. Docs, onboarding and the runbooks are updated throughout, including the security rationales that co-residency makes false: separate SSH credentials no longer put a host boundary between beta and prod, and the boundary that remains is the database and the filesystem.
2026-07-25 22:01:29 +00:00
# root-owned 0400 key, which needs passwordless sudo.
#
# On vps1 that is the normal case: the `agent` user has passwordless sudo and
# the key sits at the fleet-standard path, so this block does nothing. The
# fallback exists for the laptop case — a checkout where the key lives only in
# the operator's own keyring (that machine is where `sops` is run), and where a
# non-interactive shell could not prompt for sudo even if the key were in /etc.
# Preferring the standard path when readable keeps vps1 on the fleet convention
# and means provisioning dev needs no second copy of the estate's recovery root.
secrets: vault dev and Centralis; render dev without common.yaml Brings the last two credential stores under SOPS, so all three environments and the control plane are managed the same way. dev.yaml — dev was not "intentionally not wired in", it was broken The vault README claimed dev had no real secrets. Both halves were false. Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls back to https://thermograph.org, so dev's IndexNow pings and verification links claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real base URL, and 12 values total. And dev already held production credentials: backend-deploy-dev.yml injects S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned keypair for that bucket is read-write — on the bucket holding prod's backups. Dev does not need them; without bucket creds the lake service 503s and history falls through to the archive. Those workflow lines still need removing. dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of common.yaml's sixteen values are live production credentials, the render is a plaintext concatenation consumed through env_file:, and dev is the operator's desktop AND the Forgejo runner executing unreviewed dev-branch code with the docker socket mounted. An override in dev.yaml would not help — last-wins governs consumers, but prod's value is still physically a line in the file. Verified: current renderer leaks 28 lines onto dev including both S3 keypairs and the VAPID private key; patched gives exactly 12 keys and no prod credential. Beta's render is byte-identical either way. centralis.prod.yaml — its own file, its own renderer /etc/centralis.env was hand-edited, which is how a JSON registry got written unquoted into a shell-sourced file tonight and silently collapsed three identities to one. The renderer now owns the file. It is service- AND environment-scoped, not folded into prod.yaml, because /etc/thermograph.env is loaded into every container in the app stack. Folding these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which authenticate an endpoint carrying run_on_host and sql_query(write) — into the web backend's environment, turning a read-anything bug in the app into a foothold on the control plane. Quoting is guaranteed three ways rather than assumed. Critically, `sops -d --output-type dotenv` emits values verbatim, so reusing the existing render path would have reproduced tonight's bug exactly. The new function decrypts to JSON and emits POSIX single-quoted assignments; it then sources its own output in a clean shell and compares every value before touching /etc; and it refuses to write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after sourcing. Tested against 16 hostile values including embedded quotes, newlines and ;rm -rf /. Value-identity proven, not asserted: rendered vs live compared as *effective* values on prod (a textual diff would report a false difference, and would report a false match if the vault had captured quote characters as part of the value), plus both env files producing a byte-identical `docker compose config` digest. 9 keys, both token subjects preserved. Nothing deployed. Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw dotenv only because none of its 32 current values contains a shell-special character. It is one quoted secret away from the same bug. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-25 01:10:51 +00:00
if [ -z "${THERMOGRAPH_AGE_KEY:-}" ] && [ ! -r /etc/thermograph/age.key ] \
&& [ -r "$HOME/.config/sops/age/keys.txt" ]; then
export THERMOGRAPH_AGE_KEY="$HOME/.config/sops/age/keys.txt"
fi
# Fail closed rather than leak. If this box is provisioned to render (an env marker
# AND a key) but the render-secrets.sh in this checkout predates
# THERMOGRAPH_SECRETS_SKIP_COMMON, the render would quietly concatenate common.yaml
# into dev's /etc/thermograph.env -- exactly the leak (1) exists to prevent, with a
# success exit code. Checking a sibling script for the flag is blunt, but the two
# files are versioned together in this same checkout, and the alternative failure
# mode is unreviewed code running with the estate's object-storage keys. Remove this
# once the flag is unconditionally present in render-secrets.sh.
_render_sh="$(dirname "$0")/render-secrets.sh"
_secrets_marker="${THERMOGRAPH_SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}"
_age_key="${THERMOGRAPH_AGE_KEY:-/etc/thermograph/age.key}"
if [ -s "$_secrets_marker" ] && [ -f "$_age_key" ] && [ -f "$_render_sh" ] \
&& ! grep -q 'THERMOGRAPH_SECRETS_SKIP_COMMON' "$_render_sh"; then
echo "!! This box is provisioned to render SOPS secrets (env marker + age key)," >&2
echo "!! but $_render_sh does not honour" >&2
echo "!! THERMOGRAPH_SECRETS_SKIP_COMMON, so it would render common.yaml -- the" >&2
echo "!! fleet's shared PRODUCTION credentials -- into /etc/thermograph.env on the" >&2
echo "!! CI-runner box. Refusing to deploy." >&2
echo "!! Fix: update deploy/render-secrets.sh, or remove $_secrets_marker to fall" >&2
echo "!! back to the un-vaulted dev path." >&2
exit 1
fi
# deploy.sh needs POSTGRES_PASSWORD to interpolate the compose file (both to init the
# db container and to build backend's THERMOGRAPH_DATABASE_URL). On a provisioned dev
# box it arrives from dev.yaml via the render above -- deploy.sh sources
# /etc/thermograph.env AFTER calling the renderer, so the vault value wins over this
# line. This default is the fallback for the un-vaulted paths that remain: a fresh
# box before provisioning, and a bare `make dev-up`. It is the same value dev.yaml
# holds and the value dev's existing pgdata volume was initialized with -- keep the
# two in step, and note that changing it needs an ALTER ROLE against the running
# database as well (see deploy/secrets/README.md), not just an edit.
export POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-thermograph-dev}"
secrets: vault dev and Centralis; render dev without common.yaml Brings the last two credential stores under SOPS, so all three environments and the control plane are managed the same way. dev.yaml — dev was not "intentionally not wired in", it was broken The vault README claimed dev had no real secrets. Both halves were false. Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls back to https://thermograph.org, so dev's IndexNow pings and verification links claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real base URL, and 12 values total. And dev already held production credentials: backend-deploy-dev.yml injects S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned keypair for that bucket is read-write — on the bucket holding prod's backups. Dev does not need them; without bucket creds the lake service 503s and history falls through to the archive. Those workflow lines still need removing. dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of common.yaml's sixteen values are live production credentials, the render is a plaintext concatenation consumed through env_file:, and dev is the operator's desktop AND the Forgejo runner executing unreviewed dev-branch code with the docker socket mounted. An override in dev.yaml would not help — last-wins governs consumers, but prod's value is still physically a line in the file. Verified: current renderer leaks 28 lines onto dev including both S3 keypairs and the VAPID private key; patched gives exactly 12 keys and no prod credential. Beta's render is byte-identical either way. centralis.prod.yaml — its own file, its own renderer /etc/centralis.env was hand-edited, which is how a JSON registry got written unquoted into a shell-sourced file tonight and silently collapsed three identities to one. The renderer now owns the file. It is service- AND environment-scoped, not folded into prod.yaml, because /etc/thermograph.env is loaded into every container in the app stack. Folding these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which authenticate an endpoint carrying run_on_host and sql_query(write) — into the web backend's environment, turning a read-anything bug in the app into a foothold on the control plane. Quoting is guaranteed three ways rather than assumed. Critically, `sops -d --output-type dotenv` emits values verbatim, so reusing the existing render path would have reproduced tonight's bug exactly. The new function decrypts to JSON and emits POSIX single-quoted assignments; it then sources its own output in a clean shell and compares every value before touching /etc; and it refuses to write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after sourcing. Tested against 16 hostile values including embedded quotes, newlines and ;rm -rf /. Value-identity proven, not asserted: rendered vs live compared as *effective* values on prod (a textual diff would report a false difference, and would report a false match if the vault had captured quote characters as part of the value), plus both env files producing a byte-identical `docker compose config` digest. 9 keys, both token subjects preserved. Nothing deployed. Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw dotenv only because none of its 32 current values contains a shell-special character. It is one quoted secret away from the same bug. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-25 01:10:51 +00:00
# REGISTRY_TOKEN (for `docker login` in deploy.sh) is deliberately NOT in dev.yaml
# and has no default here. dev pulls with the persistent `docker login` credential
# already in this host's docker config -- the same arrangement the prod/beta CI
# deploy paths use, and the reason deploy.sh's login step is conditional. Putting a
# registry token in dev's vault would only add a second copy of a credential the box
# already has, in a file more processes can read. Export it by hand for a one-off run
# on a box that has never logged in; a genuine auth problem fails loudly at `pull`.
echo "==> LAN-dev deploy: APP_DIR=$APP_DIR BRANCH=$BRANCH COMPOSE_FILE=$COMPOSE_FILE"
exec "$(dirname "$0")/deploy.sh" "$@"