Repos moved to the Jinemi org; the container packages did not follow, since
Forgejo does not transfer packages with a repo. The deploy path still resolved
`emi/thermograph/*` — a user_redirect to admin_emi — while build-push.yml
derives its push path from ${github.repository}, now jinemi/thermograph. The
next backend or frontend build would have published somewhere no deploy looks.
Point the image paths at jinemi/thermograph/* (lowercase: OCI references admit
no uppercase, which is why build-push.yml already pipes through tr), the clone
URLs at Jinemi/thermograph, and the registry logins at admin_emi — the account
that actually owns the tokens, rather than the redirect.
The live tags and both ci-runner tags were copied into the Jinemi namespace
first, so the switch has something to pull. thermograph-infra,
thermograph-observability and the retired */app packages stay under admin_emi;
they did not move.
render_centralis_secrets has existed since the OpenBao work and had no caller
on any deploy path -- only verify-centralis-render.sh, a check. A renderer
nothing calls is not a migration, it is a file. /etc/centralis.env has stayed
hand-maintained the whole time, with the vault holding a copy nothing consumed.
That gap is why turning on Google sign-in for Centralis had no safe home: the
only way to set CENTRALIS_GOOGLE_CLIENT_ID was to hand-edit the live file,
which is precisely what render-secrets.sh's header warns against.
THE GUARD IS THE POINT. On 2026-07-24 this estate lost Centralis' token
registry by writing a file missing a key the live one had. The renderer proves
its own output round-trips through bash, but it cannot know about a key that
exists only in the live file. So this job renders to a scratch path first,
compares KEY SETS (names only, both sides sourced in `env -i` so the
comparison is over what bash actually ends up with), and refuses to write --
failing loudly -- if the live file has anything the vault does not. Rendering
turns from a way to lose a hand-edit into the thing that catches one.
Verified the guard against a live file carrying a hand-added
CENTRALIS_GOOGLE_CLIENT_ID: correctly identified as a key that would be
deleted, write refused. Also verified the empty/unreadable-live-file path,
which needs `|| true` to survive pipefail.
It recreates the container where the other three jobs deliberately roll
nothing, and the difference is real rather than an exception: Centralis'
compose file interpolates /etc/centralis.env at PARSE time, so rendering
without recreating changes nothing -- the silent no-op this estate keeps
getting bitten by. `docker restart` would not do it either. One container, no
replicas, a two-second recreate. It also skips the recreate entirely when the
rendered file is byte-identical to the live one, so a no-op push does not
bounce a healthy container.
Three fixes, all on the path between "the migration looks ready" and "the
migration is ready".
verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and
render-secrets-openbao.sh fell through to the bare default -- prod's credential
-- for every environment. On vps2 that made `--env beta` authenticate as
tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta.
`--all` failed the same way, taking dev with it. deploy.sh and
deploy-stack.sh always sourced it; only the verifier did not, which is the
worst place for the omission: the tool whose job is to notice divergence was
itself diverging from the path it verifies. It now calls thermograph_topology
per environment and takes TG_SKIP_COMMON from there rather than re-deriving it.
infra-sync.yml renders all three env files on every push touching infra/**, and
none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was
therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the
selector would have changed deploy.sh and deploy-stack.sh but not this
workflow, which would have kept re-rendering from SOPS -- silent while parity
holds, and a hard failure the moment the SOPS files are retired. The one-line
cutover the README describes was never sufficient on its own.
ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from
vps1, nightly, failing loudly on a mismatch. README.md called this the gate for
a cutover; nothing implemented it. A gate that exists only in prose gets
satisfied by assertion. It grants CI no vault access -- the host renders, CI
only asks it to.
First measured run, immediately after the verifier fix:
dev 12 keys PASS, byte-identical
beta 24 keys FAIL, 3 values differ
prod 32 keys FAIL, 3 values differ
The same three keys on both: THERMOGRAPH_S3_SECRET_KEY,
THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in
common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed.
dev passes because dev never layers common. One `seed-from-sops.sh --env
common` fixes beta and prod together, and the 7-day clock starts after that.
The estate had exactly one registered runner, on the desktop. It went offline
at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required
check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both
application databases and for Forgejo -- did not fire. Forgejo queued that
scheduled run rather than dropping it, so it completed on reconnect and nothing
was lost. A longer outage would have meant real gaps.
Three files claimed an "always-on Swarm-hosted runner" existed and that the
estate therefore no longer depended on the desktop. It did not exist: an early
revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was
removed. That claim is why a single point of failure sat unnoticed. Corrected
in docker-stack.yml, forgejo/README.md and register-lan-runner.sh.
The new runner is a plain restart:always container, not a Swarm service: a
Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would
be gone exactly when the cluster is what is broken. It runs from
/opt/forgejo-runner rather than in place, because the checkout is reset on
every prod deploy and one `git clean -fdx` there would destroy the
registration.
vps2 runs prod, so the socket mount is bounded rather than assumed benign:
capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no
job can bind-mount /etc/thermograph.env, and no thermograph network joined.
This defends against accident, not against a hostile workflow author -- stated
plainly in the compose header rather than implied.
The desktop runner stays registered as extra capacity. Nothing may assume it
is up.
The estate had zero alert rules. The only contact point was Grafana's factory
default pointing at the literal string <example@email.com>, and beta's
untracked compose override routed Grafana's SMTP at prod's Postfix — so
alerts, had any existed, would have been delivered by the box most likely to
be on fire. Prod served 86 5xx in 24h including five /healthz failures and
nobody was told.
Alerting (deploys to beta, which is where Grafana runs):
- 12 Loki-based rules. There is no Prometheus in this estate and Loki is the
only datasource, so every rule is log-derived.
- Thresholds come from a 24h backtest that happens to contain a real ~20min
prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so
one bad request is 1.4% and a ratio alert would scream all night. The 5xx
burst rule fires on the outage's 45 and 22 buckets and on nothing else in
the day; the largest benign bucket all day was 5.
- Routed to a new private #ops-alerts channel, not to any existing channel —
#weather-events, #announcements and #prod are product surfaces that notify
real subscribers.
- AlertingWatchdog is a dead-man's switch; its value is its absence.
- Delivery is proven, not assumed: the rules were provisioned into a throwaway
Grafana against beta's live Loki and a real alert arrived in Discord. This
matters because GET /api/v1/provisioning/contact-points returns [REDACTED]
for the URL — a contact point holding an uninterpolated env var looks
perfectly healthy and pages nobody. The only proof is a message arriving.
CI gains a structural check, because the existing one only proves YAML parses:
an alert rule whose condition names a missing refId is valid YAML, provisions
cleanly, and never fires. It also hard-fails on a literal Discord webhook in
the repo. Verified against all three breakages deliberately introduced.
Postfix supervision:
- The 13h outage was a boot-ordering race, not a Docker renumbering: postfix
started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing
docker_gwbridge address, and dockerd did not finish starting until 08:10:16.
Stock postfix@.service is ordered only After=network-online.target and ships
no Restart=, so one lost race became a permanent outage.
- An ExecStartPre gate now blocks up to 60s until every inet_interfaces
address actually exists, which absorbs the transient case inside a single
start attempt. That makes bounded retry correct: 5 attempts in 600s, then
failed — a genuinely broken config reaches a visible failed state in ~100s
instead of re-fataling every 15s forever.
- A 5-minute watchdog timer retries indefinitely and runs reset-failed, so
"failed" still self-heals. Worst case is ~5 minutes, not 13 hours.
- Wants=, not Requires=: a dockerd failure must not take down the loopback and
mesh listeners that do not depend on Docker at all.
Health checks must read config with `postconf -c`, never postmulti/postqueue/
postfix — those three RESOLVE inet_interfaces and so fatal precisely when an
address is missing, which made the first version of this check report
status=ok bound=0/0. A health check that fails open is worse than none.
DEPLOY.md carries the monitoring contract, including that systemctl is-active
postfix is a known-false signal: postfix.service is a wrapper whose
ExecStart=/bin/true, so it reports active forever while the real postfix@-
instance is failed with zero listeners. Reproduced live.
Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and
app JSONL, not journald — so no Postfix line reaches Loki and the mail rules
cannot fire until loki.source.journal is added.
Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
main took two direct PRs (#19 shell-lint, #21 the thermograph-daemon Go
service) while dev accumulated the ERA5 lake stack; both sides added compose/
stack services and touched the same seams. Resolutions keep both worlds:
- compose + Swarm stack: lake AND daemon are sibling services; backend env
carries THERMOGRAPH_LAKE_URL and THERMOGRAPH_INTERNAL_TOKEN.
- deploy.sh: backend deploys roll (backend lake daemon); all adds frontend.
- requirements.txt: websockets/apscheduler stay removed (moved to the Go
daemon), duckdb stays (the lake role's engine).
Merged tree: backend suite green, both compose configs validate.
Adds .forgejo/workflows/shell-lint.yml (pinned shellcheck v0.11.0 + sha256, -x,
default severity, fail on any finding, not path-filtered) and drives all 26
scripts to zero findings.
Two defects shellcheck cannot see:
render-secrets.sh left DECRYPTED vault contents in /tmp whenever a sops decrypt
failed -- the caller's set -e aborted the function before either cleanup ran.
Now removed on every exit path, with `|| return 1` on both sops calls so a
decrypt failure can never write a partial /etc/thermograph.env regardless of the
caller's shell options. Explicitly not a `trap ... RETURN`: such a trap set in a
sourced function persists into the caller's shell and re-fires when the caller's
next `.`/source completes, where the function-local tmp is unset -- fatal and
silent under deploy.sh's set -u. The file now records that reasoning.
autoscale.sh ran `set -eu` without pipefail while piping docker stats into awk,
so a failed left side was swallowed and the loop autoscaled on empty input.
Promoted to pipefail with a missed sample treated as a skip; verified busybox ash
in docker:27-cli supports it.
Also: capture-fixtures.sh's `jq . || cat` ran cat after jq had consumed stdin,
silently writing truncated fixtures; deploy.sh/deploy-stack.sh `# shellcheck
source=` paths corrected for the monorepo layout.
Prod DB dumps and Forgejo (db + data volume) are now copied off-box to the
era5-thermograph S3 bucket, age-encrypted to the vault recipient so the host
age key at /etc/thermograph/age.key decrypts them for restore. Nothing plaintext
leaves the box; no local intermediate (streamed via age | rclone rcat).
ops-cron: the prod pg_dump job gains an off-box push; a new forgejo-backup job
dumps Forgejo's Postgres + tars its data volume on beta (the git host had NO
backup at all). 30-day off-box retention on both. S3 creds are Forgejo Actions
secrets (S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY), mirrored into the SOPS vault
(prod.yaml/beta.yaml) for host-side use. Contabo needs path-style addressing.
Adds deploy/backup/README.md with the DR restore runbook.
Backend runs its full hermetic suite, frontend its unit tier, inside the
just-built image -- the dev-only feature both app repos carried in their own
build.yml (deleted here in favor of the root workflow).
One root workflow set replaces the four repos' copies (deleted -- root-only
is where Forgejo reads them, and dead copies are a trap): per-domain
build-push with explicit image paths (emi/thermograph/backend|frontend; the
old github.repository-derived path collides in a monorepo), path-filtered
per-domain beta/prod/dev deploys, a domain-input reusable build check, a
single always-reporting PR gate (path-filtered required checks deadlock
auto-merge), a new infra-sync pipeline (host checkout + secrets render on
infra/** pushes), and ports of secrets-guard / ops-cron /
observability-validate to monorepo paths.
