21 commits

Author	SHA1	Message	Date
Emi Griffith	5f7075e2cf	build-push: drop the empty ${{ }} that silenced the registry login ... Forgejo evaluates ${{ }} expressions inside a step's `run:` script, comments included. An EMPTY expression is a parse error, and the runner's response is to drop the step -- no log line, no error, no failed status. The login step simply never ran, so every subsequent `docker push` went out anonymous. The registry then answers `unauthorized: reqPackageAccess` (or, depending on the client path, `no basic auth credentials`), which reads exactly like a revoked token or a missing scope. It is neither. Both are downstream of a comment. Isolated on one branch, one variable, back to back: * empty expression present -> both legs fail, no `Login Succeeded` in the log * empty expression removed -> both legs pass, `Login Succeeded` present, sha-df409f88b3fd published for backend and frontend Nothing was wrong with the credential. The token, its scope and whether it sat at repo or organization level were all ruled out first: pushes to jinemi/thermograph/* succeed by hand from vps1 and from the runner host, with matching and mismatched usernames, and the run log shows REGISTRY_TOKEN arriving in the job environment. Do not write a bare ${{ }} in a run block, in a comment or otherwise.	2026-08-01 11:17:48 -07:00
Emi Griffith	d6553a7a05	build-push: stop a whitespace-tainted REGISTRY_TOKEN failing as "denied" ... `docker login --password-stdin` strips exactly ONE trailing newline. `echo "$T"` adds one. So a secret pasted WITH a trailing newline -- which is what copying from a terminal, or an editor that terminates files with one, produces -- arrives at the registry as "<token>\n" and gets back: Error response from daemon: Get "https://.../v2/": denied: with no further detail. That is indistinguishable from a revoked or wrong token. On 2026-08-01 it stopped every build and deploy in the estate and cost an afternoon of diagnosis on a credential that was in fact valid: the same token, tested by hand, returned 200 on /v2/ and was issued a pull,push-scoped registry token for jinemi/thermograph/backend. So: strip leading/trailing whitespace and CR/LF before the pipe, and never `echo` a credential into stdin. The token now travels through the ENVIRONMENT rather than being interpolated into the script text. `${{ }}` is substituted before bash parses the line, so a value containing a quote or a newline changes the shape of the command itself, not merely its arguments. Two diagnostics, deliberately asymmetric: * empty/unset -> HARD FAILURE naming where to set it. There is no case where proceeding helps. * wrong shape -> WARNING only (length, and whether it is outside [0-9a-f]), then attempt the login anyway. A hard assertion on token format would block every build the day Forgejo changes that format, which is a worse failure than the one being prevented. The warning is enough to turn the registry's opaque "denied:" into a diagnosis. Neither diagnostic prints the value; only its length and character class. Note the username is not a factor: tested against the live registry, all of admin_emi, emi and jinemi authenticate identically with a valid token and are each issued a push-scoped token. Forgejo's container registry authenticates on the token, not the username.	2026-08-01 10:16:32 -07:00
Emi Griffith	df409f88b3	registry: move image and repo references to the Jinemi namespace ... Repos moved to the Jinemi org; the container packages did not follow, since Forgejo does not transfer packages with a repo. The deploy path still resolved `emi/thermograph/*` — a user_redirect to admin_emi — while build-push.yml derives its push path from ${github.repository}, now jinemi/thermograph. The next backend or frontend build would have published somewhere no deploy looks. Point the image paths at jinemi/thermograph/* (lowercase: OCI references admit no uppercase, which is why build-push.yml already pipes through tr), the clone URLs at Jinemi/thermograph, and the registry logins at admin_emi — the account that actually owns the tokens, rather than the redirect. The live tags and both ci-runner tags were copied into the Jinemi namespace first, so the switch has something to pull. thermograph-infra, thermograph-observability and the retired */app packages stay under admin_emi; they did not move.	2026-08-01 09:25:02 -07:00
Emi Griffith	b62eb60121	infra-sync: render /etc/centralis.env from the vault ... render_centralis_secrets has existed since the OpenBao work and had no caller on any deploy path -- only verify-centralis-render.sh, a check. A renderer nothing calls is not a migration, it is a file. /etc/centralis.env has stayed hand-maintained the whole time, with the vault holding a copy nothing consumed. That gap is why turning on Google sign-in for Centralis had no safe home: the only way to set CENTRALIS_GOOGLE_CLIENT_ID was to hand-edit the live file, which is precisely what render-secrets.sh's header warns against. THE GUARD IS THE POINT. On 2026-07-24 this estate lost Centralis' token registry by writing a file missing a key the live one had. The renderer proves its own output round-trips through bash, but it cannot know about a key that exists only in the live file. So this job renders to a scratch path first, compares KEY SETS (names only, both sides sourced in `env -i` so the comparison is over what bash actually ends up with), and refuses to write -- failing loudly -- if the live file has anything the vault does not. Rendering turns from a way to lose a hand-edit into the thing that catches one. Verified the guard against a live file carrying a hand-added CENTRALIS_GOOGLE_CLIENT_ID: correctly identified as a key that would be deleted, write refused. Also verified the empty/unreadable-live-file path, which needs `|| true` to survive pipefail. It recreates the container where the other three jobs deliberately roll nothing, and the difference is real rather than an exception: Centralis' compose file interpolates /etc/centralis.env at PARSE time, so rendering without recreating changes nothing -- the silent no-op this estate keeps getting bitten by. `docker restart` would not do it either. One container, no replicas, a two-second recreate. It also skips the recreate entirely when the rendered file is byte-identical to the live one, so a no-op push does not bounce a healthy container.	2026-08-01 09:10:34 -07:00
Emi Griffith	7b14b9062c	openbao: make the parity gate actually work, and run it nightly ... Three fixes, all on the path between "the migration looks ready" and "the migration is ready". verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and render-secrets-openbao.sh fell through to the bare default -- prod's credential -- for every environment. On vps2 that made `--env beta` authenticate as tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta. `--all` failed the same way, taking dev with it. deploy.sh and deploy-stack.sh always sourced it; only the verifier did not, which is the worst place for the omission: the tool whose job is to notice divergence was itself diverging from the path it verifies. It now calls thermograph_topology per environment and takes TG_SKIP_COMMON from there rather than re-deriving it. infra-sync.yml renders all three env files on every push touching infra/**, and none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the selector would have changed deploy.sh and deploy-stack.sh but not this workflow, which would have kept re-rendering from SOPS -- silent while parity holds, and a hard failure the moment the SOPS files are retired. The one-line cutover the README describes was never sufficient on its own. ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from vps1, nightly, failing loudly on a mismatch. README.md called this the gate for a cutover; nothing implemented it. A gate that exists only in prose gets satisfied by assertion. It grants CI no vault access -- the host renders, CI only asks it to. First measured run, immediately after the verifier fix: dev 12 keys PASS, byte-identical beta 24 keys FAIL, 3 values differ prod 32 keys FAIL, 3 values differ The same three keys on both: THERMOGRAPH_S3_SECRET_KEY, THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed. dev passes because dev never layers common. One `seed-from-sops.sh --env common` fixes beta and prod together, and the 7-day clock starts after that.	2026-08-01 08:36:35 -07:00
Emi Griffith	001e6b1365	ci: add an always-on Actions runner on vps2 ... The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up.	2026-08-01 07:54:28 -07:00
emi	7583258509	infra-sync: refuse to render dev's vault onto a host that is not dev (#107)	2026-07-26 07:05:09 +00:00
emi	d138f00a20	infra: split the estate into vps1/vps2 — beta joins prod, dev gets a home (#103)	2026-07-26 06:56:38 +00:00
emi	d42a57a011	ci: collapse the eight deploy and build-push workflows into two (#87)	2026-07-25 07:48:49 +00:00
emi	a4ecb51401	web/worker: add a process-level liveness heartbeat (#80)	2026-07-25 04:13:47 +00:00
Emi Griffith	e4693dce58	observability: add the estate's first alerting; supervise Postfix ... The estate had zero alert rules. The only contact point was Grafana's factory default pointing at the literal string <example@email.com>, and beta's untracked compose override routed Grafana's SMTP at prod's Postfix — so alerts, had any existed, would have been delivered by the box most likely to be on fire. Prod served 86 5xx in 24h including five /healthz failures and nobody was told. Alerting (deploys to beta, which is where Grafana runs): - 12 Loki-based rules. There is no Prometheus in this estate and Loki is the only datasource, so every rule is log-derived. - Thresholds come from a 24h backtest that happens to contain a real ~20min prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so one bad request is 1.4% and a ratio alert would scream all night. The 5xx burst rule fires on the outage's 45 and 22 buckets and on nothing else in the day; the largest benign bucket all day was 5. - Routed to a new private #ops-alerts channel, not to any existing channel — #weather-events, #announcements and #prod are product surfaces that notify real subscribers. - AlertingWatchdog is a dead-man's switch; its value is its absence. - Delivery is proven, not assumed: the rules were provisioned into a throwaway Grafana against beta's live Loki and a real alert arrived in Discord. This matters because GET /api/v1/provisioning/contact-points returns [REDACTED] for the URL — a contact point holding an uninterpolated env var looks perfectly healthy and pages nobody. The only proof is a message arriving. CI gains a structural check, because the existing one only proves YAML parses: an alert rule whose condition names a missing refId is valid YAML, provisions cleanly, and never fires. It also hard-fails on a literal Discord webhook in the repo. Verified against all three breakages deliberately introduced. Postfix supervision: - The 13h outage was a boot-ordering race, not a Docker renumbering: postfix started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing docker_gwbridge address, and dockerd did not finish starting until 08:10:16. Stock postfix@.service is ordered only After=network-online.target and ships no Restart=, so one lost race became a permanent outage. - An ExecStartPre gate now blocks up to 60s until every inet_interfaces address actually exists, which absorbs the transient case inside a single start attempt. That makes bounded retry correct: 5 attempts in 600s, then failed — a genuinely broken config reaches a visible failed state in ~100s instead of re-fataling every 15s forever. - A 5-minute watchdog timer retries indefinitely and runs reset-failed, so "failed" still self-heals. Worst case is ~5 minutes, not 13 hours. - Wants=, not Requires=: a dockerd failure must not take down the loopback and mesh listeners that do not depend on Docker at all. Health checks must read config with `postconf -c`, never postmulti/postqueue/ postfix — those three RESOLVE inet_interfaces and so fatal precisely when an address is missing, which made the first version of this check report status=ok bound=0/0. A health check that fails open is worse than none. DEPLOY.md carries the monitoring contract, including that systemctl is-active postfix is a known-false signal: postfix.service is a wrapper whose ExecStart=/bin/true, so it reports active forever while the real postfix@- instance is failed with zero listeners. Reproduced live. Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and app JSONL, not journald — so no Postfix line reaches Loki and the mail rules cannot fire until loki.source.journal is added. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY	2026-07-24 13:19:19 -07:00
Emi Griffith	e91b49a000	Merge remote-tracking branch 'origin/dev' into feature/storage-backups-merged ... # Conflicts: # infra/deploy/secrets/beta.yaml # infra/deploy/secrets/prod.yaml	2026-07-24 13:07:10 -07:00
emi	0862395dda	Log hygiene: Alloy CPU, Loki chunks/limits, Caddy field-stripping (#36)	2026-07-24 04:37:41 +00:00
Emi Griffith	30b0c70bbb	Reconcile: merge main (shellcheck guard, Go daemon) into dev ... main took two direct PRs (#19 shell-lint, #21 the thermograph-daemon Go service) while dev accumulated the ERA5 lake stack; both sides added compose/ stack services and touched the same seams. Resolutions keep both worlds: - compose + Swarm stack: lake AND daemon are sibling services; backend env carries THERMOGRAPH_LAKE_URL and THERMOGRAPH_INTERNAL_TOKEN. - deploy.sh: backend deploys roll (backend lake daemon); all adds frontend. - requirements.txt: websockets/apscheduler stay removed (moved to the Go daemon), duckdb stays (the lake role's engine). Merged tree: backend suite green, both compose configs validate.	2026-07-23 17:06:52 -07:00
emi	8e09155aaf	Run the ERA5 lake service in every environment (#20)	2026-07-23 22:32:13 +00:00
emi	de8e847f9f	shell: add shellcheck CI guard and drive the tree to zero findings (#19) ... Adds .forgejo/workflows/shell-lint.yml (pinned shellcheck v0.11.0 + sha256, -x, default severity, fail on any finding, not path-filtered) and drives all 26 scripts to zero findings. Two defects shellcheck cannot see: render-secrets.sh left DECRYPTED vault contents in /tmp whenever a sops decrypt failed -- the caller's set -e aborted the function before either cleanup ran. Now removed on every exit path, with `|| return 1` on both sops calls so a decrypt failure can never write a partial /etc/thermograph.env regardless of the caller's shell options. Explicitly not a `trap ... RETURN`: such a trap set in a sourced function persists into the caller's shell and re-fires when the caller's next `.`/source completes, where the function-local tmp is unset -- fatal and silent under deploy.sh's set -u. The file now records that reasoning. autoscale.sh ran `set -eu` without pipefail while piping docker stats into awk, so a failed left side was swallowed and the loop autoscaled on empty input. Promoted to pipefail with a missed sample treated as a skip; verified busybox ash in docker:27-cli supports it. Also: capture-fixtures.sh's `jq . || cat` ran cat after jq had consumed stdin, silently writing truncated fixtures; deploy.sh/deploy-stack.sh `# shellcheck source=` paths corrected for the monorepo layout.	2026-07-23 22:26:05 +00:00
Emi Griffith	6086b40fe6	Add off-box encrypted backups to Contabo Object Storage ... Prod DB dumps and Forgejo (db + data volume) are now copied off-box to the era5-thermograph S3 bucket, age-encrypted to the vault recipient so the host age key at /etc/thermograph/age.key decrypts them for restore. Nothing plaintext leaves the box; no local intermediate (streamed via age | rclone rcat). ops-cron: the prod pg_dump job gains an off-box push; a new forgejo-backup job dumps Forgejo's Postgres + tars its data volume on beta (the git host had NO backup at all). 30-day off-box retention on both. S3 creds are Forgejo Actions secrets (S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY), mirrored into the SOPS vault (prod.yaml/beta.yaml) for host-side use. Contabo needs path-style addressing. Adds deploy/backup/README.md with the DR restore runbook.	2026-07-23 13:42:09 -07:00
emi	af839c6cd0	Finish the domain-sha tag keying: three deploy workflows were missed (#6)	2026-07-23 13:58:57 +00:00
emi	0b46844cf5	Key image tags to the last domain-touching commit, not the branch tip (#3)	2026-07-23 13:42:28 +00:00
Emi Griffith	a9ceb8f03f	CI: port the dev-branch in-image test step into the reusable build check ... Backend runs its full hermetic suite, frontend its unit tier, inside the just-built image -- the dev-only feature both app repos carried in their own build.yml (deleted here in favor of the root workflow).	2026-07-22 22:24:31 -07:00
Emi Griffith	7b2db07722	CI: port the split repos' workflows to per-domain path-filtered monorepo pipelines ... One root workflow set replaces the four repos' copies (deleted -- root-only is where Forgejo reads them, and dead copies are a trap): per-domain build-push with explicit image paths (emi/thermograph/backend|frontend; the old github.repository-derived path collides in a monorepo), path-filtered per-domain beta/prod/dev deploys, a domain-input reusable build check, a single always-reporting PR gate (path-filtered required checks deadlock auto-merge), a new infra-sync pipeline (host checkout + secrets render on infra/** pushes), and ports of secrets-guard / ops-cron / observability-validate to monorepo paths.	2026-07-22 22:11:33 -07:00