56 commits

Author	SHA1	Message	Date
Emi Griffith	af21d8e477	docs: correct file references and the dev reachability claim ... Audited the five CLAUDE.md files and all twenty-one README.md files against the tree, machine-checking every in-repo path they name and verifying the testable claims against the live hosts. The one that matters is in the root file: dev was documented as reachable on the mesh at 10.10.0.2:8137. It is not, and never was from anywhere but vps1 — infra/docker-compose.yml binds the port to 127.0.0.1, and the address answers from neither vps2 nor vps1 itself. Anyone following it gets a connection refused with nothing to explain it. The rest are stale paths, several from the reunification: * assetlinks.json moved under frontend/static/ in the subtree merge; the TWA README kept the pre-merge path in both places it names it. Following it would put the file where nothing serves it and Android app-link verification would fail silently. * push.py and notify.py now live in backend/notifications/. * INFRA.md and deploy/stack/README have never existed in this repo, in any branch. * the Caddyfile is at deploy/stack/lb/Caddyfile. * three bare relative paths that resolve for a reader but not from the directory the file sits in: units.js is the frontend's, deploy.sh is infra's, entrypoint.sh is the backend's. Also records why mesh clients must pin the ROOT_URL host and not only the image host: the registry's bearer-token realm follows ROOT_URL, so pinning git.thermograph.org alone still sends the token request out the public route, where the /v2/* matcher returns 403 and docker falls back to anonymous. That surfaces as `unauthorized: reqPackageAccess`, indistinguishable from a bad credential. Verified true and left alone: the four-domain layout, both .claude runbooks, the absence of any domain-level .forgejo directory, the pinned compose project name, the deploy contract, prod's eight stack services, beta's five prefixed ones with no db of its own, dev's five, and every documented make target.	2026-08-01 11:49:28 -07:00
Emi Griffith	486a194086	forgejo: record why the runner is on vps2, rather than leaving a contradiction ... This README said, in bold, that a runner must never go on prod or beta. A runner has been registered and running on vps2 since 2026-08-01. Both statements cannot stand: an instruction the estate visibly ignores teaches readers to ignore the next one too. The original objection is kept rather than deleted, because it is correct on its merits -- docker_host: automount hands job containers the host's Docker socket, which on vps2 is root over both the prod and beta stacks. What changed is that the alternative proved worse. The desktop was the ONLY registered runner in the estate; when it dropped on 2026-07-31 nothing merged, nothing deployed and the nightly backup did not fire for 21 hours. On a repo where every branch is protected and every change is a PR, one absent runner freezes everything, and the backup hangs off the same path. So the section now records the trade and the bounds actually applied -- capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty, no thermograph network -- and states plainly that this is defence against accident, not against a hostile workflow author. Forgejo's database already stored VPS2_SSH_KEY, which is root on that box; what changed is that the material is now reachable by a job rather than only at rest. It also keeps the original advice for the case it was actually written for: if you are adding CAPACITY rather than REDUNDANCY, raise capacity or use a box that hosts nothing.	2026-08-01 09:59:14 -07:00
Emi Griffith	73fcb174ef	Merge remote-tracking branch 'origin/dev' into fix/registry-namespace-admin-emi	2026-08-01 09:38:16 -07:00
Emi Griffith	a83a0b7d32	forgejo: make dev.jinemi.com canonical (ROOT_URL and OAuth callback) ... Forgejo derives every absolute URL from its single ROOT_URL, so this is what moves clone URLs, the Google OAuth callback, webhook payload URLs and mail links onto the new domain. Both redirect URIs are registered on the Google OAuth client and were verified against Google before the flip — login is SSO-only, so a mismatch locks everyone out of the UI including out of undoing it. README documents the probe so the check is repeatable rather than assumed. git.thermograph.org stays on the same Caddy site block and stays load-bearing: the registry host baked into image names, the mesh /etc/hosts pins that make the /v2/* matcher see a mesh source IP, and the runners' registered instance URL all still use it. Renaming the registry is a separate migration; the docs now say so where someone would otherwise retire the name as dead weight. The stack has no auto-deploy, so this takes effect on a manual `docker stack deploy` from the manager.	2026-08-01 09:32:12 -07:00
Emi Griffith	6745f8db27	Merge remote-tracking branch 'origin/dev' into fix/registry-namespace-admin-emi	2026-08-01 09:31:18 -07:00
Emi Griffith	000a8a1048	Merge dev into feat/forgejo-jinemi-alias (dev moved while #148 was open)	2026-08-01 09:27:04 -07:00
Emi Griffith	df409f88b3	registry: move image and repo references to the Jinemi namespace ... Repos moved to the Jinemi org; the container packages did not follow, since Forgejo does not transfer packages with a repo. The deploy path still resolved `emi/thermograph/*` — a user_redirect to admin_emi — while build-push.yml derives its push path from ${github.repository}, now jinemi/thermograph. The next backend or frontend build would have published somewhere no deploy looks. Point the image paths at jinemi/thermograph/* (lowercase: OCI references admit no uppercase, which is why build-push.yml already pipes through tr), the clone URLs at Jinemi/thermograph, and the registry logins at admin_emi — the account that actually owns the tokens, rather than the redirect. The live tags and both ci-runner tags were copied into the Jinemi namespace first, so the switch has something to pull. thermograph-infra, thermograph-observability and the retired */app packages stay under admin_emi; they did not move.	2026-08-01 09:25:02 -07:00
Emi Griffith	3993b7552f	lb: trust the host Caddy's X-Forwarded-Proto ... Every OAuth redirect URI the backend builds is currently http://, whatever scheme the user actually arrived with. There are two Caddy hops in front of the app: the host Caddy terminates TLS and sets X-Forwarded-Proto: https, then proxies to 127.0.0.1:8137 over plain HTTP; the stack LB is the second hop. Caddy preserves an incoming X-Forwarded-* header only when the immediate peer is a trusted proxy, and trusted_proxies was never set here -- so the LB overwrote the header with the scheme of the connection it had just received, which is http. accounts/oauth.py:_redirect_uri builds the callback from x-forwarded-proto, and that URI must match what is registered in the provider's console. Measured on vps2, same request to each hop: direct to web, XFP=https -> https://thermograph.org/api/v2/... through the LB, XFP=https -> http://thermograph.org/api/v2/... through the PATCHED LB -> https://thermograph.org/api/v2/... through the PATCHED LB, no XFP-> http://thermograph.org/api/v2/... The last line is the point of using trusted_proxies rather than hardcoding `header_up X-Forwarded-Proto https`: the LB still reports the truth when nothing in front of it claims otherwise. private_ranges covers 127.0.0.1/8 and the Docker bridge ranges, which is where the host Caddy reaches this container from. The LB binds loopback only, precisely so it cannot be reached un-fronted, so the only party that can set these headers is the host Caddy -- trusting the local peer widens nothing. Applied to beta's LB too. Beta is where a provider change gets tested before it reaches thermograph.org, so it needs the same behaviour or the test is not a test.	2026-08-01 16:24:34 +00:00
Emi Griffith	75968980f9	openbao: make verify-parity --all host-aware ... --all expanded to (dev beta prod) and then tried to check all three from wherever it was run. No host carries all three: dev is on vps1, beta and prod are on vps2. So --all could not succeed anywhere, and the way it failed was misleading rather than obvious -- on vps1, prod resolves TG_BAO_APPROLE to /etc/thermograph/openbao-approle, which on THAT box holds dev's credential, so prod's check authenticated as tg-dev and took a 403 from tg-host-prod.hcl. That reads like a policy bug. It is "wrong box". --all now means "every environment that lives on THIS host". Residency is decided by the environment's own address being bound here (TG_SSH_HOST against the local interfaces), and skips are announced rather than silent. Residency is deliberately NOT decided by TG_APP_DIR existing. That was the first attempt and it was wrong: vps1 still carries a stale /opt/thermograph from before the estate split (d5357d0, #106), so prod looked resident there and got checked anyway. Verified against both live hosts. A leftover directory is exactly the kind of thing that outlives the arrangement it belonged to. Zero environments checked is now a FAILURE. Previously a run that skipped everything would exit 0, and a green run is precisely what the 7-day cutover gate counts -- "nothing was wrong" and "nothing was examined" must not look alike. The success line reports how many were checked and says plainly that a green run on one host says nothing about the other. Live-verified on both boxes: vps1 --all -> dev PASS, beta and prod skipped, exit 0 vps2 --all -> dev skipped, beta and prod checked (both currently failing on the three known stale common.yaml keys)	2026-08-01 09:18:05 -07:00
Emi Griffith	396b80cf91	forgejo: serve dev.jinemi.com alongside git.thermograph.org ... Both hostnames go on a single Caddy site block. That is load-bearing rather than cosmetic: the /v2/* mesh-only matcher that keeps the OCI registry API off the public internet (hazard #15) is scoped per block, so a separate block for the new name would serve the same Forgejo with the registry open. git.thermograph.org stays canonical — Forgejo has one ROOT_URL and builds every absolute URL from it, so clone URLs, the OAuth callback and post-login redirects keep naming the .org host. Flipping FORGEJO_DOMAIN additionally requires the Google OAuth redirect URI, the registry host baked into image names and runner labels, and the mesh /etc/hosts pins; README documents those prerequisites. Config verified with `caddy validate` and by asserting against the adapted JSON that the /v2 403 guard matches both hostnames.	2026-08-01 09:01:59 -07:00
Emi Griffith	7b14b9062c	openbao: make the parity gate actually work, and run it nightly ... Three fixes, all on the path between "the migration looks ready" and "the migration is ready". verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and render-secrets-openbao.sh fell through to the bare default -- prod's credential -- for every environment. On vps2 that made `--env beta` authenticate as tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta. `--all` failed the same way, taking dev with it. deploy.sh and deploy-stack.sh always sourced it; only the verifier did not, which is the worst place for the omission: the tool whose job is to notice divergence was itself diverging from the path it verifies. It now calls thermograph_topology per environment and takes TG_SKIP_COMMON from there rather than re-deriving it. infra-sync.yml renders all three env files on every push touching infra/**, and none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the selector would have changed deploy.sh and deploy-stack.sh but not this workflow, which would have kept re-rendering from SOPS -- silent while parity holds, and a hard failure the moment the SOPS files are retired. The one-line cutover the README describes was never sufficient on its own. ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from vps1, nightly, failing loudly on a mismatch. README.md called this the gate for a cutover; nothing implemented it. A gate that exists only in prose gets satisfied by assertion. It grants CI no vault access -- the host renders, CI only asks it to. First measured run, immediately after the verifier fix: dev 12 keys PASS, byte-identical beta 24 keys FAIL, 3 values differ prod 32 keys FAIL, 3 values differ The same three keys on both: THERMOGRAPH_S3_SECRET_KEY, THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed. dev passes because dev never layers common. One `seed-from-sops.sh --env common` fixes beta and prod together, and the 7-day clock starts after that.	2026-08-01 08:36:35 -07:00
Emi Griffith	001e6b1365	ci: add an always-on Actions runner on vps2 ... The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up.	2026-08-01 07:54:28 -07:00
Emi Griffith	b2077b5294	secrets: make the two-host age key quorum a checked policy ... The copies of /etc/thermograph/age.key on vps1 and vps2 have always read as provisioning residue — a side effect of provision-secrets.sh rather than a decision. They are in fact the estate's recovery quorum: that key is the single recovery root for all five SOPS vaults AND for every off-box backup, since ops-cron.yml:142,197 pipe those dumps through `age -r` to the same recipient. On 2026-07-30 the operator's copy was shredded from the desktop before it had been copied to its replacement machine, leaving zero operator-side copies. These two were the only surviving material and the key was restored from vps2. Documenting the ordering rule that would have prevented it, and adding the check that makes the quorum verifiable rather than assumed. verify-age-quorum.sh asserts presence, permissions and hash equality across both hosts. Divergence is treated as seriously as absence: two hosts holding different keys means one renders secrets the other cannot read, and a restore silently picks the wrong one. Prints SHA-256 digests and modes only — a digest of a 32-byte random key is not reversible, so it is safe in a CI log, and the key is never read or copied anywhere. Exit status makes it usable as a cron gate. Also replaces the "wasn't re-verified for this pass" hedge in the README with what was actually measured: vps1 0440 root:deploy, vps2 0400 root:root, digests equal. The dev render does not need vps1's group bit — it runs as `agent`, which is not in group `deploy` and reaches the key through passwordless sudo, with /opt/thermograph-dev owned agent:agent. Tightening vps1 to 0400 root:root is therefore expected to be safe, and is called out as the follow-up rather than done here because it changes a live host on the deploy path. The check reports that group read as an advisory rather than a failure. It is the current provisioned state, and a check that fails on day one is a check that is ignored by day three.	2026-07-31 05:12:58 +00:00
Emi Griffith	11f1275998	rotate Contabo S3 credentials	2026-07-31 05:10:41 +00:00
Emi Griffith	657e27ba9f	openbao: render Centralis from the vault ... Centralis could not migrate at all. `seed-from-sops.sh --all` refused centralis.prod with `CENTRALIS_TOKENS: contains a shell metacharacter`, and render-secrets-openbao.sh had no Centralis path whatsoever — so the seeded data would have had nothing to read it. Both halves come from applying the app stack's rules to a file that does not share its shape. /etc/thermograph.env is raw unquoted KEY=value. /etc/centralis.env is consumed by exactly one thing and that thing is a shell: sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up' CENTRALIS_TOKENS is JSON, so it can never satisfy a no-metacharacters rule, and rendering it unquoted is precisely the 2026-07-24 failure: bash strips the quotes, Centralis fails closed on the malformed registry and serves a single `shared` identity, indistinguishable from the file never being written. So: - seed-from-sops.sh validates against the renderer that will consume the path. The dotenv rules still apply to common and env/*; centralis/* is exempt because it is single-quoted downstream. - render-secrets-openbao.sh gains thermograph_render_openbao_centralis, mirroring render_centralis_secrets: fetch as JSON, emit POSIX single-quoted assignments, then PROVE the file by sourcing it under `env -i` with `set -a` and comparing every value back before anything touches the destination. Mirrored rather than shared, per the existing note above thermograph_render_openbao — consolidate when the SOPS path is deleted. - render_centralis_secrets gains the same early-return backend dispatch the app path already uses, so the new function is reachable. Without it the SOPS path was the only path regardless of TG_SECRETS_BACKEND. Verified against a synthetic registry containing a JSON token map, an apostrophe, and `$(...)`/backtick/backslash/double-quote payloads: 5 keys round-trip byte-for-byte through `set -a; . file`, CENTRALIS_TOKENS parses to the right subjects, and the command substitution does not execute. The negative case is covered too — fed the unquoted 2026-07-24 shape, the round-trip check reports the value as changed (by length, never content) and refuses to write. Also corrects the claim in render-secrets-openbao.sh that the SOPS path never tripped the metacharacter hazard "because every current value happens to be alphanumeric". CENTRALIS_TOKENS is the counter-example; it never tripped those paths because it was never in them.	2026-07-30 19:55:17 -07:00
Emi Griffith	5001269b37	openbao: derive the AppRole credential path per environment ... render-secrets-openbao.sh resolved the AppRole file to a single host-wide default, /etc/thermograph/openbao-approle — which is prod's. On vps2 that made beta's render authenticate as tg-prod, and tg-host-prod.hcl denies thermograph/data/env/beta, so beta could never render from OpenBao. It surfaces as `cannot read thermograph/env/beta`: the policy working correctly against the wrong identity. env-topology.sh exists precisely because vps2 runs two environments and one host-wide marker cannot answer "which environment is this". The AppRole path is the same question and had been given the same host-wide answer. It is now derived per environment alongside host, checkout, branch, ports and DB role, and the renderer resolves THERMOGRAPH_BAO_APPROLE, then TG_BAO_APPROLE, then the conventional path — the same precedence render-secrets.sh:44 already uses for the backend selector. Only beta differs. prod and dev keep the conventional path; dev is alone on vps1 so there is no collision there. Beta parity passes when the path is supplied by hand (24 keys = 8 + 16 shared); this makes it automatic. Derivation confirmed by sourcing env-topology.sh for all three environments.	2026-07-30 19:49:28 -07:00
Emi Griffith	702bcd920c	openbao: fix bootstrap so it completes on 2.6.x ... bootstrap.sh could not run to completion on OpenBao 2.6.1: - Release asset names were wrong. `bao_<ver>_linux_amd64.tar.gz` and `bao_<ver>_SHA256SUMS` both 404; the assets are `openbao_<ver>_...` and `checksums.txt`. The tarball is now saved under its real name and the checksum grep anchored to end-of-line, because checksums.txt also lists a .sbom.json and `sha256sum -c` resolves each line by the filename inside it. - `openssl rand -base64 32` appends a newline and the static seal reads the key file raw, so the service failed with `Error configuring seal "static": unknown encoding for AES-256 key`. - The audit stanza relied on the block label being the device type. 2.6.1 requires explicit `type` and `path` with device settings under `options`. Worth noting the intermediate state: with type and path but a bare `file_path`, the server starts and silently ignores the log location, which is the worse failure given a wedged audit device stops OpenBao answering at all. - `disable_mlock` is unsupported in 2.6.1 and warned on every start. - Nothing opened port 8200 and ufw defaults to deny(incoming), so vps1 could not reach the vault. dev renders on vps1, so this would have surfaced as a timeout during cutover rather than here. bootstrap-policies.sh installed beta's credentials as `deploy:deploy`, but vps2 has no `deploy` user and both environments deploy as `agent` (env-topology.sh sets TG_SSH_TARGET=agent@ for both; deploy.yml uses one VPS2_SSH_USER). install_creds also printed a success line after a failed chown: the call site wrapped it in `|| echo`, which suppresses `set -e` for everything inside the function, so it fell through to chmod and reported an ownership it never applied. It now removes the half-written file and returns non-zero. Corrects the isolation rationale accordingly — separate credentials buy audit attribution and a policy boundary against mistakes, not deploy-user isolation. Documents the 2.6.0 root-token change (HCSEC-2026-08): `generate-root` now targets an authenticated endpoint, so recovery keys cannot mint a replacement token and revoking the last root token before a second admin identity exists is a one-way door. Also corrects the runbook's `verify-parity.sh --all`, which cannot work from a single host given the AppRole CIDR bindings.	2026-07-30 19:47:22 -07:00
Emi Griffith	7878763b66	forgejo: restart the db service on a clean exit, not just on failure ... forgejo_db has been at 0/1 replicas since 2026-07-29 01:24 UTC. Postgres hit an invalid data-directory lock file ("could not open file postmaster.pid ... performing immediate shutdown because data directory lock file is invalid") and exited 0. With restart_policy.condition=on-failure, Swarm read the zero status as successful completion, marked the task Complete, and never rescheduled it. The forgejo service itself stayed Up and kept serving its homepage, so the outage presented as every repository page, the whole API and all CI returning 500 with "dial tcp: lookup db on 127.0.0.11:53: no such host" — including the auth path, which is why API calls reported "user does not exist [uid: 0]" rather than a database error. on-failure cannot distinguish "finished successfully" from "shut itself down and should be restarted", and Postgres exits 0 on several such paths, so it is the wrong policy for an always-on stateful service. This is the durable fix; it does not restart the currently stopped task.	2026-07-29 21:42:01 -07:00
Emi Griffith	6bd07d33cb	openbao: add a dormant OpenBao backend alongside SOPS ... Phase 1 of moving the secret store to OpenBao. Nothing is cut over: TG_SECRETS_BACKEND defaults to sops for all three environments, so SOPS remains authoritative until an environment is flipped in a reviewed PR. The reason for the migration is one specific failure class. infra/CLAUDE.md requires that vps1 never hold prod credentials, but that rule is currently enforced by a caller-side shell flag (THERMOGRAPH_SECRETS_SKIP_COMMON) at three call sites. A fourth call site, or one by-hand render, writes both S3 keypairs, the VAPID private key, REGISTRY_TOKEN and the IndexNow/metrics tokens onto the Forgejo CI-runner box and exits 0. policies/tg-host-dev.hcl makes that a 403 instead: dev's identity cannot read the common path at all. Shape: - vps2 only, native systemd (not a container: a Swarm service is circular, and a plain docker run is one `prune -a` from gone), raft storage (2.6.0 deprecated the file backend), static auto-unseal so a reboot cannot block deploys, mesh-only self-signed TLS (not ACME — that would put DNS in the boot chain of the secret store). - AppRole per environment, CIDR-bound, 5-minute tokens. Each environment's secret-id is owned by its own deploy user, which is a boundary that does not exist today since prod and beta both reach the same root-owned age key. - Render still produces the same raw unquoted dotenv: Centralis compares /etc/thermograph.env textually and the Swarm entrypoint parses it by line. The backend dispatch in render-secrets.sh returns early rather than refactoring the shared write path, so the SOPS path stays byte-identical. Verified by rendering dev (12 keys) and prod (32 keys) through it, matching the live hosts. The duplicated write path is noted for consolidation once prod has been on OpenBao for a release cycle. The OpenBao renderer rejects values containing a newline or a shell metacharacter. /etc/thermograph.env is sourced by bash in six places, so such a value is a command-execution path on the deploy host; the SOPS path has always had this hazard and avoids it only because every current value happens to be alphanumeric. Also records that the age keypair is NOT retired by this migration: it is the backup encryption key for every off-box dump (ops-cron.yml:142,197, 30-day retention), so deleting it with the vault files would silently destroy a month of database recoverability. verify-parity.sh is the cutover gate — it renders both backends and diffs them, reporting key names and counts only, never values.	2026-07-29 21:35:20 -07:00
emi	deb039ee24	accounts: sign in with Google, on a shared provider-agnostic OAuth engine (#122)	2026-07-27 00:56:43 +00:00
emi	0617a3b067	db: raise max_connections to 200, cap work_mem at 64MB, trim async pool overflow (#124)	2026-07-26 19:48:50 +00:00
emi	8572c4aa6d	accounts: sign in with Discord, sharing the existing link callback (#115)	2026-07-26 18:16:55 +00:00
emi	801c767115	dev: leave /healthz outside the basic-auth boundary (#113)	2026-07-26 08:28:18 +00:00
emi	4122f4353b	dev: serve at dev.thermograph.org behind basic auth, bound to loopback (#111)	2026-07-26 07:49:43 +00:00
emi	64986189e1	deploy.sh: honour TG_POST_DEPLOY on the compose path too (#109)	2026-07-26 07:28:43 +00:00
emi	5c41ee4dad	provision-env-db: revoke PUBLIC connect on the owner database too (#108)	2026-07-26 07:17:20 +00:00
emi	7583258509	infra-sync: refuse to render dev's vault onto a host that is not dev (#107)	2026-07-26 07:05:09 +00:00
emi	d138f00a20	infra: split the estate into vps1/vps2 — beta joins prod, dev gets a home (#103)	2026-07-26 06:56:38 +00:00
emi	d42a57a011	ci: collapse the eight deploy and build-push workflows into two (#87)	2026-07-25 07:48:49 +00:00
emi	9b6b12a5da	infra: mirror LAN dev secrets under $HOME for snap-confined Docker (#85)	2026-07-25 07:16:08 +00:00
emi	e84b1f7937	docs: rewrite the agent context layer to match the live system (#81)	2026-07-25 07:08:54 +00:00
emi	2c7ceefab4	infra/forgejo: give the LAN runner's PATH ~/.local/bin (#84)	2026-07-25 06:59:21 +00:00
emi	a4ecb51401	web/worker: add a process-level liveness heartbeat (#80)	2026-07-25 04:13:47 +00:00
Emi Griffith	33cd69e44e	secrets: silence two SC2016s that are the intended behaviour ... shell-lint failed on two single-quoted strings. Both are deliberate and double-quoting either one would break it silently rather than loudly. In render_centralis_secrets, the verification step runs `bash -c 'set -a; . "$1"'` with the rendered file passed as an argument. The "$1" must be expanded by that inner shell. Double-quoted, it would interpolate the OUTER $1 — the function's own first argument, the repo root — so the check would source the wrong path and still report success. That check exists precisely to catch a bad render, so a false pass there is worse than no check. In verify-centralis-render.sh, the string is executed by the REMOTE shell over ssh. Double-quoted, mktemp would run locally and the script would send the host a path that does not exist there. Suppressed with the reason at each site rather than loosened globally. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY	2026-07-24 18:13:08 -07:00
Emi Griffith	43c9e2052a	secrets: vault dev and Centralis; render dev without common.yaml ... Brings the last two credential stores under SOPS, so all three environments and the control plane are managed the same way. dev.yaml — dev was not "intentionally not wired in", it was broken The vault README claimed dev had no real secrets. Both halves were false. Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls back to https://thermograph.org, so dev's IndexNow pings and verification links claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real base URL, and 12 values total. And dev already held production credentials: backend-deploy-dev.yml injects S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned keypair for that bucket is read-write — on the bucket holding prod's backups. Dev does not need them; without bucket creds the lake service 503s and history falls through to the archive. Those workflow lines still need removing. dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of common.yaml's sixteen values are live production credentials, the render is a plaintext concatenation consumed through env_file:, and dev is the operator's desktop AND the Forgejo runner executing unreviewed dev-branch code with the docker socket mounted. An override in dev.yaml would not help — last-wins governs consumers, but prod's value is still physically a line in the file. Verified: current renderer leaks 28 lines onto dev including both S3 keypairs and the VAPID private key; patched gives exactly 12 keys and no prod credential. Beta's render is byte-identical either way. centralis.prod.yaml — its own file, its own renderer /etc/centralis.env was hand-edited, which is how a JSON registry got written unquoted into a shell-sourced file tonight and silently collapsed three identities to one. The renderer now owns the file. It is service- AND environment-scoped, not folded into prod.yaml, because /etc/thermograph.env is loaded into every container in the app stack. Folding these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which authenticate an endpoint carrying run_on_host and sql_query(write) — into the web backend's environment, turning a read-anything bug in the app into a foothold on the control plane. Quoting is guaranteed three ways rather than assumed. Critically, `sops -d --output-type dotenv` emits values verbatim, so reusing the existing render path would have reproduced tonight's bug exactly. The new function decrypts to JSON and emits POSIX single-quoted assignments; it then sources its own output in a clean shell and compares every value before touching /etc; and it refuses to write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after sourcing. Tested against 16 hostile values including embedded quotes, newlines and ;rm -rf /. Value-identity proven, not asserted: rendered vs live compared as *effective* values on prod (a textual diff would report a false difference, and would report a false match if the vault had captured quote characters as part of the value), plus both env files producing a byte-identical `docker compose config` digest. 9 keys, both token subjects preserved. Nothing deployed. Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw dotenv only because none of its 32 current values contains a shell-special character. It is one quoted secret away from the same bug. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY	2026-07-24 18:10:51 -07:00
Emi Griffith	cb69a22c0f	secrets: factor shared values into common.yaml; stop the renderer failing open ... Two changes to how credentials are stored and distributed. 1. common.yaml now exists. The renderer has always concatenated common.yaml then <env>.yaml, host winning, and the README has always documented common.yaml with a checkmark. The file was never created. So every value shared by prod and beta was duplicated in both vaults, free to drift apart with nothing to detect it. 16 values move to common.yaml: VAPID keypair, metrics token, IndexNow key, REGISTRY_TOKEN, the S3 endpoint/bucket and both S3 keypairs, plus shared config. 5 stay per-host because their values genuinely differ (APP_CPUS, DB_CPUS, DB_MEMORY, WORKERS, THERMOGRAPH_BASE_URL). 8 exist only on prod — the Discord and mail credentials, which beta does not have at all. Three more are held back deliberately despite being identical today: POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_DATABASE_URL. These are the credentials that let one environment act as another, and beta is the more exposed box — it serves public Forgejo and Grafana. They match only because beta was seeded from prod. Keeping them per-host costs one line each and preserves the ability to diverge; putting them in common.yaml would encode the equivalence as intentional and make breaking it a migration rather than an edit. Reasoning recorded in the vault README, since a future reader will otherwise "fix" it. Done without any plaintext leaving prod: ciphertext was shipped up, decrypted against the host's age key, recombined, re-encrypted, and shipped back. The consolidation refuses to write unless it has proved merged(common + env) equals the original env vault exactly — same keys, same values — and re-verifies from the written files afterwards. Both checks passed for prod and beta. 2. render_thermograph_secrets no longer fails open. One `return 0` covered two different situations: "this host is not configured for SOPS" (true of the LAN dev box, and correct to succeed) and "this host IS configured but its vault is not where we looked". The second is a failure, and returning 0 made it a silent no-op wearing a success code — a deploy would report success having rendered nothing, and the host would keep serving whatever /etc/thermograph.env already held, including after a rotation. The likely cause is passing the wrong root: the function wants the directory containing deploy/secrets, which on the hosts is /opt/thermograph/infra, not /opt/thermograph. That mistake looked exactly like "not configured here", which is why it went unnoticed. It now exits 1 and names the probable cause. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY	2026-07-24 17:40:26 -07:00
Emi Griffith	7ed375f32e	key-gaps: the key regex was blind to digits, inventing missing secrets ... `grep -oE "^[A-Z_]+="` cannot match a key name containing a digit. This estate has exactly six such names — all the *_S3_* keys — so a prod audit returned 26 keys against 32 real ones, and reported the S3 and lake credentials as missing from both live hosts. They were present the whole time. The phantom was independently reproduced twice by re-running the same pattern, which is what made it convincing, and it was briefly recorded as one of two root causes of the lake being unqueryable (#56). That issue has one cause: the missing duckdb-lake image. An audit that under-reports is worse than no audit. A missing-secret finding sends someone to provision a credential that already exists, and in a rotation tool it would justify writing over one. Fixed to `^[A-Z][A-Z0-9_]*=` in both the skill and key_gaps.py's docstring. The match still stops at the `=`, so no value is read — that property is the reason this grep exists rather than a parser. Centralis's secrets_gaps carried the same bug and is fixed separately, with a regression test. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY	2026-07-24 15:57:15 -07:00
Emi Griffith	e4693dce58	observability: add the estate's first alerting; supervise Postfix ... The estate had zero alert rules. The only contact point was Grafana's factory default pointing at the literal string <example@email.com>, and beta's untracked compose override routed Grafana's SMTP at prod's Postfix — so alerts, had any existed, would have been delivered by the box most likely to be on fire. Prod served 86 5xx in 24h including five /healthz failures and nobody was told. Alerting (deploys to beta, which is where Grafana runs): - 12 Loki-based rules. There is no Prometheus in this estate and Loki is the only datasource, so every rule is log-derived. - Thresholds come from a 24h backtest that happens to contain a real ~20min prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so one bad request is 1.4% and a ratio alert would scream all night. The 5xx burst rule fires on the outage's 45 and 22 buckets and on nothing else in the day; the largest benign bucket all day was 5. - Routed to a new private #ops-alerts channel, not to any existing channel — #weather-events, #announcements and #prod are product surfaces that notify real subscribers. - AlertingWatchdog is a dead-man's switch; its value is its absence. - Delivery is proven, not assumed: the rules were provisioned into a throwaway Grafana against beta's live Loki and a real alert arrived in Discord. This matters because GET /api/v1/provisioning/contact-points returns [REDACTED] for the URL — a contact point holding an uninterpolated env var looks perfectly healthy and pages nobody. The only proof is a message arriving. CI gains a structural check, because the existing one only proves YAML parses: an alert rule whose condition names a missing refId is valid YAML, provisions cleanly, and never fires. It also hard-fails on a literal Discord webhook in the repo. Verified against all three breakages deliberately introduced. Postfix supervision: - The 13h outage was a boot-ordering race, not a Docker renumbering: postfix started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing docker_gwbridge address, and dockerd did not finish starting until 08:10:16. Stock postfix@.service is ordered only After=network-online.target and ships no Restart=, so one lost race became a permanent outage. - An ExecStartPre gate now blocks up to 60s until every inet_interfaces address actually exists, which absorbs the transient case inside a single start attempt. That makes bounded retry correct: 5 attempts in 600s, then failed — a genuinely broken config reaches a visible failed state in ~100s instead of re-fataling every 15s forever. - A 5-minute watchdog timer retries indefinitely and runs reset-failed, so "failed" still self-heals. Worst case is ~5 minutes, not 13 hours. - Wants=, not Requires=: a dockerd failure must not take down the loopback and mesh listeners that do not depend on Docker at all. Health checks must read config with `postconf -c`, never postmulti/postqueue/ postfix — those three RESOLVE inet_interfaces and so fatal precisely when an address is missing, which made the first version of this check report status=ok bound=0/0. A health check that fails open is worse than none. DEPLOY.md carries the monitoring contract, including that systemctl is-active postfix is a known-false signal: postfix.service is a wrapper whose ExecStart=/bin/true, so it reports active forever while the real postfix@- instance is failed with zero listeners. Reproduced live. Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and app JSONL, not journald — so no Postfix line reaches Loki and the mail rules cannot fire until loki.source.journal is added. Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY	2026-07-24 13:19:19 -07:00
Emi Griffith	e91b49a000	Merge remote-tracking branch 'origin/dev' into feature/storage-backups-merged ... # Conflicts: # infra/deploy/secrets/beta.yaml # infra/deploy/secrets/prod.yaml	2026-07-24 13:07:10 -07:00
emi	67d1734e99	ops/iceberg.sh: read-only Iceberg lake queries across all environments (#23)	2026-07-24 19:34:30 +00:00
emi	864d38d772	ops/dbq.sh: read-only Postgres queries across all environments (#18)	2026-07-24 19:28:58 +00:00
emi	578fb563a4	Roll the daemon with backend deploys in the Swarm stack (#40)	2026-07-24 18:59:46 +00:00
emi	0862395dda	Log hygiene: Alloy CPU, Loki chunks/limits, Caddy field-stripping (#36)	2026-07-24 04:37:41 +00:00
emi	eaaa4f1ad9	Fix lake extension bake user; disable the daemon healthcheck in the stack (#30)	2026-07-24 03:47:39 +00:00
Emi Griffith	30b0c70bbb	Reconcile: merge main (shellcheck guard, Go daemon) into dev ... main took two direct PRs (#19 shell-lint, #21 the thermograph-daemon Go service) while dev accumulated the ERA5 lake stack; both sides added compose/ stack services and touched the same seams. Resolutions keep both worlds: - compose + Swarm stack: lake AND daemon are sibling services; backend env carries THERMOGRAPH_LAKE_URL and THERMOGRAPH_INTERNAL_TOKEN. - deploy.sh: backend deploys roll (backend lake daemon); all adds frontend. - requirements.txt: websockets/apscheduler stay removed (moved to the Go daemon), duckdb stays (the lake role's engine). Merged tree: backend suite green, both compose configs validate.	2026-07-23 17:06:52 -07:00
emi	16bddb6625	Iceberg conversion container for the ERA5 lake (infra/lake-iceberg) (#24)	2026-07-23 22:56:27 +00:00
emi	2d3f37c474	daemon: move the Discord gateway and scheduler out of the web process into Go (#21) ... The gateway bot and APScheduler were long-lived stateful I/O loops running inside the async web app under a leader election. They move into a single Go binary that owns ONLY that I/O -- websocket, RESUME, heartbeat, backoff, timers. It owns no grading logic. Anything needing data calls back over a new internal-only surface (/internal/discord/grade, /internal/jobs/*). Grading depends on polars and the parquet cache; reimplementing it in Go would let the bot's grades drift from the API's. The grade route returns gateway-ready JSON and Go relays the bytes verbatim. The binary ships in the backend image and runs as a second compose service off the same tag, so the two ends of the /internal/* contract can never skew. deploy.sh rolls daemon alongside backend -- without that the service would never be created, since a single-service deploy uses --no-deps. It also probes the image first and skips the daemon when rolling a tag that predates the binary: infra tracks main while image tags are env-staged, so a host can legitimately be asked to roll an older backend image, and creating the service anyway would leave a container crash-looping on a missing binary. replicas: 1 with order: stop-first replaces the leader election -- Discord permits one gateway connection per bot token. THERMOGRAPH_INTERNAL_TOKEN is optional: both ends derive it from THERMOGRAPH_AUTH_SECRET via HMAC under a domain-separation label, so this needs no new vault entry. The derivation is pinned to a shared cross-language test vector asserted on both sides, so drift fails CI instead of 401ing every call. Fail closed when neither secret is set. Improvements over the Python: a close intended for RESUME uses 4000 rather than 1000 (Discord invalidates a session closed 1000, so the old default defeated its own resume); MESSAGE_CREATE runs on a bounded worker pool; and a malformed HELLO returns an error rather than a clean reconnect, which would otherwise reset backoff and hot-loop against the gateway. 365 Python tests pass; Go build/vet/test -race clean; shellcheck 0 findings.	2026-07-23 22:49:54 +00:00
emi	8e09155aaf	Run the ERA5 lake service in every environment (#20)	2026-07-23 22:32:13 +00:00
emi	de8e847f9f	shell: add shellcheck CI guard and drive the tree to zero findings (#19) ... Adds .forgejo/workflows/shell-lint.yml (pinned shellcheck v0.11.0 + sha256, -x, default severity, fail on any finding, not path-filtered) and drives all 26 scripts to zero findings. Two defects shellcheck cannot see: render-secrets.sh left DECRYPTED vault contents in /tmp whenever a sops decrypt failed -- the caller's set -e aborted the function before either cleanup ran. Now removed on every exit path, with `|| return 1` on both sops calls so a decrypt failure can never write a partial /etc/thermograph.env regardless of the caller's shell options. Explicitly not a `trap ... RETURN`: such a trap set in a sourced function persists into the caller's shell and re-fires when the caller's next `.`/source completes, where the function-local tmp is unset -- fatal and silent under deploy.sh's set -u. The file now records that reasoning. autoscale.sh ran `set -eu` without pipefail while piping docker stats into awk, so a failed left side was swallowed and the loop autoscaled on empty input. Promoted to pipefail with a missed sample treated as a skip; verified busybox ash in docker:27-cli supports it. Also: capture-fixtures.sh's `jq . || cat` ran cat after jq had consumed stdin, silently writing truncated fixtures; deploy.sh/deploy-stack.sh `# shellcheck source=` paths corrected for the monorepo layout.	2026-07-23 22:26:05 +00:00
emi	d9537798c6	Vault the ERA5 lake bucket credentials (prod) (#16)	2026-07-23 21:27:32 +00:00