584 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
admin_emi	880ef395de	Merge pull request 'promote: dev → main (doc accuracy pass, runner unit fix)' (#158) from dev into main	2026-08-01 18:58:24 +00:00
Emi Griffith	9f9c436424	Merge pull request 'forgejo: make the desktop runner unit restart-always, fix the vps2 config path' (#157) from feat/forgejo-runners into dev	2026-08-01 18:52:51 +00:00
Emi Griffith	fe844a3193	Merge dev into feat/forgejo-runners (dev moved while #157 was open)	2026-08-01 11:52:12 -07:00
admin_emi	83b074fa9e	Merge pull request 'docs: correct file references and the dev reachability claim' (#156) from fix/doc-accuracy into dev	2026-08-01 18:51:08 +00:00
Emi Griffith	f25466ca76	forgejo: make the desktop runner unit restart-always, fix the vps2 config path ... Both runners exist and are up; these are the two defects found while verifying that. The desktop's systemd --user unit had Restart=on-failure. That is the same distinction that took Forgejo down for 27 hours on 2026-07-29 (docker-stack.yml records it for the db service): a daemon that exits 0 is not "finished successfully", and on-failure cannot tell that from a clean shutdown. The failure mode is silent — unit inactive (dead), runner offline in the UI, and every protected-branch merge blocked on a check nothing will produce. The StartLimit directives that bound that retry loop were in [Service], where systemd accepts them without complaint and ignores them; the live unit was running the 10s default rather than the intended 300s. Moved to [Unit], which is where they are read. runner-vps2/README told you to copy config.yaml next to docker-compose.yml, but the compose file mounts ./data:/data and loads --config /data/config.yaml, so a config there is invisible to the container — the daemon starts on defaults with no --add-host, and every registry push then fails as if the credential were wrong. vps2 had a stray copy at the documented path proving the instruction had been followed.	2026-08-01 11:50:42 -07:00
Emi Griffith	af21d8e477	docs: correct file references and the dev reachability claim ... Audited the five CLAUDE.md files and all twenty-one README.md files against the tree, machine-checking every in-repo path they name and verifying the testable claims against the live hosts. The one that matters is in the root file: dev was documented as reachable on the mesh at 10.10.0.2:8137. It is not, and never was from anywhere but vps1 — infra/docker-compose.yml binds the port to 127.0.0.1, and the address answers from neither vps2 nor vps1 itself. Anyone following it gets a connection refused with nothing to explain it. The rest are stale paths, several from the reunification: * assetlinks.json moved under frontend/static/ in the subtree merge; the TWA README kept the pre-merge path in both places it names it. Following it would put the file where nothing serves it and Android app-link verification would fail silently. * push.py and notify.py now live in backend/notifications/. * INFRA.md and deploy/stack/README have never existed in this repo, in any branch. * the Caddyfile is at deploy/stack/lb/Caddyfile. * three bare relative paths that resolve for a reader but not from the directory the file sits in: units.js is the frontend's, deploy.sh is infra's, entrypoint.sh is the backend's. Also records why mesh clients must pin the ROOT_URL host and not only the image host: the registry's bearer-token realm follows ROOT_URL, so pinning git.thermograph.org alone still sends the token request out the public route, where the /v2/* matcher returns 403 and docker falls back to anonymous. That surfaces as `unauthorized: reqPackageAccess`, indistinguishable from a bad credential. Verified true and left alone: the four-domain layout, both .claude runbooks, the absence of any domain-level .forgejo directory, the pinned compose project name, the deploy contract, prod's eight stack services, beta's five prefixed ones with no db of its own, dev's five, and every documented make target.	2026-08-01 11:49:28 -07:00
Emi Griffith	955a64a18e	Merge pull request 'promote: dev → main (Forgejo on dev.jinemi.com; jinemi registry namespace)' (#152) from dev into main	2026-08-01 18:27:41 +00:00
admin_emi	62ba381d06	Merge pull request 'build-push: drop the empty ${{ }} that silenced the registry login' (#155) from fix/build-push-empty-expression into dev	2026-08-01 18:18:45 +00:00
Emi Griffith	5f7075e2cf	build-push: drop the empty ${{ }} that silenced the registry login ... Forgejo evaluates ${{ }} expressions inside a step's `run:` script, comments included. An EMPTY expression is a parse error, and the runner's response is to drop the step -- no log line, no error, no failed status. The login step simply never ran, so every subsequent `docker push` went out anonymous. The registry then answers `unauthorized: reqPackageAccess` (or, depending on the client path, `no basic auth credentials`), which reads exactly like a revoked token or a missing scope. It is neither. Both are downstream of a comment. Isolated on one branch, one variable, back to back: * empty expression present -> both legs fail, no `Login Succeeded` in the log * empty expression removed -> both legs pass, `Login Succeeded` present, sha-df409f88b3fd published for backend and frontend Nothing was wrong with the credential. The token, its scope and whether it sat at repo or organization level were all ruled out first: pushes to jinemi/thermograph/* succeed by hand from vps1 and from the runner host, with matching and mismatched usernames, and the run log shows REGISTRY_TOKEN arriving in the job environment. Do not write a bare ${{ }} in a run block, in a comment or otherwise.	2026-08-01 11:17:48 -07:00
Emi Griffith	d6553a7a05	build-push: stop a whitespace-tainted REGISTRY_TOKEN failing as "denied" ... `docker login --password-stdin` strips exactly ONE trailing newline. `echo "$T"` adds one. So a secret pasted WITH a trailing newline -- which is what copying from a terminal, or an editor that terminates files with one, produces -- arrives at the registry as "<token>\n" and gets back: Error response from daemon: Get "https://.../v2/": denied: with no further detail. That is indistinguishable from a revoked or wrong token. On 2026-08-01 it stopped every build and deploy in the estate and cost an afternoon of diagnosis on a credential that was in fact valid: the same token, tested by hand, returned 200 on /v2/ and was issued a pull,push-scoped registry token for jinemi/thermograph/backend. So: strip leading/trailing whitespace and CR/LF before the pipe, and never `echo` a credential into stdin. The token now travels through the ENVIRONMENT rather than being interpolated into the script text. `${{ }}` is substituted before bash parses the line, so a value containing a quote or a newline changes the shape of the command itself, not merely its arguments. Two diagnostics, deliberately asymmetric: * empty/unset -> HARD FAILURE naming where to set it. There is no case where proceeding helps. * wrong shape -> WARNING only (length, and whether it is outside [0-9a-f]), then attempt the login anyway. A hard assertion on token format would block every build the day Forgejo changes that format, which is a worse failure than the one being prevented. The warning is enough to turn the registry's opaque "denied:" into a diagnosis. Neither diagnostic prints the value; only its length and character class. Note the username is not a factor: tested against the live registry, all of admin_emi, emi and jinemi authenticate identically with a valid token and are each issued a push-scoped token. Forgejo's container registry authenticates on the token, not the username.	2026-08-01 10:16:32 -07:00
Emi Griffith	486a194086	forgejo: record why the runner is on vps2, rather than leaving a contradiction ... This README said, in bold, that a runner must never go on prod or beta. A runner has been registered and running on vps2 since 2026-08-01. Both statements cannot stand: an instruction the estate visibly ignores teaches readers to ignore the next one too. The original objection is kept rather than deleted, because it is correct on its merits -- docker_host: automount hands job containers the host's Docker socket, which on vps2 is root over both the prod and beta stacks. What changed is that the alternative proved worse. The desktop was the ONLY registered runner in the estate; when it dropped on 2026-07-31 nothing merged, nothing deployed and the nightly backup did not fire for 21 hours. On a repo where every branch is protected and every change is a PR, one absent runner freezes everything, and the backup hangs off the same path. So the section now records the trade and the bounds actually applied -- capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty, no thermograph network -- and states plainly that this is defence against accident, not against a hostile workflow author. Forgejo's database already stored VPS2_SSH_KEY, which is root on that box; what changed is that the material is now reachable by a job rather than only at rest. It also keeps the original advice for the case it was actually written for: if you are adding CAPACITY rather than REDUNDANCY, raise capacity or use a box that hosts nothing.	2026-08-01 09:59:14 -07:00
admin_emi	9e0128bf01	Merge pull request 'registry: move image and repo references to the Jinemi namespace' (#150) from fix/registry-namespace-admin-emi into dev	2026-08-01 16:41:22 +00:00
Emi Griffith	73fcb174ef	Merge remote-tracking branch 'origin/dev' into fix/registry-namespace-admin-emi	2026-08-01 09:38:16 -07:00
admin_emi	209cce1cb5	Merge pull request 'forgejo: make dev.jinemi.com canonical (ROOT_URL and OAuth callback)' (#151) from feat/forgejo-oauth-jinemi into dev	2026-08-01 16:33:16 +00:00
Emi Griffith	a83a0b7d32	forgejo: make dev.jinemi.com canonical (ROOT_URL and OAuth callback) ... Forgejo derives every absolute URL from its single ROOT_URL, so this is what moves clone URLs, the Google OAuth callback, webhook payload URLs and mail links onto the new domain. Both redirect URIs are registered on the Google OAuth client and were verified against Google before the flip — login is SSO-only, so a mismatch locks everyone out of the UI including out of undoing it. README documents the probe so the check is repeatable rather than assumed. git.thermograph.org stays on the same Caddy site block and stays load-bearing: the registry host baked into image names, the mesh /etc/hosts pins that make the /v2/* matcher see a mesh source IP, and the runners' registered instance URL all still use it. Renaming the registry is a separate migration; the docs now say so where someone would otherwise retire the name as dead weight. The stack has no auto-deploy, so this takes effect on a manual `docker stack deploy` from the manager.	2026-08-01 09:32:12 -07:00
Emi Griffith	6745f8db27	Merge remote-tracking branch 'origin/dev' into fix/registry-namespace-admin-emi	2026-08-01 09:31:18 -07:00
admin_emi	7fca3556e8	Merge pull request 'forgejo: serve dev.jinemi.com alongside git.thermograph.org' (#148) from feat/forgejo-jinemi-alias into dev	2026-08-01 16:28:49 +00:00
Emi Griffith	000a8a1048	Merge dev into feat/forgejo-jinemi-alias (dev moved while #148 was open)	2026-08-01 09:27:04 -07:00
Emi Griffith	df409f88b3	registry: move image and repo references to the Jinemi namespace ... Repos moved to the Jinemi org; the container packages did not follow, since Forgejo does not transfer packages with a repo. The deploy path still resolved `emi/thermograph/*` — a user_redirect to admin_emi — while build-push.yml derives its push path from ${github.repository}, now jinemi/thermograph. The next backend or frontend build would have published somewhere no deploy looks. Point the image paths at jinemi/thermograph/* (lowercase: OCI references admit no uppercase, which is why build-push.yml already pipes through tr), the clone URLs at Jinemi/thermograph, and the registry logins at admin_emi — the account that actually owns the tokens, rather than the redirect. The live tags and both ci-runner tags were copied into the Jinemi namespace first, so the switch has something to pull. thermograph-infra, thermograph-observability and the retired */app packages stay under admin_emi; they did not move.	2026-08-01 09:25:02 -07:00
Emi Griffith	3993b7552f	lb: trust the host Caddy's X-Forwarded-Proto ... Every OAuth redirect URI the backend builds is currently http://, whatever scheme the user actually arrived with. There are two Caddy hops in front of the app: the host Caddy terminates TLS and sets X-Forwarded-Proto: https, then proxies to 127.0.0.1:8137 over plain HTTP; the stack LB is the second hop. Caddy preserves an incoming X-Forwarded-* header only when the immediate peer is a trusted proxy, and trusted_proxies was never set here -- so the LB overwrote the header with the scheme of the connection it had just received, which is http. accounts/oauth.py:_redirect_uri builds the callback from x-forwarded-proto, and that URI must match what is registered in the provider's console. Measured on vps2, same request to each hop: direct to web, XFP=https -> https://thermograph.org/api/v2/... through the LB, XFP=https -> http://thermograph.org/api/v2/... through the PATCHED LB -> https://thermograph.org/api/v2/... through the PATCHED LB, no XFP-> http://thermograph.org/api/v2/... The last line is the point of using trusted_proxies rather than hardcoding `header_up X-Forwarded-Proto https`: the LB still reports the truth when nothing in front of it claims otherwise. private_ranges covers 127.0.0.1/8 and the Docker bridge ranges, which is where the host Caddy reaches this container from. The LB binds loopback only, precisely so it cannot be reached un-fronted, so the only party that can set these headers is the host Caddy -- trusting the local peer widens nothing. Applied to beta's LB too. Beta is where a provider change gets tested before it reaches thermograph.org, so it needs the same behaviour or the test is not a test.	2026-08-01 16:24:34 +00:00
Emi Griffith	75968980f9	openbao: make verify-parity --all host-aware ... --all expanded to (dev beta prod) and then tried to check all three from wherever it was run. No host carries all three: dev is on vps1, beta and prod are on vps2. So --all could not succeed anywhere, and the way it failed was misleading rather than obvious -- on vps1, prod resolves TG_BAO_APPROLE to /etc/thermograph/openbao-approle, which on THAT box holds dev's credential, so prod's check authenticated as tg-dev and took a 403 from tg-host-prod.hcl. That reads like a policy bug. It is "wrong box". --all now means "every environment that lives on THIS host". Residency is decided by the environment's own address being bound here (TG_SSH_HOST against the local interfaces), and skips are announced rather than silent. Residency is deliberately NOT decided by TG_APP_DIR existing. That was the first attempt and it was wrong: vps1 still carries a stale /opt/thermograph from before the estate split (d5357d0, #106), so prod looked resident there and got checked anyway. Verified against both live hosts. A leftover directory is exactly the kind of thing that outlives the arrangement it belonged to. Zero environments checked is now a FAILURE. Previously a run that skipped everything would exit 0, and a green run is precisely what the 7-day cutover gate counts -- "nothing was wrong" and "nothing was examined" must not look alike. The success line reports how many were checked and says plainly that a green run on one host says nothing about the other. Live-verified on both boxes: vps1 --all -> dev PASS, beta and prod skipped, exit 0 vps2 --all -> dev skipped, beta and prod checked (both currently failing on the three known stale common.yaml keys)	2026-08-01 09:18:05 -07:00
Emi Griffith	5b5f28560d	gitignore: ignore .claude/settings.local.json ... It was untracked but not ignored, so a single `git add -A` would have committed one operator's local permission allowlist — including which vault secrets they may decrypt — as shared repo config. `.claude/settings.json` stays tracked as the reviewed team config.	2026-08-01 09:16:06 -07:00
Emi Griffith	b62eb60121	infra-sync: render /etc/centralis.env from the vault ... render_centralis_secrets has existed since the OpenBao work and had no caller on any deploy path -- only verify-centralis-render.sh, a check. A renderer nothing calls is not a migration, it is a file. /etc/centralis.env has stayed hand-maintained the whole time, with the vault holding a copy nothing consumed. That gap is why turning on Google sign-in for Centralis had no safe home: the only way to set CENTRALIS_GOOGLE_CLIENT_ID was to hand-edit the live file, which is precisely what render-secrets.sh's header warns against. THE GUARD IS THE POINT. On 2026-07-24 this estate lost Centralis' token registry by writing a file missing a key the live one had. The renderer proves its own output round-trips through bash, but it cannot know about a key that exists only in the live file. So this job renders to a scratch path first, compares KEY SETS (names only, both sides sourced in `env -i` so the comparison is over what bash actually ends up with), and refuses to write -- failing loudly -- if the live file has anything the vault does not. Rendering turns from a way to lose a hand-edit into the thing that catches one. Verified the guard against a live file carrying a hand-added CENTRALIS_GOOGLE_CLIENT_ID: correctly identified as a key that would be deleted, write refused. Also verified the empty/unreadable-live-file path, which needs `|| true` to survive pipefail. It recreates the container where the other three jobs deliberately roll nothing, and the difference is real rather than an exception: Centralis' compose file interpolates /etc/centralis.env at PARSE time, so rendering without recreating changes nothing -- the silent no-op this estate keeps getting bitten by. `docker restart` would not do it either. One container, no replicas, a two-second recreate. It also skips the recreate entirely when the rendered file is byte-identical to the live one, so a no-op push does not bounce a healthy container.	2026-08-01 09:10:34 -07:00
Emi Griffith	396b80cf91	forgejo: serve dev.jinemi.com alongside git.thermograph.org ... Both hostnames go on a single Caddy site block. That is load-bearing rather than cosmetic: the /v2/* mesh-only matcher that keeps the OCI registry API off the public internet (hazard #15) is scoped per block, so a separate block for the new name would serve the same Forgejo with the registry open. git.thermograph.org stays canonical — Forgejo has one ROOT_URL and builds every absolute URL from it, so clone URLs, the OAuth callback and post-login redirects keep naming the .org host. Flipping FORGEJO_DOMAIN additionally requires the Google OAuth redirect URI, the registry host baked into image names and runner labels, and the mesh /etc/hosts pins; README documents those prerequisites. Config verified with `caddy validate` and by asserting against the adapted JSON that the /v2 403 guard matches both hostnames.	2026-08-01 09:01:59 -07:00
Emi Griffith	7b14b9062c	openbao: make the parity gate actually work, and run it nightly ... Three fixes, all on the path between "the migration looks ready" and "the migration is ready". verify-parity.sh never sourced env-topology.sh, so TG_BAO_APPROLE was unset and render-secrets-openbao.sh fell through to the bare default -- prod's credential -- for every environment. On vps2 that made `--env beta` authenticate as tg-prod and take a 403 from tg-host-prod.hcl on thermograph/data/env/beta. `--all` failed the same way, taking dev with it. deploy.sh and deploy-stack.sh always sourced it; only the verifier did not, which is the worst place for the omission: the tool whose job is to notice divergence was itself diverging from the path it verifies. It now calls thermograph_topology per environment and takes TG_SKIP_COMMON from there rather than re-deriving it. infra-sync.yml renders all three env files on every push touching infra/**, and none of its three jobs sourced env-topology.sh either. TG_SECRETS_BACKEND was therefore unset and render-secrets.sh:44 defaulted to sops. Flipping the selector would have changed deploy.sh and deploy-stack.sh but not this workflow, which would have kept re-rendering from SOPS -- silent while parity holds, and a hard failure the moment the SOPS files are retired. The one-line cutover the README describes was never sufficient on its own. ops-cron.yml gains a secrets-parity job: prod and beta from vps2, dev from vps1, nightly, failing loudly on a mismatch. README.md called this the gate for a cutover; nothing implemented it. A gate that exists only in prose gets satisfied by assertion. It grants CI no vault access -- the host renders, CI only asks it to. First measured run, immediately after the verifier fix: dev 12 keys PASS, byte-identical beta 24 keys FAIL, 3 values differ prod 32 keys FAIL, 3 values differ The same three keys on both: THERMOGRAPH_S3_SECRET_KEY, THERMOGRAPH_LAKE_S3_SECRET_KEY, THERMOGRAPH_VAPID_CONTACT. All three live in common.yaml, seeded 2026-07-31 02:30Z -- before the Contabo rotation landed. dev passes because dev never layers common. One `seed-from-sops.sh --env common` fixes beta and prod together, and the 7-day clock starts after that.	2026-08-01 08:36:35 -07:00
Emi Griffith	001e6b1365	ci: add an always-on Actions runner on vps2 ... The estate had exactly one registered runner, on the desktop. It went offline at 2026-07-31 16:31Z; for the next 21 hours no PR could satisfy a required check, no deploy could run, and the 03:00Z ops-cron -- the only backup for both application databases and for Forgejo -- did not fire. Forgejo queued that scheduled run rather than dropping it, so it completed on reconnect and nothing was lost. A longer outage would have meant real gaps. Three files claimed an "always-on Swarm-hosted runner" existed and that the estate therefore no longer depended on the desktop. It did not exist: an early revision of docker-stack.yml ran one as a Docker-in-Docker sidecar and it was removed. That claim is why a single point of failure sat unnoticed. Corrected in docker-stack.yml, forgejo/README.md and register-lan-runner.sh. The new runner is a plain restart:always container, not a Swarm service: a Swarm-scheduled runner cannot redeploy the Swarm that schedules it, so CI would be gone exactly when the cluster is what is broken. It runs from /opt/forgejo-runner rather than in place, because the checkout is reset on every prod deploy and one `git clean -fdx` there would destroy the registration. vps2 runs prod, so the socket mount is bounded rather than assumed benign: capacity 1, --cpus=2/--memory=4g on job containers, valid_volumes empty so no job can bind-mount /etc/thermograph.env, and no thermograph network joined. This defends against accident, not against a hostile workflow author -- stated plainly in the compose header rather than implied. The desktop runner stays registered as extra capacity. Nothing may assume it is up.	2026-08-01 07:54:28 -07:00
Emi Griffith	90142cebbf	Merge branch 'release' into main ... Reconciles the four commits that landed on release via the hotfix path and were never down-merged: the onboarding guide (a23c667), its follow-up fixes (18f50ad, #99), and two alerting fixes (9b0fae5 #94, 1f9a6f0 #96). The resulting tree is byte-identical to main — every one of those changes already reached main independently, so this carries no content. What it carries is ancestry: release was not an ancestor of main, which made main -> release unmergeable under any merge style and is why #132 reported mergeable=false. Conflicts were add/add on docs/onboarding/* and one comment line in observability/grafana/provisioning/alerting/rules-prod.yml. All resolved to main's side: release's copies predate the vps1/vps2 cutover and still describe beta on vps1, dev on the desktop, and Forgejo co-located with beta. BRANCHING.md requires this down-merge in the same session as the hotfix. That step was missed; this closes it.	2026-08-01 07:29:21 -07:00
admin_emi	7e68c07b27	Merge pull request 'promote: dev → main' (#139) from dev into main	2026-07-31 05:18:02 +00:00
admin_emi	4ae50ff759	Merge pull request 'secrets: make the two-host age key quorum a checked policy' (#137) from feat/age-recovery-quorum into dev	2026-07-31 05:14:22 +00:00
Emi Griffith	b2077b5294	secrets: make the two-host age key quorum a checked policy ... The copies of /etc/thermograph/age.key on vps1 and vps2 have always read as provisioning residue — a side effect of provision-secrets.sh rather than a decision. They are in fact the estate's recovery quorum: that key is the single recovery root for all five SOPS vaults AND for every off-box backup, since ops-cron.yml:142,197 pipe those dumps through `age -r` to the same recipient. On 2026-07-30 the operator's copy was shredded from the desktop before it had been copied to its replacement machine, leaving zero operator-side copies. These two were the only surviving material and the key was restored from vps2. Documenting the ordering rule that would have prevented it, and adding the check that makes the quorum verifiable rather than assumed. verify-age-quorum.sh asserts presence, permissions and hash equality across both hosts. Divergence is treated as seriously as absence: two hosts holding different keys means one renders secrets the other cannot read, and a restore silently picks the wrong one. Prints SHA-256 digests and modes only — a digest of a 32-byte random key is not reversible, so it is safe in a CI log, and the key is never read or copied anywhere. Exit status makes it usable as a cron gate. Also replaces the "wasn't re-verified for this pass" hedge in the README with what was actually measured: vps1 0440 root:deploy, vps2 0400 root:root, digests equal. The dev render does not need vps1's group bit — it runs as `agent`, which is not in group `deploy` and reaches the key through passwordless sudo, with /opt/thermograph-dev owned agent:agent. Tightening vps1 to 0400 root:root is therefore expected to be safe, and is called out as the follow-up rather than done here because it changes a live host on the deploy path. The check reports that group read as an advisory rather than a failure. It is the current provisioned state, and a check that fails on day one is a check that is ignored by day three.	2026-07-31 05:12:58 +00:00
admin_emi	b32c7b6b11	Merge pull request 'rotate Contabo S3 credentials' (#138) from fix/rotate-contabo-s3 into dev	2026-07-31 05:12:41 +00:00
Emi Griffith	11f1275998	rotate Contabo S3 credentials	2026-07-31 05:10:41 +00:00
Emi Griffith	657e27ba9f	openbao: render Centralis from the vault ... Centralis could not migrate at all. `seed-from-sops.sh --all` refused centralis.prod with `CENTRALIS_TOKENS: contains a shell metacharacter`, and render-secrets-openbao.sh had no Centralis path whatsoever — so the seeded data would have had nothing to read it. Both halves come from applying the app stack's rules to a file that does not share its shape. /etc/thermograph.env is raw unquoted KEY=value. /etc/centralis.env is consumed by exactly one thing and that thing is a shell: sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up' CENTRALIS_TOKENS is JSON, so it can never satisfy a no-metacharacters rule, and rendering it unquoted is precisely the 2026-07-24 failure: bash strips the quotes, Centralis fails closed on the malformed registry and serves a single `shared` identity, indistinguishable from the file never being written. So: - seed-from-sops.sh validates against the renderer that will consume the path. The dotenv rules still apply to common and env/*; centralis/* is exempt because it is single-quoted downstream. - render-secrets-openbao.sh gains thermograph_render_openbao_centralis, mirroring render_centralis_secrets: fetch as JSON, emit POSIX single-quoted assignments, then PROVE the file by sourcing it under `env -i` with `set -a` and comparing every value back before anything touches the destination. Mirrored rather than shared, per the existing note above thermograph_render_openbao — consolidate when the SOPS path is deleted. - render_centralis_secrets gains the same early-return backend dispatch the app path already uses, so the new function is reachable. Without it the SOPS path was the only path regardless of TG_SECRETS_BACKEND. Verified against a synthetic registry containing a JSON token map, an apostrophe, and `$(...)`/backtick/backslash/double-quote payloads: 5 keys round-trip byte-for-byte through `set -a; . file`, CENTRALIS_TOKENS parses to the right subjects, and the command substitution does not execute. The negative case is covered too — fed the unquoted 2026-07-24 shape, the round-trip check reports the value as changed (by length, never content) and refuses to write. Also corrects the claim in render-secrets-openbao.sh that the SOPS path never tripped the metacharacter hazard "because every current value happens to be alphanumeric". CENTRALIS_TOKENS is the counter-example; it never tripped those paths because it was never in them.	2026-07-30 19:55:17 -07:00
Emi Griffith	5001269b37	openbao: derive the AppRole credential path per environment ... render-secrets-openbao.sh resolved the AppRole file to a single host-wide default, /etc/thermograph/openbao-approle — which is prod's. On vps2 that made beta's render authenticate as tg-prod, and tg-host-prod.hcl denies thermograph/data/env/beta, so beta could never render from OpenBao. It surfaces as `cannot read thermograph/env/beta`: the policy working correctly against the wrong identity. env-topology.sh exists precisely because vps2 runs two environments and one host-wide marker cannot answer "which environment is this". The AppRole path is the same question and had been given the same host-wide answer. It is now derived per environment alongside host, checkout, branch, ports and DB role, and the renderer resolves THERMOGRAPH_BAO_APPROLE, then TG_BAO_APPROLE, then the conventional path — the same precedence render-secrets.sh:44 already uses for the backend selector. Only beta differs. prod and dev keep the conventional path; dev is alone on vps1 so there is no collision there. Beta parity passes when the path is supplied by hand (24 keys = 8 + 16 shared); this makes it automatic. Derivation confirmed by sourcing env-topology.sh for all three environments.	2026-07-30 19:49:28 -07:00
Emi Griffith	702bcd920c	openbao: fix bootstrap so it completes on 2.6.x ... bootstrap.sh could not run to completion on OpenBao 2.6.1: - Release asset names were wrong. `bao_<ver>_linux_amd64.tar.gz` and `bao_<ver>_SHA256SUMS` both 404; the assets are `openbao_<ver>_...` and `checksums.txt`. The tarball is now saved under its real name and the checksum grep anchored to end-of-line, because checksums.txt also lists a .sbom.json and `sha256sum -c` resolves each line by the filename inside it. - `openssl rand -base64 32` appends a newline and the static seal reads the key file raw, so the service failed with `Error configuring seal "static": unknown encoding for AES-256 key`. - The audit stanza relied on the block label being the device type. 2.6.1 requires explicit `type` and `path` with device settings under `options`. Worth noting the intermediate state: with type and path but a bare `file_path`, the server starts and silently ignores the log location, which is the worse failure given a wedged audit device stops OpenBao answering at all. - `disable_mlock` is unsupported in 2.6.1 and warned on every start. - Nothing opened port 8200 and ufw defaults to deny(incoming), so vps1 could not reach the vault. dev renders on vps1, so this would have surfaced as a timeout during cutover rather than here. bootstrap-policies.sh installed beta's credentials as `deploy:deploy`, but vps2 has no `deploy` user and both environments deploy as `agent` (env-topology.sh sets TG_SSH_TARGET=agent@ for both; deploy.yml uses one VPS2_SSH_USER). install_creds also printed a success line after a failed chown: the call site wrapped it in `|| echo`, which suppresses `set -e` for everything inside the function, so it fell through to chmod and reported an ownership it never applied. It now removes the half-written file and returns non-zero. Corrects the isolation rationale accordingly — separate credentials buy audit attribution and a policy boundary against mistakes, not deploy-user isolation. Documents the 2.6.0 root-token change (HCSEC-2026-08): `generate-root` now targets an authenticated endpoint, so recovery keys cannot mint a replacement token and revoking the last root token before a second admin identity exists is a one-way door. Also corrects the runbook's `verify-parity.sh --all`, which cannot work from a single host given the AppRole CIDR bindings.	2026-07-30 19:47:22 -07:00
emi	dbc79351a2	Merge pull request 'promote: dev → main' (#131) from dev into main	2026-07-30 18:19:26 +00:00
emi	2776a6cdb2	Merge pull request 'openbao: add a dormant OpenBao backend alongside SOPS' (#130) from openbao-backend into dev	2026-07-30 18:13:22 +00:00
Emi Griffith	1941b184e5	Merge dev into openbao branch (dev moved when #128 landed)	2026-07-30 11:09:45 -07:00
Emi Griffith	8a2c838663	forgejo: restart the db service on a clean exit, not just on failure ... forgejo_db has been at 0/1 replicas since 2026-07-29 01:24 UTC. Postgres hit an invalid data-directory lock file ("could not open file postmaster.pid ... performing immediate shutdown because data directory lock file is invalid") and exited 0. With restart_policy.condition=on-failure, Swarm read the zero status as successful completion, marked the task Complete, and never rescheduled it. The forgejo service itself stayed Up and kept serving its homepage, so the outage presented as every repository page, the whole API and all CI returning 500 with "dial tcp: lookup db on 127.0.0.11:53: no such host" — including the auth path, which is why API calls reported "user does not exist [uid: 0]" rather than a database error. on-failure cannot distinguish "finished successfully" from "shut itself down and should be restarted", and Postgres exits 0 on several such paths, so it is the wrong policy for an always-on stateful service. This is the durable fix; it does not restart the currently stopped task.	2026-07-30 06:30:30 -07:00
Emi Griffith	7878763b66	forgejo: restart the db service on a clean exit, not just on failure ... forgejo_db has been at 0/1 replicas since 2026-07-29 01:24 UTC. Postgres hit an invalid data-directory lock file ("could not open file postmaster.pid ... performing immediate shutdown because data directory lock file is invalid") and exited 0. With restart_policy.condition=on-failure, Swarm read the zero status as successful completion, marked the task Complete, and never rescheduled it. The forgejo service itself stayed Up and kept serving its homepage, so the outage presented as every repository page, the whole API and all CI returning 500 with "dial tcp: lookup db on 127.0.0.11:53: no such host" — including the auth path, which is why API calls reported "user does not exist [uid: 0]" rather than a database error. on-failure cannot distinguish "finished successfully" from "shut itself down and should be restarted", and Postgres exits 0 on several such paths, so it is the wrong policy for an always-on stateful service. This is the durable fix; it does not restart the currently stopped task.	2026-07-29 21:42:01 -07:00
Emi Griffith	6bd07d33cb	openbao: add a dormant OpenBao backend alongside SOPS ... Phase 1 of moving the secret store to OpenBao. Nothing is cut over: TG_SECRETS_BACKEND defaults to sops for all three environments, so SOPS remains authoritative until an environment is flipped in a reviewed PR. The reason for the migration is one specific failure class. infra/CLAUDE.md requires that vps1 never hold prod credentials, but that rule is currently enforced by a caller-side shell flag (THERMOGRAPH_SECRETS_SKIP_COMMON) at three call sites. A fourth call site, or one by-hand render, writes both S3 keypairs, the VAPID private key, REGISTRY_TOKEN and the IndexNow/metrics tokens onto the Forgejo CI-runner box and exits 0. policies/tg-host-dev.hcl makes that a 403 instead: dev's identity cannot read the common path at all. Shape: - vps2 only, native systemd (not a container: a Swarm service is circular, and a plain docker run is one `prune -a` from gone), raft storage (2.6.0 deprecated the file backend), static auto-unseal so a reboot cannot block deploys, mesh-only self-signed TLS (not ACME — that would put DNS in the boot chain of the secret store). - AppRole per environment, CIDR-bound, 5-minute tokens. Each environment's secret-id is owned by its own deploy user, which is a boundary that does not exist today since prod and beta both reach the same root-owned age key. - Render still produces the same raw unquoted dotenv: Centralis compares /etc/thermograph.env textually and the Swarm entrypoint parses it by line. The backend dispatch in render-secrets.sh returns early rather than refactoring the shared write path, so the SOPS path stays byte-identical. Verified by rendering dev (12 keys) and prod (32 keys) through it, matching the live hosts. The duplicated write path is noted for consolidation once prod has been on OpenBao for a release cycle. The OpenBao renderer rejects values containing a newline or a shell metacharacter. /etc/thermograph.env is sourced by bash in six places, so such a value is a command-execution path on the deploy host; the SOPS path has always had this hazard and avoids it only because every current value happens to be alphanumeric. Also records that the age keypair is NOT retired by this migration: it is the backup encryption key for every off-box dump (ops-cron.yml:142,197, 30-day retention), so deleting it with the vault files would silently destroy a month of database recoverability. verify-parity.sh is the cutover gate — it renders both backends and diffs them, reporting key names and counts only, never values.	2026-07-29 21:35:20 -07:00
emi	94de933cab	Merge pull request 'promote: dev → main (sign in with Google; shared OAuth engine)' (#127) from dev into main	2026-07-27 01:00:41 +00:00
emi	deb039ee24	accounts: sign in with Google, on a shared provider-agnostic OAuth engine (#122)	2026-07-27 00:56:43 +00:00
Emi Griffith	1d4defefb9	BRANCHING: the escape hatch, now that apply_to_admins binds the owner too ... Protection an admin walks through is documentation rather than a control, so apply_to_admins goes on — but that removes the direct-push route for everyone, including in an emergency, and the way back in should be written down before it is needed rather than improvised. There is no force flag; the supported route is lift the rule, act, restore it, with both halves audited. Names the failure mode that actually justifies it: a renamed status context makes the required check unsatisfiable, and with apply_to_admins on that blocks every merge into dev and main with no exemption for anyone.	2026-07-26 15:05:26 -07:00
Emi Griffith	890cb4a764	docs: the orchestrator runbook and the module map ... .claude/BRANCHING.md — how the trunk-based flow is actually run: the serialised merge queue (Forgejo has no native one, so whoever orchestrates is it), the two promotions and who decides each, and the hotfix down-merge that has to happen in the same session or it becomes a regression scheduled for the next promotion. Two things it states because getting them wrong is expensive here. CI does not gate PRs into `release` — pr-build.yml fires only for dev and main — so a required check on that branch would make every production promotion unmergeable with a 405 naming no cause; closing that gap means changing the workflow first and turning the requirement on second. And the chain cannot fast-forward: every promotion leaves a merge commit on the target the source never receives, so promotions are merge commits until someone deliberately reconciles the branches. .claude/ownership.md — which tasks may run concurrently, seamed along the four domains, plus the shared spine that has to be serialised: the workflows (CI lives only in the root .forgejo/), env-topology.sh, the deploy entry points and the CLAUDE.md files. Names the recurring collisions too — a schema change spans alembic and whatever reads the column, and if the payload shape moves it is PAYLOAD_VER and the frontend as well, which is one task and not two.	2026-07-26 12:53:33 -07:00
emi	ad4d2da120	promote: dev → main	2026-07-26 19:52:21 +00:00
emi	0617a3b067	db: raise max_connections to 200, cap work_mem at 64MB, trim async pool overflow (#124)	2026-07-26 19:48:50 +00:00
emi	beb5fa5259	bookmarks: don't fabricate a local save when the server rejects it (#121)	2026-07-26 19:17:46 +00:00
emi	2a4f516798	frontend: load bookmarks-ui.js from the Go SSR home template too (#120)	2026-07-26 19:17:46 +00:00
emi	3ec72ca75b	bookmarks: don't fabricate a local save when the server rejects it (#121)	2026-07-26 18:51:02 +00:00