Commit graph

42 commits

Author SHA1 Message Date
Emi Griffith
657e27ba9f openbao: render Centralis from the vault
All checks were successful
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 5s
secrets-guard / encrypted (push) Successful in 4s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
Sync infra to hosts / sync-prod (push) Has been skipped
shell-lint / shellcheck (push) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / changes (pull_request) Successful in 7s
shell-lint / shellcheck (pull_request) Successful in 7s
Centralis could not migrate at all. `seed-from-sops.sh --all` refused
centralis.prod with `CENTRALIS_TOKENS: contains a shell metacharacter`, and
render-secrets-openbao.sh had no Centralis path whatsoever — so the seeded data
would have had nothing to read it.

Both halves come from applying the app stack's rules to a file that does not share
its shape. /etc/thermograph.env is raw unquoted KEY=value. /etc/centralis.env is
consumed by exactly one thing and that thing is a shell:

    sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up'

CENTRALIS_TOKENS is JSON, so it can never satisfy a no-metacharacters rule, and
rendering it unquoted is precisely the 2026-07-24 failure: bash strips the quotes,
Centralis fails closed on the malformed registry and serves a single `shared`
identity, indistinguishable from the file never being written.

So:

- seed-from-sops.sh validates against the renderer that will consume the path. The
  dotenv rules still apply to common and env/*; centralis/* is exempt because it is
  single-quoted downstream.
- render-secrets-openbao.sh gains thermograph_render_openbao_centralis, mirroring
  render_centralis_secrets: fetch as JSON, emit POSIX single-quoted assignments,
  then PROVE the file by sourcing it under `env -i` with `set -a` and comparing
  every value back before anything touches the destination. Mirrored rather than
  shared, per the existing note above thermograph_render_openbao — consolidate when
  the SOPS path is deleted.
- render_centralis_secrets gains the same early-return backend dispatch the app
  path already uses, so the new function is reachable. Without it the SOPS path was
  the only path regardless of TG_SECRETS_BACKEND.

Verified against a synthetic registry containing a JSON token map, an apostrophe,
and `$(...)`/backtick/backslash/double-quote payloads: 5 keys round-trip
byte-for-byte through `set -a; . file`, CENTRALIS_TOKENS parses to the right
subjects, and the command substitution does not execute. The negative case is
covered too — fed the unquoted 2026-07-24 shape, the round-trip check reports the
value as changed (by length, never content) and refuses to write.

Also corrects the claim in render-secrets-openbao.sh that the SOPS path never
tripped the metacharacter hazard "because every current value happens to be
alphanumeric". CENTRALIS_TOKENS is the counter-example; it never tripped those
paths because it was never in them.
2026-07-30 19:55:17 -07:00
Emi Griffith
5001269b37 openbao: derive the AppRole credential path per environment
All checks were successful
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / changes (pull_request) Successful in 7s
PR build (required check) / gate (pull_request) Successful in 1s
secrets-guard / encrypted (pull_request) Successful in 9s
shell-lint / shellcheck (pull_request) Successful in 8s
render-secrets-openbao.sh resolved the AppRole file to a single host-wide default,
/etc/thermograph/openbao-approle — which is prod's. On vps2 that made beta's render
authenticate as tg-prod, and tg-host-prod.hcl denies thermograph/data/env/beta, so
beta could never render from OpenBao. It surfaces as `cannot read
thermograph/env/beta`: the policy working correctly against the wrong identity.

env-topology.sh exists precisely because vps2 runs two environments and one
host-wide marker cannot answer "which environment is this". The AppRole path is the
same question and had been given the same host-wide answer. It is now derived per
environment alongside host, checkout, branch, ports and DB role, and the renderer
resolves THERMOGRAPH_BAO_APPROLE, then TG_BAO_APPROLE, then the conventional path —
the same precedence render-secrets.sh:44 already uses for the backend selector.

Only beta differs. prod and dev keep the conventional path; dev is alone on vps1 so
there is no collision there.

Beta parity passes when the path is supplied by hand (24 keys = 8 + 16 shared);
this makes it automatic. Derivation confirmed by sourcing env-topology.sh for all
three environments.
2026-07-30 19:49:28 -07:00
Emi Griffith
702bcd920c openbao: fix bootstrap so it completes on 2.6.x
All checks were successful
PR build (required check) / changes (pull_request) Successful in 7s
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / gate (pull_request) Successful in 1s
PR build (required check) / build-backend (pull_request) Has been skipped
secrets-guard / encrypted (pull_request) Successful in 4s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
bootstrap.sh could not run to completion on OpenBao 2.6.1:

- Release asset names were wrong. `bao_<ver>_linux_amd64.tar.gz` and
  `bao_<ver>_SHA256SUMS` both 404; the assets are `openbao_<ver>_...` and
  `checksums.txt`. The tarball is now saved under its real name and the checksum
  grep anchored to end-of-line, because checksums.txt also lists a .sbom.json and
  `sha256sum -c` resolves each line by the filename inside it.
- `openssl rand -base64 32` appends a newline and the static seal reads the key
  file raw, so the service failed with `Error configuring seal "static": unknown
  encoding for AES-256 key`.
- The audit stanza relied on the block label being the device type. 2.6.1 requires
  explicit `type` and `path` with device settings under `options`. Worth noting the
  intermediate state: with type and path but a bare `file_path`, the server starts
  and silently ignores the log location, which is the worse failure given a wedged
  audit device stops OpenBao answering at all.
- `disable_mlock` is unsupported in 2.6.1 and warned on every start.
- Nothing opened port 8200 and ufw defaults to deny(incoming), so vps1 could not
  reach the vault. dev renders on vps1, so this would have surfaced as a timeout
  during cutover rather than here.

bootstrap-policies.sh installed beta's credentials as `deploy:deploy`, but vps2 has
no `deploy` user and both environments deploy as `agent` (env-topology.sh sets
TG_SSH_TARGET=agent@ for both; deploy.yml uses one VPS2_SSH_USER). install_creds
also printed a success line after a failed chown: the call site wrapped it in
`|| echo`, which suppresses `set -e` for everything inside the function, so it fell
through to chmod and reported an ownership it never applied. It now removes the
half-written file and returns non-zero.

Corrects the isolation rationale accordingly — separate credentials buy audit
attribution and a policy boundary against mistakes, not deploy-user isolation.

Documents the 2.6.0 root-token change (HCSEC-2026-08): `generate-root` now targets
an authenticated endpoint, so recovery keys cannot mint a replacement token and
revoking the last root token before a second admin identity exists is a one-way
door. Also corrects the runbook's `verify-parity.sh --all`, which cannot work from
a single host given the AppRole CIDR bindings.
2026-07-30 19:47:22 -07:00
Emi Griffith
7878763b66 forgejo: restart the db service on a clean exit, not just on failure
All checks were successful
PR build (required check) / changes (pull_request) Successful in 5s
PR build (required check) / validate-observability (pull_request) Has been skipped
secrets-guard / encrypted (pull_request) Successful in 4s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
forgejo_db has been at 0/1 replicas since 2026-07-29 01:24 UTC. Postgres hit
an invalid data-directory lock file ("could not open file postmaster.pid ...
performing immediate shutdown because data directory lock file is invalid")
and exited 0. With restart_policy.condition=on-failure, Swarm read the zero
status as successful completion, marked the task Complete, and never
rescheduled it.

The forgejo service itself stayed Up and kept serving its homepage, so the
outage presented as every repository page, the whole API and all CI returning
500 with "dial tcp: lookup db on 127.0.0.11:53: no such host" — including the
auth path, which is why API calls reported "user does not exist [uid: 0]"
rather than a database error.

on-failure cannot distinguish "finished successfully" from "shut itself down
and should be restarted", and Postgres exits 0 on several such paths, so it is
the wrong policy for an always-on stateful service.

This is the durable fix; it does not restart the currently stopped task.
2026-07-29 21:42:01 -07:00
Emi Griffith
6bd07d33cb openbao: add a dormant OpenBao backend alongside SOPS
Phase 1 of moving the secret store to OpenBao. Nothing is cut over:
TG_SECRETS_BACKEND defaults to sops for all three environments, so SOPS
remains authoritative until an environment is flipped in a reviewed PR.

The reason for the migration is one specific failure class. infra/CLAUDE.md
requires that vps1 never hold prod credentials, but that rule is currently
enforced by a caller-side shell flag (THERMOGRAPH_SECRETS_SKIP_COMMON) at
three call sites. A fourth call site, or one by-hand render, writes both S3
keypairs, the VAPID private key, REGISTRY_TOKEN and the IndexNow/metrics
tokens onto the Forgejo CI-runner box and exits 0. policies/tg-host-dev.hcl
makes that a 403 instead: dev's identity cannot read the common path at all.

Shape:
- vps2 only, native systemd (not a container: a Swarm service is circular,
  and a plain docker run is one `prune -a` from gone), raft storage (2.6.0
  deprecated the file backend), static auto-unseal so a reboot cannot block
  deploys, mesh-only self-signed TLS (not ACME — that would put DNS in the
  boot chain of the secret store).
- AppRole per environment, CIDR-bound, 5-minute tokens. Each environment's
  secret-id is owned by its own deploy user, which is a boundary that does
  not exist today since prod and beta both reach the same root-owned age key.
- Render still produces the same raw unquoted dotenv: Centralis compares
  /etc/thermograph.env textually and the Swarm entrypoint parses it by line.

The backend dispatch in render-secrets.sh returns early rather than
refactoring the shared write path, so the SOPS path stays byte-identical.
Verified by rendering dev (12 keys) and prod (32 keys) through it, matching
the live hosts. The duplicated write path is noted for consolidation once
prod has been on OpenBao for a release cycle.

The OpenBao renderer rejects values containing a newline or a shell
metacharacter. /etc/thermograph.env is sourced by bash in six places, so
such a value is a command-execution path on the deploy host; the SOPS path
has always had this hazard and avoids it only because every current value
happens to be alphanumeric.

Also records that the age keypair is NOT retired by this migration: it is
the backup encryption key for every off-box dump (ops-cron.yml:142,197,
30-day retention), so deleting it with the vault files would silently
destroy a month of database recoverability.

verify-parity.sh is the cutover gate — it renders both backends and diffs
them, reporting key names and counts only, never values.
2026-07-29 21:35:20 -07:00
emi
deb039ee24 accounts: sign in with Google, on a shared provider-agnostic OAuth engine (#122)
All checks were successful
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 11s
secrets-guard / encrypted (push) Successful in 17s
shell-lint / shellcheck (push) Successful in 15s
Build + push images (Forgejo registry) / build-push (frontend) (push) Successful in 1m5s
Build + push images (Forgejo registry) / build-push (backend) (push) Successful in 1m8s
Deploy / deploy (backend) (push) Successful in 1m29s
Deploy / deploy (frontend) (push) Successful in 1m38s
secrets-guard / encrypted (pull_request) Successful in 8s
PR build (required check) / changes (pull_request) Successful in 10s
shell-lint / shellcheck (pull_request) Successful in 11s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Successful in 59s
PR build (required check) / build-backend (pull_request) Successful in 1m22s
PR build (required check) / gate (pull_request) Successful in 1s
2026-07-27 00:56:43 +00:00
emi
0617a3b067 db: raise max_connections to 200, cap work_mem at 64MB, trim async pool overflow (#124)
All checks were successful
Sync infra to hosts / sync-beta (push) Has been skipped
Build + push images (Forgejo registry) / build-push (frontend) (push) Successful in 9s
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 12s
Deploy / deploy (frontend) (push) Successful in 19s
secrets-guard / encrypted (push) Successful in 18s
shell-lint / shellcheck (push) Successful in 16s
Build + push images (Forgejo registry) / build-push (backend) (push) Successful in 55s
Deploy / deploy (backend) (push) Successful in 1m22s
PR build (required check) / changes (pull_request) Successful in 8s
secrets-guard / encrypted (pull_request) Successful in 6s
PR build (required check) / build-frontend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 10s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-backend (pull_request) Successful in 55s
PR build (required check) / gate (pull_request) Successful in 2s
2026-07-26 19:48:50 +00:00
emi
8572c4aa6d accounts: sign in with Discord, sharing the existing link callback (#115)
All checks were successful
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 11s
secrets-guard / encrypted (push) Successful in 6s
shell-lint / shellcheck (push) Successful in 7s
Build + push images (Forgejo registry) / build-push (backend) (push) Successful in 1m21s
Build + push images (Forgejo registry) / build-push (frontend) (push) Successful in 1m19s
Deploy / deploy (frontend) (push) Successful in 1m27s
Deploy / deploy (backend) (push) Successful in 1m49s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / changes (pull_request) Successful in 7s
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Successful in 1m16s
PR build (required check) / build-backend (pull_request) Successful in 1m36s
PR build (required check) / gate (pull_request) Successful in 1s
2026-07-26 18:16:55 +00:00
emi
801c767115 dev: leave /healthz outside the basic-auth boundary (#113)
All checks were successful
Sync infra to hosts / sync-beta (push) Has been skipped
shell-lint / shellcheck (push) Successful in 10s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 6s
secrets-guard / encrypted (push) Successful in 6s
PR build (required check) / changes (pull_request) Successful in 5s
secrets-guard / encrypted (pull_request) Successful in 5s
shell-lint / shellcheck (pull_request) Successful in 8s
PR build (required check) / validate-observability (pull_request) Has been skipped
2026-07-26 08:28:18 +00:00
emi
4122f4353b dev: serve at dev.thermograph.org behind basic auth, bound to loopback (#111)
Some checks failed
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Failing after 5s
secrets-guard / encrypted (push) Successful in 6s
shell-lint / shellcheck (push) Successful in 16s
PR build (required check) / changes (pull_request) Successful in 14s
secrets-guard / encrypted (pull_request) Successful in 13s
shell-lint / shellcheck (pull_request) Successful in 11s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
2026-07-26 07:49:43 +00:00
emi
64986189e1 deploy.sh: honour TG_POST_DEPLOY on the compose path too (#109)
All checks were successful
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 5s
secrets-guard / encrypted (push) Successful in 5s
shell-lint / shellcheck (push) Successful in 7s
PR build (required check) / changes (pull_request) Successful in 7s
PR build (required check) / build-frontend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 9s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
Sync infra to hosts / sync-prod (push) Has been skipped
secrets-guard / encrypted (pull_request) Successful in 6s
PR build (required check) / build-backend (pull_request) Has been skipped
2026-07-26 07:28:43 +00:00
emi
5c41ee4dad provision-env-db: revoke PUBLIC connect on the owner database too (#108)
All checks were successful
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 7s
secrets-guard / encrypted (push) Successful in 5s
shell-lint / shellcheck (push) Successful in 7s
2026-07-26 07:17:20 +00:00
emi
7583258509 infra-sync: refuse to render dev's vault onto a host that is not dev (#107)
Some checks failed
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Failing after 5s
secrets-guard / encrypted (push) Successful in 5s
shell-lint / shellcheck (push) Successful in 8s
PR build (required check) / changes (pull_request) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 7s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 11s
PR build (required check) / validate-observability (pull_request) Successful in 21s
PR build (required check) / gate (pull_request) Successful in 2s
2026-07-26 07:05:09 +00:00
emi
d138f00a20 infra: split the estate into vps1/vps2 — beta joins prod, dev gets a home (#103)
Some checks failed
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Failing after 6s
secrets-guard / encrypted (push) Successful in 6s
shell-lint / shellcheck (push) Successful in 13s
Validate observability stack / validate (push) Successful in 17s
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Successful in 18s
PR build (required check) / gate (pull_request) Successful in 2s
2026-07-26 06:56:38 +00:00
emi
d42a57a011 ci: collapse the eight deploy and build-push workflows into two (#87)
All checks were successful
secrets-guard / encrypted (push) Successful in 5s
shell-lint / shellcheck (push) Successful in 6s
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 6s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 9s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
2026-07-25 07:48:49 +00:00
emi
9b6b12a5da infra: mirror LAN dev secrets under $HOME for snap-confined Docker (#85)
All checks were successful
secrets-guard / encrypted (push) Successful in 6s
shell-lint / shellcheck (push) Successful in 7s
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 6s
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / validate-observability (pull_request) Successful in 38s
PR build (required check) / build-frontend (pull_request) Successful in 1m6s
PR build (required check) / build-backend (pull_request) Successful in 1m25s
PR build (required check) / gate (pull_request) Successful in 2s
2026-07-25 07:16:08 +00:00
emi
e84b1f7937 docs: rewrite the agent context layer to match the live system (#81)
Some checks failed
shell-lint / shellcheck (push) Has been cancelled
secrets-guard / encrypted (push) Has been cancelled
Validate observability stack / validate (push) Successful in 33s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 3m5s
Deploy frontend to LAN dev server / build (push) Successful in 3m21s
Build + push frontend image (Forgejo registry) / build-push (push) Successful in 3m27s
Deploy backend to LAN dev server / build (push) Successful in 4m16s
Deploy frontend to LAN dev server / deploy (push) Successful in 31s
Deploy backend to LAN dev server / deploy (push) Successful in 1m19s
2026-07-25 07:08:54 +00:00
emi
2c7ceefab4 infra/forgejo: give the LAN runner's PATH ~/.local/bin (#84)
All checks were successful
secrets-guard / encrypted (push) Successful in 5s
shell-lint / shellcheck (push) Successful in 6s
2026-07-25 06:59:21 +00:00
emi
a4ecb51401 web/worker: add a process-level liveness heartbeat (#80)
Some checks failed
secrets-guard / encrypted (push) Successful in 24s
shell-lint / shellcheck (push) Successful in 26s
Deploy frontend to LAN dev server / build (push) Successful in 2m11s
Build + push frontend image (Forgejo registry) / build-push (push) Successful in 2m20s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 2m28s
Deploy backend to LAN dev server / build (push) Successful in 2m45s
PR build (required check) / changes (pull_request) Successful in 16s
secrets-guard / encrypted (pull_request) Successful in 16s
shell-lint / shellcheck (pull_request) Successful in 15s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
Deploy frontend to LAN dev server / deploy (push) Successful in 39s
PR build (required check) / build-backend (pull_request) Successful in 1m21s
PR build (required check) / gate (pull_request) Successful in 3s
Deploy backend to LAN dev server / deploy (push) Failing after 2m31s
2026-07-25 04:13:47 +00:00
Emi Griffith
33cd69e44e secrets: silence two SC2016s that are the intended behaviour
All checks were successful
PR build (required check) / changes (pull_request) Successful in 8s
secrets-guard / encrypted (pull_request) Successful in 6s
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
shell-lint failed on two single-quoted strings. Both are deliberate and
double-quoting either one would break it silently rather than loudly.

In render_centralis_secrets, the verification step runs `bash -c 'set -a; . "$1"'`
with the rendered file passed as an argument. The "$1" must be expanded by that
inner shell. Double-quoted, it would interpolate the OUTER $1 — the function's
own first argument, the repo root — so the check would source the wrong path and
still report success. That check exists precisely to catch a bad render, so a
false pass there is worse than no check.

In verify-centralis-render.sh, the string is executed by the REMOTE shell over
ssh. Double-quoted, mktemp would run locally and the script would send the host
a path that does not exist there.

Suppressed with the reason at each site rather than loosened globally.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-24 18:13:08 -07:00
Emi Griffith
43c9e2052a secrets: vault dev and Centralis; render dev without common.yaml
Some checks failed
PR build (required check) / changes (pull_request) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Failing after 9s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
Brings the last two credential stores under SOPS, so all three environments and
the control plane are managed the same way.

dev.yaml — dev was not "intentionally not wired in", it was broken

The vault README claimed dev had no real secrets. Both halves were false.

Dev needed secrets it did not have: thermograph-dev-daemon-1 has been in a crash
loop, restarting every ~60s with "neither THERMOGRAPH_INTERNAL_TOKEN nor
THERMOGRAPH_AUTH_SECRET is set; refusing to start" — so dev has had no Discord
gateway and no job timers. With THERMOGRAPH_BASE_URL unset the app also falls
back to https://thermograph.org, so dev's IndexNow pings and verification links
claimed to be prod. dev.yaml gives it its own generated AUTH_SECRET and a real
base URL, and 12 values total.

And dev already held production credentials: backend-deploy-dev.yml injects
S3_ACCESS_KEY/S3_SECRET_KEY into every dev deploy, and the only provisioned
keypair for that bucket is read-write — on the bucket holding prod's backups.
Dev does not need them; without bucket creds the lake service 503s and history
falls through to the archive. Those workflow lines still need removing.

dev renders dev.yaml ALONE, via THERMOGRAPH_SECRETS_SKIP_COMMON=1. Eleven of
common.yaml's sixteen values are live production credentials, the render is a
plaintext concatenation consumed through env_file:, and dev is the operator's
desktop AND the Forgejo runner executing unreviewed dev-branch code with the
docker socket mounted. An override in dev.yaml would not help — last-wins
governs consumers, but prod's value is still physically a line in the file.
Verified: current renderer leaks 28 lines onto dev including both S3 keypairs
and the VAPID private key; patched gives exactly 12 keys and no prod credential.
Beta's render is byte-identical either way.

centralis.prod.yaml — its own file, its own renderer

/etc/centralis.env was hand-edited, which is how a JSON registry got written
unquoted into a shell-sourced file tonight and silently collapsed three
identities to one. The renderer now owns the file.

It is service- AND environment-scoped, not folded into prod.yaml, because
/etc/thermograph.env is loaded into every container in the app stack. Folding
these in would put CENTRALIS_AUTH_TOKEN and CENTRALIS_TOKENS — which
authenticate an endpoint carrying run_on_host and sql_query(write) — into the
web backend's environment, turning a read-anything bug in the app into a
foothold on the control plane.

Quoting is guaranteed three ways rather than assumed. Critically, `sops -d
--output-type dotenv` emits values verbatim, so reusing the existing render path
would have reproduced tonight's bug exactly. The new function decrypts to JSON
and emits POSIX single-quoted assignments; it then sources its own output in a
clean shell and compares every value before touching /etc; and it refuses to
write unless CENTRALIS_TOKENS parses as a non-empty subject->token object after
sourcing. Tested against 16 hostile values including embedded quotes, newlines
and ;rm -rf /.

Value-identity proven, not asserted: rendered vs live compared as *effective*
values on prod (a textual diff would report a false difference, and would report
a false match if the vault had captured quote characters as part of the value),
plus both env files producing a byte-identical `docker compose config` digest.
9 keys, both token subjects preserved. Nothing deployed.

Note for later: /etc/thermograph.env is also shell-sourced, and survives on raw
dotenv only because none of its 32 current values contains a shell-special
character. It is one quoted secret away from the same bug.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-24 18:10:51 -07:00
Emi Griffith
cb69a22c0f secrets: factor shared values into common.yaml; stop the renderer failing open
All checks were successful
PR build (required check) / changes (pull_request) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 5s
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Successful in 1m11s
PR build (required check) / build-backend (pull_request) Successful in 1m25s
PR build (required check) / gate (pull_request) Successful in 2s
Two changes to how credentials are stored and distributed.

1. common.yaml now exists.

The renderer has always concatenated common.yaml then <env>.yaml, host winning,
and the README has always documented common.yaml with a checkmark. The file was
never created. So every value shared by prod and beta was duplicated in both
vaults, free to drift apart with nothing to detect it.

16 values move to common.yaml: VAPID keypair, metrics token, IndexNow key,
REGISTRY_TOKEN, the S3 endpoint/bucket and both S3 keypairs, plus shared config.
5 stay per-host because their values genuinely differ (APP_CPUS, DB_CPUS,
DB_MEMORY, WORKERS, THERMOGRAPH_BASE_URL). 8 exist only on prod — the Discord
and mail credentials, which beta does not have at all.

Three more are held back deliberately despite being identical today:
POSTGRES_PASSWORD, THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_DATABASE_URL. These are
the credentials that let one environment act as another, and beta is the more
exposed box — it serves public Forgejo and Grafana. They match only because beta
was seeded from prod. Keeping them per-host costs one line each and preserves
the ability to diverge; putting them in common.yaml would encode the equivalence
as intentional and make breaking it a migration rather than an edit. Reasoning
recorded in the vault README, since a future reader will otherwise "fix" it.

Done without any plaintext leaving prod: ciphertext was shipped up, decrypted
against the host's age key, recombined, re-encrypted, and shipped back. The
consolidation refuses to write unless it has proved merged(common + env) equals
the original env vault exactly — same keys, same values — and re-verifies from
the written files afterwards. Both checks passed for prod and beta.

2. render_thermograph_secrets no longer fails open.

One `return 0` covered two different situations: "this host is not configured
for SOPS" (true of the LAN dev box, and correct to succeed) and "this host IS
configured but its vault is not where we looked". The second is a failure, and
returning 0 made it a silent no-op wearing a success code — a deploy would
report success having rendered nothing, and the host would keep serving whatever
/etc/thermograph.env already held, including after a rotation.

The likely cause is passing the wrong root: the function wants the directory
containing deploy/secrets, which on the hosts is /opt/thermograph/infra, not
/opt/thermograph. That mistake looked exactly like "not configured here", which
is why it went unnoticed. It now exits 1 and names the probable cause.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-24 17:40:26 -07:00
Emi Griffith
7ed375f32e key-gaps: the key regex was blind to digits, inventing missing secrets
`grep -oE "^[A-Z_]+="` cannot match a key name containing a digit. This estate
has exactly six such names — all the *_S3_* keys — so a prod audit returned 26
keys against 32 real ones, and reported the S3 and lake credentials as missing
from both live hosts.

They were present the whole time. The phantom was independently reproduced
twice by re-running the same pattern, which is what made it convincing, and it
was briefly recorded as one of two root causes of the lake being unqueryable
(#56). That issue has one cause: the missing duckdb-lake image.

An audit that under-reports is worse than no audit. A missing-secret finding
sends someone to provision a credential that already exists, and in a rotation
tool it would justify writing over one.

Fixed to `^[A-Z][A-Z0-9_]*=` in both the skill and key_gaps.py's docstring. The
match still stops at the `=`, so no value is read — that property is the reason
this grep exists rather than a parser.

Centralis's secrets_gaps carried the same bug and is fixed separately, with a
regression test.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-24 15:57:15 -07:00
Emi Griffith
e4693dce58 observability: add the estate's first alerting; supervise Postfix
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 8s
shell-lint / shellcheck (pull_request) Successful in 10s
PR build (required check) / changes (pull_request) Successful in 16s
PR build (required check) / validate-observability (pull_request) Successful in 18s
PR build (required check) / gate (pull_request) Successful in 2s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
The estate had zero alert rules. The only contact point was Grafana's factory
default pointing at the literal string <example@email.com>, and beta's
untracked compose override routed Grafana's SMTP at prod's Postfix — so
alerts, had any existed, would have been delivered by the box most likely to
be on fire. Prod served 86 5xx in 24h including five /healthz failures and
nobody was told.

Alerting (deploys to beta, which is where Grafana runs):
- 12 Loki-based rules. There is no Prometheus in this estate and Loki is the
  only datasource, so every rule is log-derived.
- Thresholds come from a 24h backtest that happens to contain a real ~20min
  prod outage at 04:20Z. Absolute counts, not ratios: prod runs ~7 req/min, so
  one bad request is 1.4% and a ratio alert would scream all night. The 5xx
  burst rule fires on the outage's 45 and 22 buckets and on nothing else in
  the day; the largest benign bucket all day was 5.
- Routed to a new private #ops-alerts channel, not to any existing channel —
  #weather-events, #announcements and #prod are product surfaces that notify
  real subscribers.
- AlertingWatchdog is a dead-man's switch; its value is its absence.
- Delivery is proven, not assumed: the rules were provisioned into a throwaway
  Grafana against beta's live Loki and a real alert arrived in Discord. This
  matters because GET /api/v1/provisioning/contact-points returns [REDACTED]
  for the URL — a contact point holding an uninterpolated env var looks
  perfectly healthy and pages nobody. The only proof is a message arriving.

CI gains a structural check, because the existing one only proves YAML parses:
an alert rule whose condition names a missing refId is valid YAML, provisions
cleanly, and never fires. It also hard-fails on a literal Discord webhook in
the repo. Verified against all three breakages deliberately introduced.

Postfix supervision:
- The 13h outage was a boot-ordering race, not a Docker renumbering: postfix
  started at 08:09:49, wg0 came up at :51, postfix fataled at :52 on a missing
  docker_gwbridge address, and dockerd did not finish starting until 08:10:16.
  Stock postfix@.service is ordered only After=network-online.target and ships
  no Restart=, so one lost race became a permanent outage.
- An ExecStartPre gate now blocks up to 60s until every inet_interfaces
  address actually exists, which absorbs the transient case inside a single
  start attempt. That makes bounded retry correct: 5 attempts in 600s, then
  failed — a genuinely broken config reaches a visible failed state in ~100s
  instead of re-fataling every 15s forever.
- A 5-minute watchdog timer retries indefinitely and runs reset-failed, so
  "failed" still self-heals. Worst case is ~5 minutes, not 13 hours.
- Wants=, not Requires=: a dockerd failure must not take down the loopback and
  mesh listeners that do not depend on Docker at all.

Health checks must read config with `postconf -c`, never postmulti/postqueue/
postfix — those three RESOLVE inet_interfaces and so fatal precisely when an
address is missing, which made the first version of this check report
status=ok bound=0/0. A health check that fails open is worse than none.

DEPLOY.md carries the monitoring contract, including that systemctl is-active
postfix is a known-false signal: postfix.service is a wrapper whose
ExecStart=/bin/true, so it reports active forever while the real postfix@-
instance is failed with zero listeners. Reproduced live.

Known gap, documented not fixed: Alloy ships Docker stdout, Caddy files and
app JSONL, not journald — so no Postfix line reaches Loki and the mail rules
cannot fire until loki.source.journal is added.

Claude-Session: https://claude.ai/code/session_0182KTMrsTHJc3TcewCatJFY
2026-07-24 13:19:19 -07:00
Emi Griffith
e91b49a000 Merge remote-tracking branch 'origin/dev' into feature/storage-backups-merged
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 7s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
# Conflicts:
#	infra/deploy/secrets/beta.yaml
#	infra/deploy/secrets/prod.yaml
2026-07-24 13:07:10 -07:00
emi
67d1734e99 ops/iceberg.sh: read-only Iceberg lake queries across all environments (#23)
All checks were successful
secrets-guard / encrypted (push) Successful in 7s
shell-lint / shellcheck (push) Successful in 7s
2026-07-24 19:34:30 +00:00
emi
864d38d772 ops/dbq.sh: read-only Postgres queries across all environments (#18)
All checks were successful
secrets-guard / encrypted (push) Successful in 16s
shell-lint / shellcheck (push) Successful in 17s
2026-07-24 19:28:58 +00:00
emi
578fb563a4 Roll the daemon with backend deploys in the Swarm stack (#40)
All checks were successful
secrets-guard / encrypted (push) Successful in 6s
shell-lint / shellcheck (push) Successful in 7s
PR build (required check) / changes (pull_request) Successful in 9s
secrets-guard / encrypted (pull_request) Successful in 8s
shell-lint / shellcheck (pull_request) Successful in 9s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Successful in 23s
PR build (required check) / build-frontend (pull_request) Successful in 57s
PR build (required check) / gate (pull_request) Successful in 2s
2026-07-24 18:59:46 +00:00
emi
0862395dda Log hygiene: Alloy CPU, Loki chunks/limits, Caddy field-stripping (#36)
All checks were successful
secrets-guard / encrypted (push) Successful in 5s
shell-lint / shellcheck (push) Successful in 7s
Validate observability stack / validate (push) Successful in 12s
2026-07-24 04:37:41 +00:00
emi
eaaa4f1ad9 Fix lake extension bake user; disable the daemon healthcheck in the stack (#30)
Some checks failed
secrets-guard / encrypted (push) Successful in 5s
shell-lint / shellcheck (push) Successful in 18s
PR build (required check) / changes (pull_request) Successful in 9s
secrets-guard / encrypted (pull_request) Successful in 10s
shell-lint / shellcheck (pull_request) Successful in 8s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 1m12s
Deploy backend to LAN dev server / build (push) Successful in 1m27s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-backend (pull_request) Successful in 1m17s
PR build (required check) / build-frontend (pull_request) Successful in 1m8s
PR build (required check) / gate (pull_request) Successful in 2s
Deploy backend to LAN dev server / deploy (push) Failing after 1m55s
2026-07-24 03:47:39 +00:00
Emi Griffith
30b0c70bbb Reconcile: merge main (shellcheck guard, Go daemon) into dev
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 6s
PR build (required check) / changes (pull_request) Successful in 8s
shell-lint / shellcheck (pull_request) Successful in 9s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Successful in 1m9s
PR build (required check) / build-backend (pull_request) Successful in 1m44s
PR build (required check) / gate (pull_request) Successful in 2s
main took two direct PRs (#19 shell-lint, #21 the thermograph-daemon Go
service) while dev accumulated the ERA5 lake stack; both sides added compose/
stack services and touched the same seams. Resolutions keep both worlds:

- compose + Swarm stack: lake AND daemon are sibling services; backend env
  carries THERMOGRAPH_LAKE_URL and THERMOGRAPH_INTERNAL_TOKEN.
- deploy.sh: backend deploys roll (backend lake daemon); all adds frontend.
- requirements.txt: websockets/apscheduler stay removed (moved to the Go
  daemon), duckdb stays (the lake role's engine).

Merged tree: backend suite green, both compose configs validate.
2026-07-23 17:06:52 -07:00
emi
16bddb6625 Iceberg conversion container for the ERA5 lake (infra/lake-iceberg) (#24)
All checks were successful
secrets-guard / encrypted (push) Successful in 16s
PR build (required check) / changes (pull_request) Successful in 8s
secrets-guard / encrypted (pull_request) Successful in 8s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Successful in 1m20s
PR build (required check) / build-backend (pull_request) Successful in 1m28s
PR build (required check) / gate (pull_request) Successful in 3s
2026-07-23 22:56:27 +00:00
emi
2d3f37c474 daemon: move the Discord gateway and scheduler out of the web process into Go (#21)
All checks were successful
Sync infra to hosts / sync-beta (push) Successful in 8s
Sync infra to hosts / sync-prod (push) Successful in 7s
secrets-guard / encrypted (push) Successful in 8s
shell-lint / shellcheck (push) Successful in 10s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 1m14s
Deploy backend to beta VPS / deploy (push) Successful in 2m4s
The gateway bot and APScheduler were long-lived stateful I/O loops running
inside the async web app under a leader election. They move into a single Go
binary that owns ONLY that I/O -- websocket, RESUME, heartbeat, backoff, timers.

It owns no grading logic. Anything needing data calls back over a new
internal-only surface (/internal/discord/grade, /internal/jobs/*). Grading
depends on polars and the parquet cache; reimplementing it in Go would let the
bot's grades drift from the API's. The grade route returns gateway-ready JSON
and Go relays the bytes verbatim.

The binary ships in the backend image and runs as a second compose service off
the same tag, so the two ends of the /internal/* contract can never skew.
deploy.sh rolls daemon alongside backend -- without that the service would never
be created, since a single-service deploy uses --no-deps. It also probes the
image first and skips the daemon when rolling a tag that predates the binary:
infra tracks main while image tags are env-staged, so a host can legitimately be
asked to roll an older backend image, and creating the service anyway would
leave a container crash-looping on a missing binary.

replicas: 1 with order: stop-first replaces the leader election -- Discord
permits one gateway connection per bot token.

THERMOGRAPH_INTERNAL_TOKEN is optional: both ends derive it from
THERMOGRAPH_AUTH_SECRET via HMAC under a domain-separation label, so this needs
no new vault entry. The derivation is pinned to a shared cross-language test
vector asserted on both sides, so drift fails CI instead of 401ing every call.
Fail closed when neither secret is set.

Improvements over the Python: a close intended for RESUME uses 4000 rather than
1000 (Discord invalidates a session closed 1000, so the old default defeated its
own resume); MESSAGE_CREATE runs on a bounded worker pool; and a malformed HELLO
returns an error rather than a clean reconnect, which would otherwise reset
backoff and hot-loop against the gateway.

365 Python tests pass; Go build/vet/test -race clean; shellcheck 0 findings.
2026-07-23 22:49:54 +00:00
emi
8e09155aaf Run the ERA5 lake service in every environment (#20)
All checks were successful
secrets-guard / encrypted (push) Successful in 6s
2026-07-23 22:32:13 +00:00
emi
de8e847f9f shell: add shellcheck CI guard and drive the tree to zero findings (#19)
All checks were successful
Build + push backend image (Forgejo registry) / build-push (push) Successful in 1m23s
Build + push frontend image (Forgejo registry) / build-push (push) Successful in 1m22s
Sync infra to hosts / sync-beta (push) Successful in 9s
Sync infra to hosts / sync-prod (push) Successful in 8s
secrets-guard / encrypted (push) Successful in 6s
shell-lint / shellcheck (push) Successful in 9s
Deploy backend to beta VPS / deploy (push) Successful in 2m4s
Deploy frontend to beta VPS / deploy (push) Successful in 1m8s
Adds .forgejo/workflows/shell-lint.yml (pinned shellcheck v0.11.0 + sha256, -x,
default severity, fail on any finding, not path-filtered) and drives all 26
scripts to zero findings.

Two defects shellcheck cannot see:

render-secrets.sh left DECRYPTED vault contents in /tmp whenever a sops decrypt
failed -- the caller's set -e aborted the function before either cleanup ran.
Now removed on every exit path, with `|| return 1` on both sops calls so a
decrypt failure can never write a partial /etc/thermograph.env regardless of the
caller's shell options. Explicitly not a `trap ... RETURN`: such a trap set in a
sourced function persists into the caller's shell and re-fires when the caller's
next `.`/source completes, where the function-local tmp is unset -- fatal and
silent under deploy.sh's set -u. The file now records that reasoning.

autoscale.sh ran `set -eu` without pipefail while piping docker stats into awk,
so a failed left side was swallowed and the loop autoscaled on empty input.
Promoted to pipefail with a missed sample treated as a skip; verified busybox ash
in docker:27-cli supports it.

Also: capture-fixtures.sh's `jq . || cat` ran cat after jq had consumed stdin,
silently writing truncated fixtures; deploy.sh/deploy-stack.sh `# shellcheck
source=` paths corrected for the monorepo layout.
2026-07-23 22:26:05 +00:00
emi
d9537798c6 Vault the ERA5 lake bucket credentials (prod) (#16)
All checks were successful
secrets-guard / encrypted (push) Successful in 7s
2026-07-23 21:27:32 +00:00
emi
370a4ffdc1 ERA5 lake: bucket-hosted history primary + SQL indexer service (#15)
All checks were successful
secrets-guard / encrypted (push) Successful in 10s
Deploy backend to LAN dev server / build (push) Successful in 1m13s
Build + push backend image (Forgejo registry) / build-push (push) Successful in 1m39s
Deploy backend to LAN dev server / deploy (push) Successful in 34s
2026-07-23 21:20:35 +00:00
Emi Griffith
6086b40fe6 Add off-box encrypted backups to Contabo Object Storage
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 7s
PR build (required check) / changes (pull_request) Successful in 8s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 3s
Prod DB dumps and Forgejo (db + data volume) are now copied off-box to the
era5-thermograph S3 bucket, age-encrypted to the vault recipient so the host
age key at /etc/thermograph/age.key decrypts them for restore. Nothing plaintext
leaves the box; no local intermediate (streamed via age | rclone rcat).

ops-cron: the prod pg_dump job gains an off-box push; a new forgejo-backup job
dumps Forgejo's Postgres + tars its data volume on beta (the git host had NO
backup at all). 30-day off-box retention on both. S3 creds are Forgejo Actions
secrets (S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY), mirrored into the SOPS vault
(prod.yaml/beta.yaml) for host-side use. Contabo needs path-style addressing.

Adds deploy/backup/README.md with the DR restore runbook.
2026-07-23 13:42:09 -07:00
Emi Griffith
0ed7e89401 stack: fix bind-mount paths for the monorepo host layout
All checks were successful
Sync infra to hosts / sync-beta (push) Successful in 6s
Sync infra to hosts / sync-prod (push) Successful in 5s
secrets-guard / encrypted (push) Successful in 8s
The cutover moved the host checkout's deploy tree to /opt/thermograph/infra/deploy/,
but the stack file's absolute bind sources (db init, env-entrypoint.sh, autoscale.sh)
still pointed at the old /opt/thermograph/deploy/ -- a service update kept the stale
mounts and new tasks failed with 'bind source path does not exist'.
2026-07-23 03:34:25 -07:00
Emi Griffith
2e753f2c6f deploy: adapt scripts + compose to the monorepo host checkout
/opt/thermograph becomes a monorepo checkout: deploy.sh gains INFRA_DIR (git
ops at the root, compose work cd'd into infra/), lock/tags/render paths move
under infra/deploy/, image-path defaults become emi/thermograph/backend|
frontend across compose, stack, and the tag-prune. docker-compose.yml pins
name: thermograph -- without it compose run from .../infra derives project
"infra" and recreates the whole stack beside the running one;
deploy-dev.sh pins COMPOSE_PROJECT_NAME=thermograph-dev (env wins over the
file key) to keep LAN dev's separate project.
2026-07-22 22:11:33 -07:00
Emi Griffith
7b2db07722 CI: port the split repos' workflows to per-domain path-filtered monorepo pipelines
One root workflow set replaces the four repos' copies (deleted -- root-only
is where Forgejo reads them, and dead copies are a trap): per-domain
build-push with explicit image paths (emi/thermograph/backend|frontend; the
old github.repository-derived path collides in a monorepo), path-filtered
per-domain beta/prod/dev deploys, a domain-input reusable build check, a
single always-reporting PR gate (path-filtered required checks deadlock
auto-merge), a new infra-sync pipeline (host checkout + secrets render on
infra/** pushes), and ports of secrets-guard / ops-cron /
observability-validate to monorepo paths.
2026-07-22 22:11:33 -07:00
Emi Griffith
ae1d9bb534 Subtree-merge thermograph-infra (origin/main) into infra/
git-subtree-dir: infra
git-subtree-mainline: d6df04eab2
git-subtree-split: 99b4b3f78d
2026-07-22 22:01:11 -07:00