ba11423fd9
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 5s
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / changes (pull_request) Successful in 9s
PR build (required check) / validate-observability (pull_request) Successful in 23s
PR build (required check) / gate (pull_request) Successful in 2s
Three of the prod liveness rules were measuring streams that no longer say
anything about the thing they claim to watch.
ProdWorkerContainerSilent was unsatisfiable by construction: prod's alloy
config drops the thermograph_worker container on purpose, so the stream it
queried never reaches Loki and the rule returned zero regardless of worker
health. Firing continuously since 2026-07-24T21:05Z.
ProdWebContainerSilent measured a floor that stopped existing when the app
moved request logging off stdout onto app-json; a web container now emits
~15 uvicorn banner lines for its whole lifetime. It cleared only for the few
minutes after each deploy/autoscale event, producing 20+ false pages in a day.
NotifierHeartbeatMissing{Prod,Beta} matched a bare tag=heartbeat, which was
unambiguous until _start_heartbeat shipped at 2026-07-25T17:24Z and every
uvicorn process began beating under the same tag. The rule that had just
caught a real 17.5h notifier outage would have gone silent on the next one.
Both liveness rules now read the app's own heartbeat filtered by daemon, and
the notifier rules are pinned to daemon=subscription-notifier. db and frontend
keep their container-log rules, which remain valid. Queries verified against
live Loki: web 16, worker 4, notifier 2 per 20m window; beta notifier never
below 1 across 37 consecutive buckets.
|
||
|---|---|---|
| .claude | ||
| .forgejo/workflows | ||
| backend | ||
| frontend | ||
| infra | ||
| observability | ||
| .gitignore | ||
| CLAUDE.md | ||
| CUTOVER-NOTES.md | ||
| README.md | ||
thermograph
The Thermograph monorepo — the split repos reunified (2026-07-22) with full history via subtree merges, while keeping everything the split was actually for: per-domain images, per-domain deploys, and an async FE/BE contract.
Domains
| Dir | What | CI |
|---|---|---|
backend/ |
FastAPI graded-climate API, accounts, notifications (Discord bot, push, mail), data pipeline | build-push → image emi/thermograph/backend; deploy |
frontend/ |
Public client: static JS/CSS + SSR pages | same build-push / deploy workflows, matrixed by domain; image emi/thermograph/frontend |
infra/ |
Compose (beta, LAN dev) + the Swarm stack (prod), deploy scripts, terraform, SOPS secrets vault, ops cron | infra-sync (host checkout + secrets render), secrets-guard, ops-cron |
observability/ |
Loki + Grafana + Alloy stack | observability-validate |
thermograph-docs deliberately stays its own repo (ADRs + runbooks, no
build artifacts, different change cadence).
How CI stays decoupled
Every workflow in .forgejo/workflows/ is path-filtered to its domain: a
push touching only frontend/** builds/deploys nothing else. Images stay
separate (emi/thermograph/backend, emi/thermograph/frontend, each tagged
sha-<12hex>), deploys stay per-service (infra/deploy/deploy.sh SERVICE=backend|frontend|all), and the API version contract
(GET /api/version, PAYLOAD_VER) still lets FE and BE ship out of lockstep.
The one intentionally coupled piece is pr-build.yml: a single always-running
gate required check that builds only the domains a PR touches (a
path-filtered required check would deadlock auto-merge).
Branch model (unchanged from the split era): PRs → dev, main → beta,
release → prod; infra tracked via main on all hosts.
Before pointing anything live at this repo, read CUTOVER-NOTES.md.