thermograph/docs/onboarding/07-ci-and-release.md

7. CI and release

The branch model

PR ──(required check: `gate`)──▶ dev ──▶ LAN dev box
                                  │
                          promotion PR
                                  ▼
                                 main ──▶ beta.thermograph.org
                                  │
                          promotion PR  ← the owner's call, always
                                  ▼
                                release ──▶ thermograph.org

dev, main and release are protected — direct pushes are blocked for everyone. Forgejo has no auto-merge-on-green, so merges are explicit: either click "Auto merge when checks succeed" on a PR, or merge via the API once CI is green.

Promotion is continuous and per-decision. There is no release calendar. Size a promotion to whatever has actually soaked on the environment below.

Who may merge what

Hop	Whose call	Norm
anything → dev	yours	Merge it. An open PR into dev is unfinished work, not caution.
dev → main (beta)	yours, batched	Promote when a batch is coherent enough to be worth testing. Beta is the test environment; not promoting is not testing.
main → release (prod)	the owner's	Prepare it, recommend for or against, and ask. Never unilaterally, never by another route.

Centralis's promote(from, to) implements exactly this, and forge_pr_merge enforces it with no back door: it merges into dev and main, refuses main → release under any flag, and requires confirm_protected_base: true for the separate hotfix path.

Two things about branch divergence

Commit counts are not deliverable content. Before promoting, compare the two tips' trees. Identical trees mean the same changes are already on the target under different SHAs — promoting there merges nothing while still firing the target's deploy workflows, including infra-sync re-rendering /etc/thermograph.env on prod and beta. The ahead/behind numbers will look like real work. promote refuses this outright.

Expect dev, main and release to be mutually divergent, and expect --ff-only to fail. Every promotion merges the source into the target, creating a merge commit on the target that the source never receives — so each promotion widens the gap it just closed. That is accumulated design, not drift. (Reconciling merges back into dev do happen; be4aa94 in the history is one.)

The nine workflows

All in root .forgejo/workflows/. Never add a workflow under a domain's own .forgejo/ — Forgejo only reads the root ones, so a copy there is inert and becomes a trap for whoever edits it next.

pr-build.yml — the required check

One workflow, one stable required-check name (gate), any number of domains:

1. a changes job diffs the PR against its base (needs fetch-depth: 0 — the base commit isn't reachable from a depth-1 clone);
2. per-domain jobs run only for domains the PR touches;
3. gate reduces their results into the single status branch protection requires.

Deliberately not path-filtered at the workflow level. A path-filtered required check that never fires leaves a PR stuck on "expected — waiting for status" forever, so auto-merge never proceeds. An always-running gate whose domain jobs skip cleanly gives the same compute savings without the deadlock. This is the one intentionally coupled piece of the CI design.

build.yml — the reusable build check

Called via uses: with a domain input. The build context is that domain's subdirectory, so each Dockerfile sees exactly the tree it saw when its domain was a repo root.

For backend it then runs the hermetic pytest suite inside the image it just built — the exact interpreter and dependencies that ship. Three flags there are load-bearing: -u 0 (the image's default user can't write to site-packages), --entrypoint sh (otherwise the test command is swallowed as entrypoint args and the container just boots the server forever), and unset THERMOGRAPH_BASE (the image bakes /, which moves every route off the /thermograph prefix the tests are written against).

For frontend there's no separate test step: it's a Go binary with no interpreter in the runtime image, and gofmt/vet/go test already run in the Dockerfile's builder stage — so a failing Go test fails the Build step above.

It is deliberately not a live boot+healthz check. That was tried in the split repos and repeatedly hit the runner's own environment quirks (bridge-IP reachability, non-unique $$, squatted host ports) without ever going reliably green. Image boot is verified by hand instead — make smoke, or docker run + alembic + /healthz 200 + a real /api/v2/place 200.

build-push.yml — images

Triggers on pushes to dev/main/release touching backend/** or frontend/**, and on v*.*.* tags. Publishes git.thermograph.org/emi/thermograph/{backend,frontend}:sha-<12hex>.

• Serialised (max-parallel: 1): one physical runner, and two whole image builds racing each other buys nothing.
• Semver exception: a v*.*.* tag push carries no paths context, so a release tag builds both images. That's intended — a release names the whole set, not whichever domain moved last.
• The registry is mesh-only, so the runner host needs 10.10.0.2 git.thermograph.org in /etc/hosts.

deploy.yml — one file, two environments, two services

Formerly six near-identical files. Branch selects the environment (main → beta, release → prod); a matrix covers the services.

The style is deliberately boring: no dynamic fromJSON matrix, no cond && secrets.A || secrets.B ternary. Those are GitHub idioms a Forgejo/act runner may evaluate differently, and the failure mode here is "production does not deploy" or, worse, "deploys with an empty SSH host". So the two environments get two explicit, mutually exclusive steps and the tag is computed in shell.

Preserved from the originals, all load-bearing:

• fetch-depth: 0 — the tag is keyed to the last commit that touched that domain, not the branch tip.
• 12-hex truncation matching build-push.yml exactly.
• Per-service, per-environment concurrency with cancel-in-progress: false — a half-finished deploy must never be cancelled by a newer one.
• Separate PROD_SSH_* credentials, so a beta credential leak cannot reach prod.
• appleboy/ssh-action referenced by full URL; it isn't mirrored in Forgejo's default action registry.

There's also a per-leg refinement: the workflow-level paths filter only says backend or frontend moved, so each leg re-checks whether this push touched this domain. Without it a backend-only push would also roll the frontend, losing the independent-deploy property the whole split exists for. When there's no usable base SHA (first push, force push, workflow_dispatch) it deploys rather than skips — rolling onto the tag already running is a no-op, whereas skipping silently strands a change.

infra-sync.yml — infra's own pipeline

Push to main touching infra/** → SSH to beta and prod, fast-forward /opt/thermograph, re-render /etc/thermograph.env from the vault.

Rolls no service. Image tags are the app domains' axis, not infra's. A compose change that must recreate containers takes effect on the next app deploy, or on a by-hand SERVICE=all … infra/deploy/deploy.sh.

Note the asymmetry: app code is environment-staged (dev→main→release maps to LAN→beta→prod via image tags); infra is not — both hosts track infra via main. Prod's app images are staged by release, but its checkout follows main.

observability-validate.yml

Parses every artifact that ships to a node: both compose files, all dashboard JSON, the Loki and provisioning YAML, and the Alloy config — checked with the pinned alloy binary (v1.9.1, matching what the fleet runs), downloaded from its GitHub release, no Docker CLI needed.

Alerting gets a stricter third step: every rule's condition must name a refId that exists, every policy must route to a receiver that exists, and a literal Discord webhook URL in the repo is a hard failure. A rule pointing at a missing refId is valid YAML, provisions cleanly, and then never fires — that silent no-op is exactly what this check exists to prevent.

ops-cron.yml — the prod backup

Daily at 03:00 UTC: pg_dump on prod, plus an IndexNow ping. Both SSH into prod and run inside the already-running stack.

It uses PROD_SSH_*, not SSH_*. An earlier revision reused SSH_* — so the "prod" backup was silently dumping beta, and prod had no backup at all. If you touch this file, verify a dump actually lands in agent@prod:~/thermograph-backups/.

secrets-guard.yml and shell-lint.yml — the backstops

Both run on every PR and push, deliberately un-path-filtered: a backstop that only runs when you expect it to isn't a backstop.

• secrets-guard fails if any infra/deploy/secrets/*.yaml lacks both a sops: block and ENC[AES256_GCM values. ~1 second.
• shell-lint runs shellcheck over every *.sh found by find (not a list — so a new script is covered the moment it lands). Pinned to v0.11.0 plus a sha256, installed from the official static release rather than apt-get: the distro version drifts with the job image, and a drifted shellcheck grows new warnings that fail CI on an unrelated push. Bump the version and the hash together, and expect to fix new findings in the same PR.

Keep the tree at zero shellcheck findings. These scripts run as root over SSH against live hosts with no test suite in front of them; this is the only guard.

Runner facts

• Jobs run on the docker label — the LAN box's Forgejo Actions runner.
• Job containers get the host's docker.sock automounted (Docker-outside-of-Docker, no privileged mode). The job image (node:20-bookworm) ships no docker CLI, which is why several workflows apt-get install docker.io first.
• Forgejo's runner has only the labels it was explicitly registered with. GitHub implicitly tags every self-hosted runner self-hosted; Forgejo does not. A runs-on: [self-hosted, thermograph-lan] array is permanently unschedulable there — a real bug that was found and fixed once, and worth recognising if a job ever silently stops appearing in the Actions history.

Opening a PR

git switch -c fix/short-description        # branch off dev
# ... work ...
cd backend && make test                    # or: cd frontend/server && go test ./...
git commit                                 # concise, technical, no AI mentions
git push -u origin fix/short-description

Then open the PR into dev (Centralis's forge_pr_open, or the Forgejo web UI). Wait for gate. Merge — squash is the configured style.

If your change touches shell, expect shell-lint to have an opinion. If it touches infra/deploy/secrets/, secrets-guard will too.

Next: Infra and secrets.