thermograph/docs/onboarding/07-ci-and-release.md

7. CI and release

The branch model

PR ──(required check: `gate`)──▶ dev ──▶ dev (vps1, mesh-only)
                                  │
                          promotion PR
                                  ▼
                                 main ──▶ beta.thermograph.org (vps2)
                                  │
                          promotion PR  ← the owner's call, always
                                  ▼
                                release ──▶ thermograph.org (vps2)

dev, main and release are protected — direct pushes are blocked for everyone. Forgejo has no auto-merge-on-green, so merges are explicit: either click "Auto merge when checks succeed" on a PR, or merge via the API once CI is green.

Promotion is continuous and per-decision. There is no release calendar. Size a promotion to whatever has actually soaked on the environment below.

Who may merge what

Hop	Whose call	Norm
anything → dev	yours	Merge it. An open PR into dev is unfinished work, not caution.
dev → main (beta)	yours, batched	Promote when a batch is coherent enough to be worth testing. Beta is the test environment; not promoting is not testing.
main → release (prod)	the owner's	Prepare it, recommend for or against, and ask. Never unilaterally, never by another route.

Centralis's promote(from, to) implements exactly this, and forge_pr_merge enforces it with no back door: it merges into dev and main, refuses main → release under any flag, and requires confirm_protected_base: true for the separate hotfix path.

Two things about branch divergence

Commit counts are not deliverable content. Before promoting, compare the two tips' trees. Identical trees mean the same changes are already on the target under different SHAs — promoting there merges nothing while still firing the target's deploy workflows, including infra-sync re-rendering prod's and beta's env files (/etc/thermograph.env and /etc/thermograph-beta.env, both on vps2). The ahead/behind numbers will look like real work. promote refuses this outright.

Expect dev, main and release to be mutually divergent, and expect --ff-only to fail. Every promotion merges the source into the target, creating a merge commit on the target that the source never receives — so each promotion widens the gap it just closed. That is accumulated design, not drift. (Reconciling merges back into dev do happen; be4aa94 in the history is one.)

The nine workflows

All in root .forgejo/workflows/. Never add a workflow under a domain's own .forgejo/ — Forgejo only reads the root ones, so a copy there is inert and becomes a trap for whoever edits it next.

pr-build.yml — the required check

One workflow, one stable required-check name (gate), any number of domains:

1. a changes job diffs the PR against its base (needs fetch-depth: 0 — the base commit isn't reachable from a depth-1 clone);
2. per-domain jobs run only for domains the PR touches;
3. gate reduces their results into the single status branch protection requires.

Deliberately not path-filtered at the workflow level. A path-filtered required check that never fires leaves a PR stuck on "expected — waiting for status" forever, so auto-merge never proceeds. An always-running gate whose domain jobs skip cleanly gives the same compute savings without the deadlock. This is the one intentionally coupled piece of the CI design.

build.yml — the reusable build check

Called via uses: with a domain input. The build context is that domain's subdirectory, so each Dockerfile sees exactly the tree it saw when its domain was a repo root.

For backend it then runs the hermetic pytest suite inside the image it just built — the exact interpreter and dependencies that ship. Three flags there are load-bearing: -u 0 (the image's default user can't write to site-packages), --entrypoint sh (otherwise the test command is swallowed as entrypoint args and the container just boots the server forever), and unset THERMOGRAPH_BASE (the image bakes /, which moves every route off the /thermograph prefix the tests are written against).

For frontend there's no separate test step: it's a Go binary with no interpreter in the runtime image, and gofmt/vet/go test already run in the Dockerfile's builder stage — so a failing Go test fails the Build step above.

It is deliberately not a live boot+healthz check. That was tried in the split repos and repeatedly hit the runner's own environment quirks (bridge-IP reachability, non-unique $$, squatted host ports) without ever going reliably green. Image boot is verified by hand instead — make smoke, or docker run + alembic + /healthz 200 + a real /api/v2/place 200.

build-push.yml — images

Triggers on pushes to dev/main/release touching backend/** or frontend/**, and on v*.*.* tags. Publishes git.thermograph.org/Jinemi/thermograph/{backend,frontend}:sha-<12hex>.

• Serialised (max-parallel: 1): one physical runner, and two whole image builds racing each other buys nothing.
• Semver exception: a v*.*.* tag push carries no paths context, so a release tag builds both images. That's intended — a release names the whole set, not whichever domain moved last.
• The registry is mesh-only, so the runner host needs 10.10.0.2 git.thermograph.org in /etc/hosts.

deploy.yml — one file, three environments, two services

Formerly six near-identical files. Branch selects the environment (dev → dev, main → beta, release → prod) — three legs now, not two — and a matrix covers the services.

The style is deliberately boring: no dynamic fromJSON matrix, no cond && secrets.A || secrets.B ternary. Those are GitHub idioms a Forgejo/act runner may evaluate differently, and the failure mode here is "production does not deploy" or, worse, "deploys with an empty SSH host". So each environment gets its own explicit, mutually exclusive step and the tag is computed in shell.

Preserved from the originals, all load-bearing:

• fetch-depth: 0 — the tag is keyed to the last commit that touched that domain, not the branch tip.
• 12-hex truncation matching build-push.yml exactly.
• Per-service, per-environment concurrency with cancel-in-progress: false — a half-finished deploy must never be cancelled by a newer one.
• appleboy/ssh-action referenced by full URL; it isn't mirrored in Forgejo's default action registry.

SSH credentials are keyed by HOST, not by environment: VPS1_SSH_* and VPS2_SSH_*, replacing the old SSH_* (which actually meant beta) and PROD_SSH_* (which actually meant prod). That old naming used to read as a security boundary — "separate PROD_SSH_* credentials, so a beta credential leak cannot reach prod" — and on the old topology it happened to be true, because beta and prod were also on separate hosts. They no longer are: beta and prod are now two co-resident Swarm stacks on vps2, so a VPS2_SSH_* credential reaches both by construction, and there is no host-level SSH boundary between them to preserve. The boundary that replaces it lives at two other levels instead: the database (beta connects as its own NOSUPERUSER role to its own database, CONNECT revoked from PUBLIC on prod's) and the filesystem (separate checkouts, separate rendered env files, separate deploy locks — env-topology.sh derives all of it so nothing is shared by accident).

The credential boundary that is still real, and still matters, is vps1-vs-vps2: vps1 runs Forgejo, its own CI, and whatever's currently on dev — unreviewed by definition, since dev is exactly the branch PRs land on before review gates them into main. That is precisely why the dev environment renders dev.yaml alone and never layers common.yaml (the fleet's shared production credentials, including a read-write S3 keypair and the VAPID push-signing key) — see Infra and secrets. A VPS1_SSH_* leak should never be able to reach a production credential, which is a property common.yaml's dev exclusion buys independently of which SSH key was used to get there.

There's also a per-leg refinement: the workflow-level paths filter only says backend or frontend moved, so each leg re-checks whether this push touched this domain. Without it a backend-only push would also roll the frontend, losing the independent-deploy property the whole split exists for. When there's no usable base SHA (first push, force push, workflow_dispatch) it deploys rather than skips — rolling onto the tag already running is a no-op, whereas skipping silently strands a change.

infra-sync.yml — infra's own pipeline

Push to main touching infra/** → SSH to vps2, fast-forward both checkouts there (/opt/thermograph for prod, /opt/thermograph-beta for beta), re-render each one's own env file from the vault.

Rolls no service. Image tags are the app domains' axis, not infra's. A compose or stack change that must recreate containers takes effect on the next app deploy, or on a by-hand SERVICE=all … infra/deploy/deploy.sh (or deploy-dev.sh for dev).

Note the asymmetry: app code is environment-staged (dev→main→release maps to dev(vps1)→beta→prod(vps2) via image tags); infra is not — beta's and prod's checkouts both track infra via main, while dev's checkout tracks dev itself. Prod's app images are staged by release, but its checkout follows main.

observability-validate.yml

Parses every artifact that ships to a node: both compose files, all dashboard JSON, the Loki and provisioning YAML, and the Alloy config — checked with the pinned alloy binary (v1.9.1, matching what the fleet runs), downloaded from its GitHub release, no Docker CLI needed.

Alerting gets a stricter third step: every rule's condition must name a refId that exists, every policy must route to a receiver that exists, and a literal Discord webhook URL in the repo is a hard failure. A rule pointing at a missing refId is valid YAML, provisions cleanly, and then never fires — that silent no-op is exactly what this check exists to prevent.

ops-cron.yml — the prod backup

Daily at 03:00 UTC: pg_dump on prod, plus an IndexNow ping. Both SSH into vps2 and run inside prod's already-running stack — specifically prod's, not beta's, even though both now live on the same box.

It uses VPS2_SSH_*, not a beta-flavoured secret. An earlier revision (on the old topology) reused the credential meant for beta's host — so the "prod" backup was silently dumping beta, and prod had no backup at all. On vps2's co-residency, the host-keyed secret alone doesn't disambiguate prod from beta the way it used to; the script also needs to target prod's checkout and db specifically. If you touch this file, verify a dump actually lands in agent@vps2:~/thermograph-backups/ and that it's prod's data, not beta's.

secrets-guard.yml and shell-lint.yml — the backstops

Both run on every PR and push, deliberately un-path-filtered: a backstop that only runs when you expect it to isn't a backstop.

• secrets-guard fails if any infra/deploy/secrets/*.yaml lacks both a sops: block and ENC[AES256_GCM values. ~1 second.
• shell-lint runs shellcheck over every *.sh found by find (not a list — so a new script is covered the moment it lands). Pinned to v0.11.0 plus a sha256, installed from the official static release rather than apt-get: the distro version drifts with the job image, and a drifted shellcheck grows new warnings that fail CI on an unrelated push. Bump the version and the hash together, and expect to fix new findings in the same PR.

Keep the tree at zero shellcheck findings. These scripts run as root over SSH against live hosts with no test suite in front of them; this is the only guard.

Runner facts

• Jobs run on the docker label — vps1's Forgejo Actions runner (vps1 is where Forgejo itself lives; the desktop no longer runs a runner).
• Job containers get the host's docker.sock automounted (Docker-outside-of-Docker, no privileged mode). The job image (node:20-bookworm) ships no docker CLI, which is why several workflows apt-get install docker.io first.
• Forgejo's runner has only the labels it was explicitly registered with. GitHub implicitly tags every self-hosted runner self-hosted; Forgejo does not. A runs-on: [self-hosted, thermograph-lan] array is permanently unschedulable there — a real bug that was found and fixed once, and worth recognising if a job ever silently stops appearing in the Actions history.

Opening a PR

git switch -c fix/short-description        # branch off dev
# ... work ...
cd backend && make test                    # or: cd frontend/server && go test ./...
git commit                                 # concise, technical, no AI mentions
git push -u origin fix/short-description

Then open the PR into dev (Centralis's forge_pr_open, or the Forgejo web UI). Wait for gate. Merge — squash is the configured style.

If your change touches shell, expect shell-lint to have an opinion. If it touches infra/deploy/secrets/, secrets-guard will too.

Next: Infra and secrets.