thermograph/docs/onboarding/07-ci-and-release.md

# 7. CI and release

## The branch model

```
PR ──(required check: `gate`)──▶ dev ──▶ LAN dev box
                                  │
                          promotion PR
                                  ▼
                                 main ──▶ beta.thermograph.org
                                  │
                          promotion PR  ← the owner's call, always
                                  ▼
                                release ──▶ thermograph.org
```

`dev`, `main` and `release` are protected — direct pushes are blocked for
everyone. Forgejo has **no auto-merge-on-green**, so merges are explicit: either
click "Auto merge when checks succeed" on a PR, or merge via the API once CI is
green.

Promotion is continuous and per-decision. There is no release calendar. Size a
promotion to whatever has actually soaked on the environment below.

### Who may merge what

| Hop | Whose call | Norm |
|---|---|---|
| anything → `dev` | yours | Merge it. An open PR into `dev` is unfinished work, not caution. |
| `dev` → `main` (beta) | yours, batched | Promote when a batch is coherent enough to be worth *testing*. Beta is the test environment; not promoting is not testing. |
| `main` → `release` (prod) | **the owner's** | Prepare it, recommend for or against, and ask. Never unilaterally, never by another route. |

Centralis's `promote(from, to)` implements exactly this, and `forge_pr_merge`
enforces it with no back door: it merges into `dev` and `main`, refuses
`main → release` under any flag, and requires `confirm_protected_base: true`
for the separate hotfix path.

### Two things about branch divergence

**Commit counts are not deliverable content.** Before promoting, compare the two
tips' *trees*. Identical trees mean the same changes are already on the target
under different SHAs — promoting there merges nothing while still firing the
target's deploy workflows, including `infra-sync` re-rendering
`/etc/thermograph.env` on prod and beta. The ahead/behind numbers will look like
real work. `promote` refuses this outright.

**Expect `dev`, `main` and `release` to be mutually divergent, and expect
`--ff-only` to fail.** Every promotion merges the source into the target,
creating a merge commit *on the target* that the source never receives — so each
promotion widens the gap it just closed. That is accumulated design, not drift.
(Reconciling merges back into `dev` do happen; `be4aa94` in the history is one.)

## The nine workflows

All in root `.forgejo/workflows/`. **Never add a workflow under a domain's own
`.forgejo/`** — Forgejo only reads the root ones, so a copy there is inert and
becomes a trap for whoever edits it next.

### `pr-build.yml` — the required check

One workflow, one stable required-check name (**`gate`**), any number of
domains:

1. a `changes` job diffs the PR against its base (needs `fetch-depth: 0` — the
   base commit isn't reachable from a depth-1 clone);
2. per-domain jobs run only for domains the PR touches;
3. `gate` reduces their results into the single status branch protection
   requires.

**Deliberately not path-filtered at the workflow level.** A path-filtered
required check that never fires leaves a PR stuck on *"expected — waiting for
status"* forever, so auto-merge never proceeds. An always-running gate whose
domain jobs skip cleanly gives the same compute savings without the deadlock.
This is the one intentionally *coupled* piece of the CI design.

### `build.yml` — the reusable build check

Called via `uses:` with a `domain` input. The build **context is that domain's
subdirectory**, so each Dockerfile sees exactly the tree it saw when its domain
was a repo root.

For **backend** it then runs the hermetic pytest suite *inside the image it just
built* — the exact interpreter and dependencies that ship. Three flags there are
load-bearing: `-u 0` (the image's default user can't write to site-packages),
`--entrypoint sh` (otherwise the test command is swallowed as entrypoint args
and the container just boots the server forever), and `unset THERMOGRAPH_BASE`
(the image bakes `/`, which moves every route off the `/thermograph` prefix the
tests are written against).

For **frontend** there's no separate test step: it's a Go binary with no
interpreter in the runtime image, and `gofmt`/`vet`/`go test` already run in the
Dockerfile's builder stage — so a failing Go test fails the Build step above.

It is deliberately **not** a live boot+healthz check. That was tried in the split
repos and repeatedly hit the runner's own environment quirks (bridge-IP
reachability, non-unique `$$`, squatted host ports) without ever going reliably
green. Image boot is verified by hand instead — `make smoke`, or `docker run` +
alembic + `/healthz` 200 + a real `/api/v2/place` 200.

### `build-push.yml` — images

Triggers on pushes to `dev`/`main`/`release` touching `backend/**` or
`frontend/**`, and on `v*.*.*` tags. Publishes
`git.thermograph.org/emi/thermograph/{backend,frontend}:sha-<12hex>`.

- **Serialised** (`max-parallel: 1`): one physical runner, and two whole image
  builds racing each other buys nothing.
- **Semver exception:** a `v*.*.*` tag push carries no paths context, so a
  release tag builds **both** images. That's intended — a release names the
  whole set, not whichever domain moved last.
- The registry is mesh-only, so the runner host needs
  `10.10.0.2 git.thermograph.org` in `/etc/hosts`.

### `deploy.yml` — one file, two environments, two services

Formerly six near-identical files. Branch selects the environment
(`main → beta`, `release → prod`); a matrix covers the services.

The style is **deliberately boring**: no dynamic `fromJSON` matrix, no
`cond && secrets.A || secrets.B` ternary. Those are GitHub idioms a Forgejo/act
runner may evaluate differently, and the failure mode here is "production does
not deploy" or, worse, "deploys with an empty SSH host". So the two environments
get two explicit, mutually exclusive steps and the tag is computed in shell.

Preserved from the originals, all load-bearing:

- `fetch-depth: 0` — the tag is keyed to the last commit that touched *that
  domain*, not the branch tip.
- 12-hex truncation matching `build-push.yml` exactly.
- Per-service, per-environment concurrency with **`cancel-in-progress: false`** —
  a half-finished deploy must never be cancelled by a newer one.
- **Separate `PROD_SSH_*` credentials**, so a beta credential leak cannot reach
  prod.
- `appleboy/ssh-action` referenced by full URL; it isn't mirrored in Forgejo's
  default action registry.

There's also a per-leg refinement: the workflow-level `paths` filter only says
*backend or frontend moved*, so each leg re-checks whether **this** push touched
**this** domain. Without it a backend-only push would also roll the frontend,
losing the independent-deploy property the whole split exists for. When there's
no usable base SHA (first push, force push, `workflow_dispatch`) it deploys
rather than skips — rolling onto the tag already running is a no-op, whereas
skipping silently strands a change.

### `infra-sync.yml` — infra's own pipeline

Push to `main` touching `infra/**` → SSH to beta **and** prod, fast-forward
`/opt/thermograph`, re-render `/etc/thermograph.env` from the vault.

**Rolls no service.** Image tags are the app domains' axis, not infra's. A
compose change that must recreate containers takes effect on the next app
deploy, or on a by-hand `SERVICE=all … infra/deploy/deploy.sh`.

Note the asymmetry: app code *is* environment-staged (`dev`→`main`→`release`
maps to LAN→beta→prod via image tags); **infra is not** — both hosts track infra
via `main`. Prod's *app images* are staged by `release`, but its checkout
follows `main`.

### `observability-validate.yml`

Parses every artifact that ships to a node: both compose files, all dashboard
JSON, the Loki and provisioning YAML, and the Alloy config — checked with the
**pinned `alloy` binary (v1.9.1, matching what the fleet runs)**, downloaded
from its GitHub release, no Docker CLI needed.

Alerting gets a stricter third step: every rule's `condition` must name a refId
that exists, every policy must route to a receiver that exists, and a literal
Discord webhook URL in the repo is a hard failure. **A rule pointing at a
missing refId is valid YAML, provisions cleanly, and then never fires** — that
silent no-op is exactly what this check exists to prevent.

### `ops-cron.yml` — the prod backup

Daily at 03:00 UTC: `pg_dump` on prod, plus an IndexNow ping. Both SSH into prod
and run inside the already-running stack.

**It uses `PROD_SSH_*`, not `SSH_*`.** An earlier revision reused `SSH_*` — so
the "prod" backup was silently dumping *beta*, and prod had no backup at all. If
you touch this file, verify a dump actually lands in
`agent@prod:~/thermograph-backups/`.

### `secrets-guard.yml` and `shell-lint.yml` — the backstops

Both run on **every** PR and push, deliberately un-path-filtered: *a backstop
that only runs when you expect it to isn't a backstop.*

- `secrets-guard` fails if any `infra/deploy/secrets/*.yaml` lacks both a
  `sops:` block and `ENC[AES256_GCM` values. ~1 second.
- `shell-lint` runs shellcheck over every `*.sh` found by `find` (not a list —
  so a new script is covered the moment it lands). Pinned to **v0.11.0 plus a
  sha256**, installed from the official static release rather than `apt-get`:
  the distro version drifts with the job image, and a drifted shellcheck grows
  new warnings that fail CI on an unrelated push. Bump the version and the hash
  together, and expect to fix new findings in the same PR.

Keep the tree at **zero shellcheck findings**. These scripts run as root over
SSH against live hosts with no test suite in front of them; this is the only
guard.

## Runner facts

- Jobs run on the `docker` label — the LAN box's Forgejo Actions runner.
- Job containers get the **host's docker.sock** automounted
  (Docker-outside-of-Docker, no privileged mode). The job image
  (`node:20-bookworm`) ships no docker CLI, which is why several workflows
  `apt-get install docker.io` first.
- Forgejo's runner has **only the labels it was explicitly registered with**.
  GitHub implicitly tags every self-hosted runner `self-hosted`; Forgejo does
  not. A `runs-on: [self-hosted, thermograph-lan]` array is permanently
  unschedulable there — a real bug that was found and fixed once, and worth
  recognising if a job ever silently stops appearing in the Actions history.

## Opening a PR

```bash
git switch -c fix/short-description        # branch off dev
# ... work ...
cd backend && make test                    # or: cd frontend/server && go test ./...
git commit                                 # concise, technical, no AI mentions
git push -u origin fix/short-description
```

Then open the PR into `dev` (Centralis's `forge_pr_open`, or the Forgejo web
UI). Wait for `gate`. Merge — squash is the configured style.

If your change touches shell, expect `shell-lint` to have an opinion. If it
touches `infra/deploy/secrets/`, `secrets-guard` will too.

Next: [Infra and secrets](08-infra-secrets.md).