thermograph/docs/onboarding/07-ci-and-release.md

# 7. CI and release

## The branch model

```
PR ──(required check: `gate`)──▶ dev ──▶ dev (vps1, mesh-only)
                                  │
                          promotion PR
                                  ▼
                                 main ──▶ beta.thermograph.org (vps2)
                                  │
                          promotion PR  ← the owner's call, always
                                  ▼
                                release ──▶ thermograph.org (vps2)
```

`dev`, `main` and `release` are protected — direct pushes are blocked for
everyone. Forgejo has **no auto-merge-on-green**, so merges are explicit: either
click "Auto merge when checks succeed" on a PR, or merge via the API once CI is
green.

Promotion is continuous and per-decision. There is no release calendar. Size a
promotion to whatever has actually soaked on the environment below.

### Who may merge what

| Hop | Whose call | Norm |
|---|---|---|
| anything → `dev` | yours | Merge it. An open PR into `dev` is unfinished work, not caution. |
| `dev` → `main` (beta) | yours, batched | Promote when a batch is coherent enough to be worth *testing*. Beta is the test environment; not promoting is not testing. |
| `main` → `release` (prod) | **the owner's** | Prepare it, recommend for or against, and ask. Never unilaterally, never by another route. |

Centralis's `promote(from, to)` implements exactly this, and `forge_pr_merge`
enforces it with no back door: it merges into `dev` and `main`, refuses
`main → release` under any flag, and requires `confirm_protected_base: true`
for the separate hotfix path.

### Two things about branch divergence

**Commit counts are not deliverable content.** Before promoting, compare the two
tips' *trees*. Identical trees mean the same changes are already on the target
under different SHAs — promoting there merges nothing while still firing the
target's deploy workflows, including `infra-sync` re-rendering prod's and
beta's env files (`/etc/thermograph.env` and `/etc/thermograph-beta.env`, both
on vps2). The ahead/behind numbers will look like real work. `promote` refuses
this outright.

**Expect `dev`, `main` and `release` to be mutually divergent, and expect
`--ff-only` to fail.** Every promotion merges the source into the target,
creating a merge commit *on the target* that the source never receives — so each
promotion widens the gap it just closed. That is accumulated design, not drift.
(Reconciling merges back into `dev` do happen; `be4aa94` in the history is one.)

## The nine workflows

All in root `.forgejo/workflows/`. **Never add a workflow under a domain's own
`.forgejo/`** — Forgejo only reads the root ones, so a copy there is inert and
becomes a trap for whoever edits it next.

### `pr-build.yml` — the required check

One workflow, one stable required-check name (**`gate`**), any number of
domains:

1. a `changes` job diffs the PR against its base (needs `fetch-depth: 0` — the
   base commit isn't reachable from a depth-1 clone);
2. per-domain jobs run only for domains the PR touches;
3. `gate` reduces their results into the single status branch protection
   requires.

**Deliberately not path-filtered at the workflow level.** A path-filtered
required check that never fires leaves a PR stuck on *"expected — waiting for
status"* forever, so auto-merge never proceeds. An always-running gate whose
domain jobs skip cleanly gives the same compute savings without the deadlock.
This is the one intentionally *coupled* piece of the CI design.

### `build.yml` — the reusable build check

Called via `uses:` with a `domain` input. The build **context is that domain's
subdirectory**, so each Dockerfile sees exactly the tree it saw when its domain
was a repo root.

For **backend** it then runs the hermetic pytest suite *inside the image it just
built* — the exact interpreter and dependencies that ship. Three flags there are
load-bearing: `-u 0` (the image's default user can't write to site-packages),
`--entrypoint sh` (otherwise the test command is swallowed as entrypoint args
and the container just boots the server forever), and `unset THERMOGRAPH_BASE`
(the image bakes `/`, which moves every route off the `/thermograph` prefix the
tests are written against).

For **frontend** there's no separate test step: it's a Go binary with no
interpreter in the runtime image, and `gofmt`/`vet`/`go test` already run in the
Dockerfile's builder stage — so a failing Go test fails the Build step above.

It is deliberately **not** a live boot+healthz check. That was tried in the split
repos and repeatedly hit the runner's own environment quirks (bridge-IP
reachability, non-unique `$$`, squatted host ports) without ever going reliably
green. Image boot is verified by hand instead — `make smoke`, or `docker run` +
alembic + `/healthz` 200 + a real `/api/v2/place` 200.

### `build-push.yml` — images

Triggers on pushes to `dev`/`main`/`release` touching `backend/**` or
`frontend/**`, and on `v*.*.*` tags. Publishes
`git.thermograph.org/Jinemi/thermograph/{backend,frontend}:sha-<12hex>`.

- **Serialised** (`max-parallel: 1`): one physical runner, and two whole image
  builds racing each other buys nothing.
- **Semver exception:** a `v*.*.*` tag push carries no paths context, so a
  release tag builds **both** images. That's intended — a release names the
  whole set, not whichever domain moved last.
- The registry is mesh-only, so the runner host needs
  `10.10.0.2 git.thermograph.org` in `/etc/hosts`.

### `deploy.yml` — one file, three environments, two services

Formerly six near-identical files. Branch selects the environment
(`dev → dev`, `main → beta`, `release → prod`) — **three legs now, not two** —
and a matrix covers the services.

The style is **deliberately boring**: no dynamic `fromJSON` matrix, no
`cond && secrets.A || secrets.B` ternary. Those are GitHub idioms a Forgejo/act
runner may evaluate differently, and the failure mode here is "production does
not deploy" or, worse, "deploys with an empty SSH host". So each environment
gets its own explicit, mutually exclusive step and the tag is computed in shell.

Preserved from the originals, all load-bearing:

- `fetch-depth: 0` — the tag is keyed to the last commit that touched *that
  domain*, not the branch tip.
- 12-hex truncation matching `build-push.yml` exactly.
- Per-service, per-environment concurrency with **`cancel-in-progress: false`** —
  a half-finished deploy must never be cancelled by a newer one.
- `appleboy/ssh-action` referenced by full URL; it isn't mirrored in Forgejo's
  default action registry.

**SSH credentials are keyed by HOST, not by environment**: `VPS1_SSH_*` and
`VPS2_SSH_*`, replacing the old `SSH_*` (which actually meant beta) and
`PROD_SSH_*` (which actually meant prod). That old naming used to read as a
security boundary — "separate `PROD_SSH_*` credentials, so a beta credential
leak cannot reach prod" — and on the old topology it happened to be true,
because beta and prod were also on separate hosts. They no longer are: beta and
prod are now two co-resident Swarm stacks on vps2, so a `VPS2_SSH_*` credential
reaches both by construction, and there is no host-level SSH boundary between
them to preserve. The boundary that replaces it lives at two other levels
instead: the **database** (beta connects as its own `NOSUPERUSER` role to its
own database, `CONNECT` revoked from `PUBLIC` on prod's) and the **filesystem**
(separate checkouts, separate rendered env files, separate deploy locks —
`env-topology.sh` derives all of it so nothing is shared by accident).

The credential boundary that *is* still real, and still matters, is
**vps1-vs-vps2**: vps1 runs Forgejo, its own CI, and whatever's currently on
`dev` — unreviewed by definition, since `dev` is exactly the branch PRs land on
before review gates them into `main`. That is precisely why the `dev`
environment renders `dev.yaml` alone and never layers `common.yaml` (the
fleet's shared production credentials, including a read-write S3 keypair and
the VAPID push-signing key) — see [Infra and secrets](08-infra-secrets.md). A
`VPS1_SSH_*` leak should never be able to reach a production credential, which
is a property `common.yaml`'s dev exclusion buys independently of which SSH key
was used to get there.

There's also a per-leg refinement: the workflow-level `paths` filter only says
*backend or frontend moved*, so each leg re-checks whether **this** push touched
**this** domain. Without it a backend-only push would also roll the frontend,
losing the independent-deploy property the whole split exists for. When there's
no usable base SHA (first push, force push, `workflow_dispatch`) it deploys
rather than skips — rolling onto the tag already running is a no-op, whereas
skipping silently strands a change.

### `infra-sync.yml` — infra's own pipeline

Push to `main` touching `infra/**` → SSH to vps2, fast-forward **both**
checkouts there (`/opt/thermograph` for prod, `/opt/thermograph-beta` for
beta), re-render each one's own env file from the vault.

**Rolls no service.** Image tags are the app domains' axis, not infra's. A
compose or stack change that must recreate containers takes effect on the next
app deploy, or on a by-hand `SERVICE=all … infra/deploy/deploy.sh` (or
`deploy-dev.sh` for dev).

Note the asymmetry: app code *is* environment-staged (`dev`→`main`→`release`
maps to dev(vps1)→beta→prod(vps2) via image tags); **infra is not** — beta's and
prod's checkouts both track infra via `main`, while dev's checkout tracks `dev`
itself. Prod's *app images* are staged by `release`, but its checkout follows
`main`.

### `observability-validate.yml`

Parses every artifact that ships to a node: both compose files, all dashboard
JSON, the Loki and provisioning YAML, and the Alloy config — checked with the
**pinned `alloy` binary (v1.9.1, matching what the fleet runs)**, downloaded
from its GitHub release, no Docker CLI needed.

Alerting gets a stricter third step: every rule's `condition` must name a refId
that exists, every policy must route to a receiver that exists, and a literal
Discord webhook URL in the repo is a hard failure. **A rule pointing at a
missing refId is valid YAML, provisions cleanly, and then never fires** — that
silent no-op is exactly what this check exists to prevent.

### `ops-cron.yml` — the prod backup

Daily at 03:00 UTC: `pg_dump` on prod, plus an IndexNow ping. Both SSH into
vps2 and run inside prod's already-running stack — specifically prod's, not
beta's, even though both now live on the same box.

**It uses `VPS2_SSH_*`, not a beta-flavoured secret.** An earlier revision (on
the old topology) reused the credential meant for beta's host — so the "prod"
backup was silently dumping *beta*, and prod had no backup at all. On vps2's
co-residency, the host-keyed secret alone doesn't disambiguate prod from beta
the way it used to; the script also needs to target prod's checkout and `db`
specifically. If you touch this file, verify a dump actually lands in
`agent@vps2:~/thermograph-backups/` and that it's prod's data, not beta's.

### `secrets-guard.yml` and `shell-lint.yml` — the backstops

Both run on **every** PR and push, deliberately un-path-filtered: *a backstop
that only runs when you expect it to isn't a backstop.*

- `secrets-guard` fails if any `infra/deploy/secrets/*.yaml` lacks both a
  `sops:` block and `ENC[AES256_GCM` values. ~1 second.
- `shell-lint` runs shellcheck over every `*.sh` found by `find` (not a list —
  so a new script is covered the moment it lands). Pinned to **v0.11.0 plus a
  sha256**, installed from the official static release rather than `apt-get`:
  the distro version drifts with the job image, and a drifted shellcheck grows
  new warnings that fail CI on an unrelated push. Bump the version and the hash
  together, and expect to fix new findings in the same PR.

Keep the tree at **zero shellcheck findings**. These scripts run as root over
SSH against live hosts with no test suite in front of them; this is the only
guard.

## Runner facts

- Jobs run on the `docker` label — vps1's Forgejo Actions runner (vps1 is where
  Forgejo itself lives; the desktop no longer runs a runner).
- Job containers get the **host's docker.sock** automounted
  (Docker-outside-of-Docker, no privileged mode). The job image
  (`node:20-bookworm`) ships no docker CLI, which is why several workflows
  `apt-get install docker.io` first.
- Forgejo's runner has **only the labels it was explicitly registered with**.
  GitHub implicitly tags every self-hosted runner `self-hosted`; Forgejo does
  not. A `runs-on: [self-hosted, thermograph-lan]` array is permanently
  unschedulable there — a real bug that was found and fixed once, and worth
  recognising if a job ever silently stops appearing in the Actions history.

## Opening a PR

```bash
git switch -c fix/short-description        # branch off dev
# ... work ...
cd backend && make test                    # or: cd frontend/server && go test ./...
git commit                                 # concise, technical, no AI mentions
git push -u origin fix/short-description
```

Then open the PR into `dev` (Centralis's `forge_pr_open`, or the Forgejo web
UI). Wait for `gate`. Merge — squash is the configured style.

If your change touches shell, expect `shell-lint` to have an opinion. If it
touches `infra/deploy/secrets/`, `secrets-guard` will too.

Next: [Infra and secrets](08-infra-secrets.md).