# Dev CI/CD → LAN server on this machine

Parallel to the prod pipeline (`main` → VPS, see `DEPLOY.md`), the **`dev`**
branch continuously deploys to a **LAN server running on this computer**. Git
hosting and CI are self-hosted **Forgejo** (`git.thermograph.org`, reachable
at `http://10.10.0.2:3080` over the WireGuard mesh) — GitHub is retired.

```
open PR ──▶ CI (build + boot/health, on the LAN Forgejo runner)
              │ green
              ▼
        merge into dev (explicit — Forgejo does not auto-merge on its own;
                         see "Landing a PR" below)
              │
              ▼
        LAN runner (thermograph-lan label) runs deploy/deploy-dev.sh
              │
              ▼
        ~/thermograph-dev updated ─▶ docker compose stack (backend + frontend +
                                     Postgres 18) brought up, backend on 0.0.0.0:8137
```

Reachable at `http://<lan-ip>:8137/` from any device on your Wi-Fi.

The dev server runs the **same containerized stack as prod** (`docker-compose.yml`),
overlaid with `docker-compose.dev.yml`: **uncapped** (no CPU limits, unlike prod's
backend=4 / frontend=2 / db=2) and backend published on `0.0.0.0:8137` for the
LAN (prod binds loopback behind Caddy; frontend has no published port here
either way, no Caddy to reach it directly, so it's only reached through
backend's own reverse-proxy fallback). Bring it up by hand with `make dev-up`.

## Why it's built this way

Forgejo has native branch protection + required-status-checks + auto-merge
(unlike GitHub on a private free-tier repo, where those are paywalled). But
"auto-merge" here still means opting a PR in — either clicking **"Auto merge
when checks succeed"** in the web UI, or merging explicitly once CI is green
via the API (`POST .../pulls/{n}/merge`, `{"Do": "squash"}`). Nothing merges
a ready PR for you automatically just because checks pass, unlike GitHub's
old in-workflow `ci-cd.yml` (retired along with the rest of `.github/`).

## Moving parts

| File | Purpose |
|------|---------|
| `.forgejo/workflows/pr-build.yml` | required status check for PRs into `dev` (calls `build.yml`) |
| `.forgejo/workflows/deploy-dev.yml` | build + deploy on pushes to `dev` (direct, or a PR merge) |
| `.forgejo/workflows/build.yml` | shared build gate: deps, backend tests, JS syntax check, boot/health check |
| `deploy/deploy-dev.sh` | pull `dev` into `~/thermograph-dev`, `docker compose up` the uncapped LAN stack, health check |
| `deploy/secrets/dev.yaml` | dev's SOPS vault — its own secrets and config, rendered to `/etc/thermograph.env` at deploy time (see "Config and secrets") |
| `docker-compose.dev.yml` | dev overlay: no CPU limits, backend published on `0.0.0.0:8137` (frontend unpublished) |
| `deploy/provision-dev-lan.sh` | one-time sudo-free bootstrap of the checkout + runner |
| `deploy/forgejo/register-lan-runner.sh` | registers the Forgejo Actions runner on this machine |

`deploy-dev.yml`'s `deploy` job runs on the `thermograph-lan` label — **not**
`[self-hosted, thermograph-lan]` (the array form is what the original GitHub
version used; GitHub implicitly tags every self-hosted runner `self-hosted`,
but Forgejo's runner has only the labels it was explicitly registered with,
so the array form is permanently unschedulable there — a real bug found and
fixed once, worth knowing about if a "deploy" job ever silently stops
appearing in the Actions history again).

`deploy-dev.sh`'s git auth (`GH_TOKEN`/`GITHUB_TOKEN`, Forgejo's per-job
token exposed under that name for compatibility) is scoped to `REPO_URL`'s
own host via a per-command `http.<scheme>://<host>/.extraheader`, not a
hardcoded one — important if the Forgejo host ever changes, since a stale
hardcoded host would make git silently skip the header (no error) and fall
through to whatever ambient credentials happen to be available, not fail
loudly.

## The self-hosted runner

A Forgejo Actions **runner** (`forgejo-runner`) runs on this machine, labels
`docker` (containerized jobs, node:20-bookworm, with the host's docker.sock
automounted in — `container.docker_host: automount` in
`~/forgejo-runner/config.yaml` — so `docker build`/`push` work without
privileged Docker-in-Docker) and `thermograph-lan` (host-native jobs, for
`deploy-dev.sh`'s systemctl/docker-compose calls). Installed under
`~/forgejo-runner`, runs as the `forgejo-runner` systemd `--user` service.

```bash
# status / logs
systemctl --user status forgejo-runner
journalctl --user -u forgejo-runner -f

# re-register if the token/host ever changes
bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
```

`git.thermograph.org`'s public DNS resolves to beta's **public** IP, which
the registry's mesh-only ACL (see `deploy/forgejo/README.md`) rejects for
`/v2/` paths — `docker build-push.yml` pushes run against the *host's*
automounted daemon, so it's this **host's** own resolver that needs to route
git.thermograph.org over the mesh, not anything settable from a job
container. If registry pushes ever start failing with an HTTP/TLS mismatch
or a 403 on `/v2/`, check `getent hosts git.thermograph.org` here — it needs
to resolve to beta's WireGuard IP (`10.10.0.2`), typically via a `/etc/hosts`
line, not the public one.

## The LAN dev service (Docker Compose)

Runs as a Docker Compose stack, not a bare systemd unit — `docker ps --filter
name=thermograph-dev` shows `thermograph-dev-backend-1`, `thermograph-dev-frontend-1`
(repo-split Stage 4 split the single "app" container in two — frontend has no
published port, reached only through backend's own reverse-proxy fallback) and
`thermograph-dev-db-1` (TimescaleDB). A legacy pre-container
`thermograph-dev.service` systemd --user unit may still exist from before this
cutover; it's stale and `deploy-dev.sh` stops/disables it on every run — don't
trust `systemctl --user status thermograph-dev` as a liveness check, use `docker
ps` instead.

```bash
docker ps --filter name=thermograph-dev                                          # is it up?
docker compose -f docker-compose.yml -f docker-compose.dev.yml logs -f backend frontend   # app logs (from ~/thermograph-dev)
```

Bootstrap (or rebuild) it by hand:

```bash
bash deploy/provision-dev-lan.sh
```

The dedicated checkout at `~/thermograph-dev` is separate from your working tree,
so a deploy never touches uncommitted edits in `~/Code/Thermograph`.

## Config and secrets

Dev's configuration lives in the same SOPS vault as prod's and beta's —
`deploy/secrets/dev.yaml`, encrypted to the same age recipient, rendered to
`/etc/thermograph.env` by the same `render-secrets.sh` at deploy time. Changing a
dev value is `sops deploy/secrets/dev.yaml` → commit → deploy, exactly as for the
VPS boxes. **`deploy/secrets/README.md` is the reference**; what follows is only
what is specific to this machine.

Dev renders `dev.yaml` **alone**. It does *not* layer `common.yaml`, which is the
fleet's shared production credential set — registry token, both S3 keypairs (one
read-write pair, on the bucket that also holds prod's backups), the VAPID private
key that signs push to real subscribers, the IndexNow key, the metrics token. This
box is the CI runner and runs unreviewed `dev`-branch code; a plaintext render of
that set into `/etc/thermograph.env` would put all of it in every dev container's
environment. `deploy-dev.sh` exports `THERMOGRAPH_SECRETS_SKIP_COMMON=1` to prevent
it, and **aborts the deploy** if the renderer in the checkout cannot honour that.

So dev holds its own `THERMOGRAPH_AUTH_SECRET` (its own, not prod's — the `daemon`
service refuses to start without one), its own `POSTGRES_PASSWORD`, a LAN
`THERMOGRAPH_BASE_URL`, `THERMOGRAPH_COOKIE_SECURE=0` (plain HTTP — the shared
value is `1` and would silently break login here), and the non-secret config it
would otherwise have inherited. It holds **no** S3 keys, no VAPID keypair, no
IndexNow key, no Discord or mail credentials: the lake falls through to the
Open-Meteo archive without bucket creds, and the app generates and persists its own
VAPID and IndexNow keys into the `appdata` volume.

Two dev-only mechanics, both because this box has **no passwordless sudo** and the
runner is a `systemd --user` service with no tty:

- the age key is read from `~/.config/sops/age/keys.txt`, not
  `/etc/thermograph/age.key` — a root-owned `0400` key would send the renderer down
  its `sudo cat` path, where it would hang on a prompt no CI job can answer;
- `/etc/thermograph.env` is **pre-created** owned by the deploying user, so the
  renderer takes its in-place write path and no deploy needs `sudo` at all.

Setting this up on a fresh dev machine is four typed commands — see **"Setting up a
new dev machine"** in `deploy/secrets/README.md`. Until the
`/etc/thermograph/secrets-env` marker exists the box renders nothing and falls back
to `deploy-dev.sh`'s built-in `POSTGRES_PASSWORD` default, which is the safe state
and the one to return to (delete the marker) if the vault ever gets in the way.

`REGISTRY_TOKEN` is deliberately not in the vault: dev pulls with the persistent
`docker login` credential already in this host's docker config, like the prod/beta
CI paths do.

## Before monitoring a PR to land

A PR needs an explicit merge once CI is green — but first confirm it's
actually set up to land, or you'll monitor a PR that can never merge:

1. **Not conflicting** — `GET /api/v1/repos/{owner}/{repo}/pulls/{n}` should
   show `"mergeable": true`. If not, merge `forgejo/dev` in and resolve now —
   don't monitor a PR that can't merge.
2. **Required check actually configured** — `GET
   /api/v1/repos/{owner}/{repo}/branch_protections/dev` should list
   `status_check_contexts` containing the *exact* string Forgejo reports for
   that PR's check (workflow display name + job name + event, e.g. `"PR build
   (required check) / build (pull_request)"` — not just the job name
   `"build"`). A mismatch here means no PR can ever satisfy the check even
   with fully green CI; this bit us once already.
3. **CI actually running** — `GET
   /api/v1/repos/{owner}/{repo}/commits/{sha}/status` shows a status, not an
   empty list. If empty, check the runner is up
   (`systemctl --user status forgejo-runner`).

Only start monitoring once all three pass. Once CI is green and mergeable is
true, merge explicitly: `POST /api/v1/repos/{owner}/{repo}/pulls/{n}/merge`
with `{"Do": "squash", "delete_branch_after_merge": true}`.

## Day-to-day

- **Ship:** open a PR into `dev`, confirm CI is green, merge it explicitly —
  the merge triggers the deploy here.
- **Manual redeploy:** trigger `deploy-dev.yml` via `workflow_dispatch`
  (`POST /api/v1/repos/{owner}/{repo}/actions/workflows/deploy-dev.yml/dispatches`,
  `{"ref": "dev"}`), or locally `APP_DIR=~/thermograph-dev bash deploy/deploy-dev.sh`.
- **Promote to prod:** merge/fast-forward `dev` into `main` and push — that
  triggers the VPS pipeline in `DEPLOY.md`.

## Monitoring

Logs and dashboards moved to a **separate project** — a central Grafana + Loki
stack fed by a Grafana Alloy agent on every node (including this LAN dev box),
over the WireGuard mesh (`thermograph-observability` on Forgejo; UI at
`https://dashboard.thermograph.org`). The dev node's Alloy agent ships its
container/app logs under `host="dev"`, so LAN dev shows up alongside prod and
beta in the same dashboards. This replaced the old `scripts/dashboard.py`.

For a quick terminal check without Grafana, the app still serves raw counters at
the gated `GET /api/v2/metrics` route. Dev sets no `THERMOGRAPH_METRICS_TOKEN`, so
the route is direct-loopback-only — `curl localhost:8137/api/v2/metrics` from this
machine, which is where you already are. (A token exists on the VPS boxes to allow
reads through an SSH tunnel; dev has no reason to carry a copy, and the estate's
copy must not land here — see `deploy/secrets/README.md`.)