thermograph/infra/DEPLOY-DEV.md

# Dev CI/CD → the dev environment on vps1

Parallel to beta/prod's pipeline (`main`→beta, `release`→prod, see `DEPLOY.md`),
the **`dev`** branch continuously deploys to the **dev environment on vps1**
(`75.119.132.91`, mesh `10.10.0.2`) — `/opt/thermograph-dev`, a normal fleet
checkout reached over SSH exactly like beta and prod, **not** a stack on the
operator's desktop or LAN any more. Git hosting and CI are self-hosted
**Forgejo** (`dev.jinemi.com`, also on vps1, reachable at
`http://10.10.0.2:3080` over the WireGuard mesh) — GitHub is retired.

```
open PR ──▶ CI (build + boot/health, on the `docker`-labeled Forgejo runner)
              │ green
              ▼
        merge into dev (explicit — Forgejo does not auto-merge on its own;
                         see "Landing a PR" below)
              │
              ▼
        the Deploy workflow SSHes into vps1 (VPS1_SSH_*, THERMOGRAPH_ENV=dev)
        and runs deploy/deploy-dev.sh
              │
              ▼
        /opt/thermograph-dev updated ─▶ docker compose stack (backend + frontend +
                                         Postgres) brought up, backend published
                                         MESH-ONLY on 10.10.0.2:8137
```

Reachable at `http://10.10.0.2:8137/` from anything already on the WireGuard
mesh, and from nowhere else — no public DNS record, no Caddy site, no TLS.
vps1 is a public VPS running whatever branch is currently in flight (including
unreviewed code), so unlike the old LAN box it must **never** publish on
`0.0.0.0`.

The dev stack runs the **same containerized stack as prod** (`docker-compose.yml`),
overlaid with `docker-compose.dev.yml`: **uncapped** (no CPU limits, unlike
prod's Swarm-managed replicas) and backend published on the WireGuard address
(`10.10.0.2:8137` — set via `DEV_BIND_ADDR` in `deploy/env-topology.sh`; prod
and beta bind loopback behind their own Caddy LBs instead). Frontend has no
published port either way — reached only through backend's own
reverse-proxy fallback. Bring dev up by hand with `make dev-up`, which is a
**laptop-local convenience** (uncapped, loopback-published) — not the dev
environment itself, which only ever runs on vps1.

## Why it's built this way

Forgejo has native branch protection + required-status-checks + auto-merge
(unlike GitHub on a private free-tier repo, where those are paywalled). But
"auto-merge" here still means opting a PR in — either clicking **"Auto merge
when checks succeed"** in the web UI, or merging explicitly once CI is green
via the API (`POST .../pulls/{n}/merge`, `{"Do": "squash"}`). Nothing merges
a ready PR for you automatically just because checks pass, unlike GitHub's
old in-workflow `ci-cd.yml` (retired along with the rest of `.github/`).

**Dev moved off the desktop and onto vps1 so the desktop could retire from the
Thermograph estate entirely.** The desktop now hosts AI models and offers flex
Swarm-worker capacity — it is no longer "the LAN dev server" and no longer the
primary CI runner story. What dev keeps that beta and prod do not:

- It renders `dev.yaml` **alone**, never layering `common.yaml` (the fleet's
  shared production credentials) — vps1 also runs Forgejo and its CI, and dev
  runs whatever branch is in flight. See the long note in
  `deploy/render-secrets.sh` and `deploy/secrets/README.md`.
- It is **mesh-only** — see above. A public VPS running unreviewed branches
  must never be reachable from the open internet the way the old LAN box
  (behind a home router, reachable only on the Wi-Fi) safely could be.

## Moving parts

| File | Purpose |
|------|---------|
| `.forgejo/workflows/pr-build.yml` | required status check for PRs into `dev` (calls `build.yml`) |
| `.forgejo/workflows/deploy.yml` | the single Deploy workflow — `dev` branch pushes SSH into vps1 and run `deploy-dev.sh` |
| `.forgejo/workflows/build.yml` | shared build gate: deps, backend tests, JS syntax check, boot/health check |
| `deploy/env-topology.sh` | source of truth: dev = vps1, `/opt/thermograph-dev`, branch `dev`, compose mode, mesh-only bind address |
| `deploy/deploy-dev.sh` | thin wrapper around `deploy.sh`: dev's compose overlay + secrets policy, then delegates to the shared pull/roll/health-check logic |
| `deploy/secrets/dev.yaml` | dev's SOPS vault — its own secrets and config, rendered to `/etc/thermograph.env` at deploy time (see "Config and secrets") |
| `docker-compose.dev.yml` | dev overlay: no CPU limits, backend published on the mesh address (frontend unpublished) |
| `deploy/provision-dev.sh` | one-time bootstrap of the checkout + vault marker on vps1 |
| `deploy/forgejo/register-lan-runner.sh` | registers the Forgejo Actions runner (name predates the vps1/vps2 split — see the script's own header) |

`deploy.yml`'s dev leg SSHes in with `VPS1_SSH_*` and runs
`/opt/thermograph-dev/infra/deploy/deploy-dev.sh` directly over SSH — the same
shape as the beta and prod legs, just a different host and script. There is no
longer a `thermograph-lan`-labeled runner job doing host-native
`systemctl`/`docker-compose` calls from inside a CI job; the deploy happens the
same way for all three environments now (a `docker`-labeled runner opens an SSH
session to the target host and runs that environment's deploy script there).

`deploy-dev.sh`'s git auth (`GH_TOKEN`/`GITHUB_TOKEN`, Forgejo's per-job
token exposed under that name for compatibility) is scoped to `REPO_URL`'s
own host via a per-command `http.<scheme>://<host>/.extraheader`, not a
hardcoded one — important if the Forgejo host ever changes, since a stale
hardcoded host would make git silently skip the header (no error) and fall
through to whatever ambient credentials happen to be available, not fail
loudly.

## The Forgejo Actions runner

TODO(cutover): `deploy/forgejo/register-lan-runner.sh`'s own header comment
still describes "the desktop" as the canonical placement for this runner, and
its `docker`-labeled jobs are what actually execute `deploy.yml`'s SSH-based
dev/beta/prod deploy steps today. Whether the runner has actually moved to
vps1 (alongside Forgejo) or still runs on the desktop under the new topology
is not settled by anything read for this pass — that script is out of this
document's edit scope. Confirm where it actually lives before relying on the
description below.

A Forgejo Actions **runner** (`forgejo-runner`) serves this repo, labels
`docker` (containerized jobs, node:20-bookworm, with the host's docker.sock
automounted in — `container.docker_host: automount` in
`~/forgejo-runner/config.yaml` — so `docker build`/`push` work without
privileged Docker-in-Docker) and `thermograph-lan` (a legacy label from when a
deploy job ran host-native on the LAN box; nothing in the current `deploy.yml`
schedules against it, since every environment now deploys over SSH from the
`docker`-labeled job instead).

```bash
# status / logs
systemctl --user status forgejo-runner
journalctl --user -u forgejo-runner -f

# re-register if the token/host ever changes
bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
```

Registry pushes (`build-push.yml`) run against the runner host's own
automounted Docker daemon, so it's that **host's** own resolver that needs to
route `dev.jinemi.com` over the mesh, not anything settable from a job
container. If registry pushes ever start failing with an HTTP/TLS mismatch or
a 403 on `/v2/`, check `getent hosts dev.jinemi.com` on the runner host —
it needs to resolve to vps1's WireGuard IP (`10.10.0.2`), typically via a
`/etc/hosts` line, not the public one (a non-issue if the runner is co-located
with Forgejo on vps1 itself, since the name then resolves to itself either
way). Check `git.thermograph.org` the same way: it is still served, and a
rollback to an image tag pushed before the domain migration dials it.

## The dev environment (Docker Compose, on vps1)

Runs as a Docker Compose stack, not a bare systemd unit — `docker ps --filter
name=thermograph-dev` on vps1 shows `thermograph-dev-backend-1`,
`thermograph-dev-frontend-1` (frontend has no published port, reached only
through backend's own reverse-proxy fallback) and `thermograph-dev-db-1`
(TimescaleDB, dev's **own** container — not the shared instance on vps2).

```bash
docker ps --filter name=thermograph-dev                                          # is it up?
docker compose -f docker-compose.yml -f docker-compose.dev.yml logs -f backend frontend   # app logs (from /opt/thermograph-dev/infra)
```

Bootstrap (or rebuild) it:

```bash
sudo bash deploy/provision-dev.sh
```

replacing the retired `deploy/provision-dev-lan.sh`, which bootstrapped dev on
the operator's desktop as a sudo-free `systemd --user` stack. Dev is
provisioned like any other fleet environment now — a checkout at
`/opt/thermograph-dev`, root-owned, deployed over SSH by CI like beta and
prod. The dedicated checkout is separate from your own working tree, so a
deploy never touches uncommitted edits there.

## Config and secrets

Dev's configuration lives in the same SOPS vault as prod's and beta's —
`deploy/secrets/dev.yaml`, encrypted to the same age recipient, rendered to
`/etc/thermograph.env` by the same `render-secrets.sh` at deploy time. Changing a
dev value is `sops deploy/secrets/dev.yaml` → commit → deploy, exactly as for
beta and prod. **`deploy/secrets/README.md` is the reference**; what follows
is only what is specific to this environment.

Dev renders `dev.yaml` **alone**. It does *not* layer `common.yaml`, which is the
fleet's shared production credential set — registry token, both S3 keypairs (one
read-write pair, on the bucket that also holds prod's backups), the VAPID private
key that signs push to real subscribers, the IndexNow key, the metrics token. This
box also runs Forgejo and its CI, and executes unreviewed `dev`-branch code; a
plaintext render of that set into `/etc/thermograph.env` would put all of it in
every dev container's environment. `deploy-dev.sh` exports
`THERMOGRAPH_SECRETS_SKIP_COMMON=1` to prevent it, and **aborts the deploy** if
the renderer in the checkout cannot honour that. This is a property of the
`dev` **environment** (encoded in `deploy/env-topology.sh`'s `TG_SKIP_COMMON`),
not just of `deploy-dev.sh` — so it holds regardless of which entry point is
used to deploy it.

So dev holds its own `THERMOGRAPH_AUTH_SECRET` (its own, not prod's — the `daemon`
service refuses to start without one), its own `POSTGRES_PASSWORD`, a mesh-only
`THERMOGRAPH_BASE_URL`, `THERMOGRAPH_COOKIE_SECURE=0` (plain HTTP — the shared
value is `1` and would silently break login here), and the non-secret config it
would otherwise have inherited. It holds **no** S3 keys, no VAPID keypair, no
IndexNow key, no Discord or mail credentials: the lake falls through to the
Open-Meteo archive without bucket creds, and the app generates and persists its own
VAPID and IndexNow keys into the `appdata` volume.

Two dev-only mechanics carried over from the old LAN setup, both worth keeping
even though dev is no longer sudo-free (vps1's `agent` user has passwordless
sudo like any other fleet host) — the render path just doesn't strictly need
them any more:

- the age key can be read from `~/.config/sops/age/keys.txt` as a fallback
  when `/etc/thermograph/age.key` isn't present or readable — see
  `deploy-dev.sh`'s own comment on this;
- `/etc/thermograph.env` being pre-created and owned by the deploying user
  lets the renderer take its in-place write path without needing `sudo` at
  all, which still matters under the Forgejo runner's `systemd --user`
  context (no tty to answer a `sudo` prompt).

Setting this up on a fresh dev host is a few typed commands — see **"Setting up a
new dev machine"** in `deploy/secrets/README.md` (written for the old desktop
setup; the mechanics are the same on vps1, just root-owned rather than
sudo-free). Until the `/etc/thermograph/secrets-env` marker exists the box
renders nothing and falls back to `deploy-dev.sh`'s built-in
`POSTGRES_PASSWORD` default, which is the safe state and the one to return to
(delete the marker) if the vault ever gets in the way.

`REGISTRY_TOKEN` is deliberately not in the vault: dev pulls with the persistent
`docker login` credential already in this host's docker config, like the prod/beta
CI paths do.

## Before monitoring a PR to land

A PR needs an explicit merge once CI is green — but first confirm it's
actually set up to land, or you'll monitor a PR that can never merge:

1. **Not conflicting** — `GET /api/v1/repos/{owner}/{repo}/pulls/{n}` should
   show `"mergeable": true`. If not, merge `forgejo/dev` in and resolve now —
   don't monitor a PR that can't merge.
2. **Required check actually configured** — `GET
   /api/v1/repos/{owner}/{repo}/branch_protections/dev` should list
   `status_check_contexts` containing the *exact* string Forgejo reports for
   that PR's check (workflow display name + job name + event, e.g. `"PR build
   (required check) / build (pull_request)"` — not just the job name
   `"build"`). A mismatch here means no PR can ever satisfy the check even
   with fully green CI; this bit us once already.
3. **CI actually running** — `GET
   /api/v1/repos/{owner}/{repo}/commits/{sha}/status` shows a status, not an
   empty list. If empty, check the runner is up
   (`systemctl --user status forgejo-runner`).

Only start monitoring once all three pass. Once CI is green and mergeable is
true, merge explicitly: `POST /api/v1/repos/{owner}/{repo}/pulls/{n}/merge`
with `{"Do": "squash", "delete_branch_after_merge": true}`.

## Day-to-day

- **Ship:** open a PR into `dev`, confirm CI is green, merge it explicitly —
  the merge triggers a deploy to vps1.
- **Manual redeploy:** trigger the `Deploy` workflow via `workflow_dispatch`
  (`POST /api/v1/repos/{owner}/{repo}/actions/workflows/deploy.yml/dispatches`,
  `{"ref": "dev"}`), or directly on vps1:
  `APP_DIR=/opt/thermograph-dev bash deploy/deploy-dev.sh`.
- **Promote to beta:** merge/fast-forward `dev` into `main` and push — that
  triggers the beta leg of the same `Deploy` workflow, per `DEPLOY.md`.

## Monitoring

Logs and dashboards live in a **separate project** — a central Grafana + Loki
stack, hosted on **vps1 itself**, fed by a Grafana Alloy agent on every node
(prod and beta on vps2, dev and the desktop's own agent, all over the
WireGuard mesh; `thermograph-observability` on Forgejo; UI at
`https://dashboard.thermograph.org`). Dev's Alloy agent ships its
container/app logs under its own `host`/service labels, so dev shows up
alongside prod and beta in the same dashboards. This replaced the old
`scripts/dashboard.py`.

For a quick terminal check without Grafana, the app still serves raw counters at
the gated `GET /api/v2/metrics` route. Dev sets no `THERMOGRAPH_METRICS_TOKEN`, so
the route is direct-loopback-only — `curl localhost:8137/api/v2/metrics` from vps1
itself. (A token exists on prod/beta to allow reads through an SSH tunnel; dev has
no reason to carry a copy, and the estate's copy must not land here — see
`deploy/secrets/README.md`.)