thermograph/infra/DEPLOY-DEV.md

Dev CI/CD → the dev environment on vps1

Parallel to beta/prod's pipeline (main→beta, release→prod, see DEPLOY.md), the dev branch continuously deploys to the dev environment on vps1 (75.119.132.91, mesh 10.10.0.2) — /opt/thermograph-dev, a normal fleet checkout reached over SSH exactly like beta and prod, not a stack on the operator's desktop or LAN any more. Git hosting and CI are self-hosted Forgejo (git.thermograph.org, also on vps1, reachable at http://10.10.0.2:3080 over the WireGuard mesh) — GitHub is retired.

open PR ──▶ CI (build + boot/health, on the `docker`-labeled Forgejo runner)
              │ green
              ▼
        merge into dev (explicit — Forgejo does not auto-merge on its own;
                         see "Landing a PR" below)
              │
              ▼
        the Deploy workflow SSHes into vps1 (VPS1_SSH_*, THERMOGRAPH_ENV=dev)
        and runs deploy/deploy-dev.sh
              │
              ▼
        /opt/thermograph-dev updated ─▶ docker compose stack (backend + frontend +
                                         Postgres) brought up, backend published
                                         MESH-ONLY on 10.10.0.2:8137

Reachable at http://10.10.0.2:8137/ from anything already on the WireGuard mesh, and from nowhere else — no public DNS record, no Caddy site, no TLS. vps1 is a public VPS running whatever branch is currently in flight (including unreviewed code), so unlike the old LAN box it must never publish on 0.0.0.0.

The dev stack runs the same containerized stack as prod (docker-compose.yml), overlaid with docker-compose.dev.yml: uncapped (no CPU limits, unlike prod's Swarm-managed replicas) and backend published on the WireGuard address (10.10.0.2:8137 — set via DEV_BIND_ADDR in deploy/env-topology.sh; prod and beta bind loopback behind their own Caddy LBs instead). Frontend has no published port either way — reached only through backend's own reverse-proxy fallback. Bring dev up by hand with make dev-up, which is a laptop-local convenience (uncapped, loopback-published) — not the dev environment itself, which only ever runs on vps1.

Why it's built this way

Forgejo has native branch protection + required-status-checks + auto-merge (unlike GitHub on a private free-tier repo, where those are paywalled). But "auto-merge" here still means opting a PR in — either clicking "Auto merge when checks succeed" in the web UI, or merging explicitly once CI is green via the API (POST .../pulls/{n}/merge, {"Do": "squash"}). Nothing merges a ready PR for you automatically just because checks pass, unlike GitHub's old in-workflow ci-cd.yml (retired along with the rest of .github/).

Dev moved off the desktop and onto vps1 so the desktop could retire from the Thermograph estate entirely. The desktop now hosts AI models and offers flex Swarm-worker capacity — it is no longer "the LAN dev server" and no longer the primary CI runner story. What dev keeps that beta and prod do not:

• It renders dev.yaml alone, never layering common.yaml (the fleet's shared production credentials) — vps1 also runs Forgejo and its CI, and dev runs whatever branch is in flight. See the long note in deploy/render-secrets.sh and deploy/secrets/README.md.
• It is mesh-only — see above. A public VPS running unreviewed branches must never be reachable from the open internet the way the old LAN box (behind a home router, reachable only on the Wi-Fi) safely could be.

Moving parts

File	Purpose
.forgejo/workflows/pr-build.yml	required status check for PRs into dev (calls build.yml)
.forgejo/workflows/deploy.yml	the single Deploy workflow — dev branch pushes SSH into vps1 and run deploy-dev.sh
.forgejo/workflows/build.yml	shared build gate: deps, backend tests, JS syntax check, boot/health check
deploy/env-topology.sh	source of truth: dev = vps1, /opt/thermograph-dev, branch dev, compose mode, mesh-only bind address
deploy/deploy-dev.sh	thin wrapper around deploy.sh: dev's compose overlay + secrets policy, then delegates to the shared pull/roll/health-check logic
deploy/secrets/dev.yaml	dev's SOPS vault — its own secrets and config, rendered to /etc/thermograph.env at deploy time (see "Config and secrets")
docker-compose.dev.yml	dev overlay: no CPU limits, backend published on the mesh address (frontend unpublished)
deploy/provision-dev.sh	one-time bootstrap of the checkout + vault marker on vps1
deploy/forgejo/register-lan-runner.sh	registers the Forgejo Actions runner (name predates the vps1/vps2 split — see the script's own header)

deploy.yml's dev leg SSHes in with VPS1_SSH_* and runs /opt/thermograph-dev/infra/deploy/deploy-dev.sh directly over SSH — the same shape as the beta and prod legs, just a different host and script. There is no longer a thermograph-lan-labeled runner job doing host-native systemctl/docker-compose calls from inside a CI job; the deploy happens the same way for all three environments now (a docker-labeled runner opens an SSH session to the target host and runs that environment's deploy script there).

deploy-dev.sh's git auth (GH_TOKEN/GITHUB_TOKEN, Forgejo's per-job token exposed under that name for compatibility) is scoped to REPO_URL's own host via a per-command http.<scheme>://<host>/.extraheader, not a hardcoded one — important if the Forgejo host ever changes, since a stale hardcoded host would make git silently skip the header (no error) and fall through to whatever ambient credentials happen to be available, not fail loudly.

The Forgejo Actions runner

TODO(cutover): deploy/forgejo/register-lan-runner.sh's own header comment still describes "the desktop" as the canonical placement for this runner, and its docker-labeled jobs are what actually execute deploy.yml's SSH-based dev/beta/prod deploy steps today. Whether the runner has actually moved to vps1 (alongside Forgejo) or still runs on the desktop under the new topology is not settled by anything read for this pass — that script is out of this document's edit scope. Confirm where it actually lives before relying on the description below.

A Forgejo Actions runner (forgejo-runner) serves this repo, labels docker (containerized jobs, node:20-bookworm, with the host's docker.sock automounted in — container.docker_host: automount in ~/forgejo-runner/config.yaml — so docker build/push work without privileged Docker-in-Docker) and thermograph-lan (a legacy label from when a deploy job ran host-native on the LAN box; nothing in the current deploy.yml schedules against it, since every environment now deploys over SSH from the docker-labeled job instead).

# status / logs
systemctl --user status forgejo-runner
journalctl --user -u forgejo-runner -f

# re-register if the token/host ever changes
bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>

Registry pushes (build-push.yml) run against the runner host's own automounted Docker daemon, so it's that host's own resolver that needs to route git.thermograph.org over the mesh, not anything settable from a job container. If registry pushes ever start failing with an HTTP/TLS mismatch or a 403 on /v2/, check getent hosts git.thermograph.org on the runner host — it needs to resolve to vps1's WireGuard IP (10.10.0.2), typically via a /etc/hosts line, not the public one (a non-issue if the runner is co-located with Forgejo on vps1 itself, since git.thermograph.org then resolves to itself either way).

The dev environment (Docker Compose, on vps1)

Runs as a Docker Compose stack, not a bare systemd unit — docker ps --filter name=thermograph-dev on vps1 shows thermograph-dev-backend-1, thermograph-dev-frontend-1 (frontend has no published port, reached only through backend's own reverse-proxy fallback) and thermograph-dev-db-1 (TimescaleDB, dev's own container — not the shared instance on vps2).

docker ps --filter name=thermograph-dev                                          # is it up?
docker compose -f docker-compose.yml -f docker-compose.dev.yml logs -f backend frontend   # app logs (from /opt/thermograph-dev/infra)

Bootstrap (or rebuild) it:

sudo bash deploy/provision-dev.sh

replacing the retired deploy/provision-dev-lan.sh, which bootstrapped dev on the operator's desktop as a sudo-free systemd --user stack. Dev is provisioned like any other fleet environment now — a checkout at /opt/thermograph-dev, root-owned, deployed over SSH by CI like beta and prod. The dedicated checkout is separate from your own working tree, so a deploy never touches uncommitted edits there.

Config and secrets

Dev's configuration lives in the same SOPS vault as prod's and beta's — deploy/secrets/dev.yaml, encrypted to the same age recipient, rendered to /etc/thermograph.env by the same render-secrets.sh at deploy time. Changing a dev value is sops deploy/secrets/dev.yaml → commit → deploy, exactly as for beta and prod. deploy/secrets/README.md is the reference; what follows is only what is specific to this environment.

Dev renders dev.yaml alone. It does not layer common.yaml, which is the fleet's shared production credential set — registry token, both S3 keypairs (one read-write pair, on the bucket that also holds prod's backups), the VAPID private key that signs push to real subscribers, the IndexNow key, the metrics token. This box also runs Forgejo and its CI, and executes unreviewed dev-branch code; a plaintext render of that set into /etc/thermograph.env would put all of it in every dev container's environment. deploy-dev.sh exports THERMOGRAPH_SECRETS_SKIP_COMMON=1 to prevent it, and aborts the deploy if the renderer in the checkout cannot honour that. This is a property of the dev environment (encoded in deploy/env-topology.sh's TG_SKIP_COMMON), not just of deploy-dev.sh — so it holds regardless of which entry point is used to deploy it.

So dev holds its own THERMOGRAPH_AUTH_SECRET (its own, not prod's — the daemon service refuses to start without one), its own POSTGRES_PASSWORD, a mesh-only THERMOGRAPH_BASE_URL, THERMOGRAPH_COOKIE_SECURE=0 (plain HTTP — the shared value is 1 and would silently break login here), and the non-secret config it would otherwise have inherited. It holds no S3 keys, no VAPID keypair, no IndexNow key, no Discord or mail credentials: the lake falls through to the Open-Meteo archive without bucket creds, and the app generates and persists its own VAPID and IndexNow keys into the appdata volume.

Two dev-only mechanics carried over from the old LAN setup, both worth keeping even though dev is no longer sudo-free (vps1's agent user has passwordless sudo like any other fleet host) — the render path just doesn't strictly need them any more:

• the age key can be read from ~/.config/sops/age/keys.txt as a fallback when /etc/thermograph/age.key isn't present or readable — see deploy-dev.sh's own comment on this;
• /etc/thermograph.env being pre-created and owned by the deploying user lets the renderer take its in-place write path without needing sudo at all, which still matters under the Forgejo runner's systemd --user context (no tty to answer a sudo prompt).

Setting this up on a fresh dev host is a few typed commands — see "Setting up a new dev machine" in deploy/secrets/README.md (written for the old desktop setup; the mechanics are the same on vps1, just root-owned rather than sudo-free). Until the /etc/thermograph/secrets-env marker exists the box renders nothing and falls back to deploy-dev.sh's built-in POSTGRES_PASSWORD default, which is the safe state and the one to return to (delete the marker) if the vault ever gets in the way.

REGISTRY_TOKEN is deliberately not in the vault: dev pulls with the persistent docker login credential already in this host's docker config, like the prod/beta CI paths do.

Before monitoring a PR to land

A PR needs an explicit merge once CI is green — but first confirm it's actually set up to land, or you'll monitor a PR that can never merge:

1. Not conflicting — GET /api/v1/repos/{owner}/{repo}/pulls/{n} should show "mergeable": true. If not, merge forgejo/dev in and resolve now — don't monitor a PR that can't merge.
2. Required check actually configured — GET /api/v1/repos/{owner}/{repo}/branch_protections/dev should list status_check_contexts containing the exact string Forgejo reports for that PR's check (workflow display name + job name + event, e.g. "PR build (required check) / build (pull_request)" — not just the job name "build"). A mismatch here means no PR can ever satisfy the check even with fully green CI; this bit us once already.
3. CI actually running — GET /api/v1/repos/{owner}/{repo}/commits/{sha}/status shows a status, not an empty list. If empty, check the runner is up (systemctl --user status forgejo-runner).

Only start monitoring once all three pass. Once CI is green and mergeable is true, merge explicitly: POST /api/v1/repos/{owner}/{repo}/pulls/{n}/merge with {"Do": "squash", "delete_branch_after_merge": true}.

Day-to-day

• Ship: open a PR into dev, confirm CI is green, merge it explicitly — the merge triggers a deploy to vps1.
• Manual redeploy: trigger the Deploy workflow via workflow_dispatch (POST /api/v1/repos/{owner}/{repo}/actions/workflows/deploy.yml/dispatches, {"ref": "dev"}), or directly on vps1: APP_DIR=/opt/thermograph-dev bash deploy/deploy-dev.sh.
• Promote to beta: merge/fast-forward dev into main and push — that triggers the beta leg of the same Deploy workflow, per DEPLOY.md.

Monitoring

Logs and dashboards live in a separate project — a central Grafana + Loki stack, hosted on vps1 itself, fed by a Grafana Alloy agent on every node (prod and beta on vps2, dev and the desktop's own agent, all over the WireGuard mesh; thermograph-observability on Forgejo; UI at https://dashboard.thermograph.org). Dev's Alloy agent ships its container/app logs under its own host/service labels, so dev shows up alongside prod and beta in the same dashboards. This replaced the old scripts/dashboard.py.

For a quick terminal check without Grafana, the app still serves raw counters at the gated GET /api/v2/metrics route. Dev sets no THERMOGRAPH_METRICS_TOKEN, so the route is direct-loopback-only — curl localhost:8137/api/v2/metrics from vps1 itself. (A token exists on prod/beta to allow reads through an SSH tunnel; dev has no reason to carry a copy, and the estate's copy must not land here — see deploy/secrets/README.md.)