thermograph/infra/DEPLOY-DEV.md

Dev CI/CD → LAN server on this machine

Parallel to the prod pipeline (main → VPS, see DEPLOY.md), the dev branch continuously deploys to a LAN server running on this computer. Git hosting and CI are self-hosted Forgejo (git.thermograph.org, reachable at http://10.10.0.2:3080 over the WireGuard mesh) — GitHub is retired.

open PR ──▶ CI (build + boot/health, on the LAN Forgejo runner)
              │ green
              ▼
        merge into dev (explicit — Forgejo does not auto-merge on its own;
                         see "Landing a PR" below)
              │
              ▼
        LAN runner (thermograph-lan label) runs deploy/deploy-dev.sh
              │
              ▼
        ~/thermograph-dev updated ─▶ docker compose stack (backend + frontend +
                                     Postgres 18) brought up, backend on 0.0.0.0:8137

Reachable at http://<lan-ip>:8137/ from any device on your Wi-Fi.

The dev server runs the same containerized stack as prod (docker-compose.yml), overlaid with docker-compose.dev.yml: uncapped (no CPU limits, unlike prod's backend=4 / frontend=2 / db=2) and backend published on 0.0.0.0:8137 for the LAN (prod binds loopback behind Caddy; frontend has no published port here either way, no Caddy to reach it directly, so it's only reached through backend's own reverse-proxy fallback). Bring it up by hand with make dev-up.

Why it's built this way

Forgejo has native branch protection + required-status-checks + auto-merge (unlike GitHub on a private free-tier repo, where those are paywalled). But "auto-merge" here still means opting a PR in — either clicking "Auto merge when checks succeed" in the web UI, or merging explicitly once CI is green via the API (POST .../pulls/{n}/merge, {"Do": "squash"}). Nothing merges a ready PR for you automatically just because checks pass, unlike GitHub's old in-workflow ci-cd.yml (retired along with the rest of .github/).

Moving parts

File	Purpose
.forgejo/workflows/pr-build.yml	required status check for PRs into dev (calls build.yml)
.forgejo/workflows/deploy-dev.yml	build + deploy on pushes to dev (direct, or a PR merge)
.forgejo/workflows/build.yml	shared build gate: deps, backend tests, JS syntax check, boot/health check
deploy/deploy-dev.sh	pull dev into ~/thermograph-dev, docker compose up the uncapped LAN stack, health check
docker-compose.dev.yml	dev overlay: no CPU limits, backend published on 0.0.0.0:8137 (frontend unpublished)
deploy/provision-dev-lan.sh	one-time sudo-free bootstrap of the checkout + runner
deploy/forgejo/register-lan-runner.sh	registers the Forgejo Actions runner on this machine

deploy-dev.yml's deploy job runs on the thermograph-lan label — not [self-hosted, thermograph-lan] (the array form is what the original GitHub version used; GitHub implicitly tags every self-hosted runner self-hosted, but Forgejo's runner has only the labels it was explicitly registered with, so the array form is permanently unschedulable there — a real bug found and fixed once, worth knowing about if a "deploy" job ever silently stops appearing in the Actions history again).

deploy-dev.sh's git auth (GH_TOKEN/GITHUB_TOKEN, Forgejo's per-job token exposed under that name for compatibility) is scoped to REPO_URL's own host via a per-command http.<scheme>://<host>/.extraheader, not a hardcoded one — important if the Forgejo host ever changes, since a stale hardcoded host would make git silently skip the header (no error) and fall through to whatever ambient credentials happen to be available, not fail loudly.

The self-hosted runner

A Forgejo Actions runner (forgejo-runner) runs on this machine, labels docker (containerized jobs, node:20-bookworm, with the host's docker.sock automounted in — container.docker_host: automount in ~/forgejo-runner/config.yaml — so docker build/push work without privileged Docker-in-Docker) and thermograph-lan (host-native jobs, for deploy-dev.sh's systemctl/docker-compose calls). Installed under ~/forgejo-runner, runs as the forgejo-runner systemd --user service.

# status / logs
systemctl --user status forgejo-runner
journalctl --user -u forgejo-runner -f

# re-register if the token/host ever changes
bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>

git.thermograph.org's public DNS resolves to beta's public IP, which the registry's mesh-only ACL (see deploy/forgejo/README.md) rejects for /v2/ paths — docker build-push.yml pushes run against the host's automounted daemon, so it's this host's own resolver that needs to route git.thermograph.org over the mesh, not anything settable from a job container. If registry pushes ever start failing with an HTTP/TLS mismatch or a 403 on /v2/, check getent hosts git.thermograph.org here — it needs to resolve to beta's WireGuard IP (10.10.0.2), typically via a /etc/hosts line, not the public one.

The LAN dev service (Docker Compose)

Runs as a Docker Compose stack, not a bare systemd unit — docker ps --filter name=thermograph-dev shows thermograph-dev-backend-1, thermograph-dev-frontend-1 (repo-split Stage 4 split the single "app" container in two — frontend has no published port, reached only through backend's own reverse-proxy fallback) and thermograph-dev-db-1 (TimescaleDB). A legacy pre-container thermograph-dev.service systemd --user unit may still exist from before this cutover; it's stale and deploy-dev.sh stops/disables it on every run — don't trust systemctl --user status thermograph-dev as a liveness check, use docker ps instead.

docker ps --filter name=thermograph-dev                                          # is it up?
docker compose -f docker-compose.yml -f docker-compose.dev.yml logs -f backend frontend   # app logs (from ~/thermograph-dev)

Bootstrap (or rebuild) it by hand:

bash deploy/provision-dev-lan.sh

Config: port/base path are baked into the unit from deploy/deploy-dev.sh (PORT, THERMOGRAPH_BASE). Change them there and redeploy. The dedicated checkout at ~/thermograph-dev is separate from your working tree, so a deploy never touches uncommitted edits in ~/Code/Thermograph.

Before monitoring a PR to land

A PR needs an explicit merge once CI is green — but first confirm it's actually set up to land, or you'll monitor a PR that can never merge:

1. Not conflicting — GET /api/v1/repos/{owner}/{repo}/pulls/{n} should show "mergeable": true. If not, merge forgejo/dev in and resolve now — don't monitor a PR that can't merge.
2. Required check actually configured — GET /api/v1/repos/{owner}/{repo}/branch_protections/dev should list status_check_contexts containing the exact string Forgejo reports for that PR's check (workflow display name + job name + event, e.g. "PR build (required check) / build (pull_request)" — not just the job name "build"). A mismatch here means no PR can ever satisfy the check even with fully green CI; this bit us once already.
3. CI actually running — GET /api/v1/repos/{owner}/{repo}/commits/{sha}/status shows a status, not an empty list. If empty, check the runner is up (systemctl --user status forgejo-runner).

Only start monitoring once all three pass. Once CI is green and mergeable is true, merge explicitly: POST /api/v1/repos/{owner}/{repo}/pulls/{n}/merge with {"Do": "squash", "delete_branch_after_merge": true}.

Day-to-day

• Ship: open a PR into dev, confirm CI is green, merge it explicitly — the merge triggers the deploy here.
• Manual redeploy: trigger deploy-dev.yml via workflow_dispatch (POST /api/v1/repos/{owner}/{repo}/actions/workflows/deploy-dev.yml/dispatches, {"ref": "dev"}), or locally APP_DIR=~/thermograph-dev bash deploy/deploy-dev.sh.
• Promote to prod: merge/fast-forward dev into main and push — that triggers the VPS pipeline in DEPLOY.md.

Monitoring

Logs and dashboards moved to a separate project — a central Grafana + Loki stack fed by a Grafana Alloy agent on every node (including this LAN dev box), over the WireGuard mesh (thermograph-observability on Forgejo; UI at https://dashboard.thermograph.org). The dev node's Alloy agent ships its container/app logs under host="dev", so LAN dev shows up alongside prod and beta in the same dashboards. This replaced the old scripts/dashboard.py.

For a quick terminal check without Grafana, the app still serves raw counters at the gated GET /api/v2/metrics route (loopback-only; set THERMOGRAPH_METRICS_TOKEN to read it over an SSH tunnel).