172 lines
8.7 KiB
Markdown
172 lines
8.7 KiB
Markdown
# Dev CI/CD → LAN server on this machine
|
|
|
|
Parallel to the prod pipeline (`main` → VPS, see `DEPLOY.md`), the **`dev`**
|
|
branch continuously deploys to a **LAN server running on this computer**. Git
|
|
hosting and CI are self-hosted **Forgejo** (`git.thermograph.org`, reachable
|
|
at `http://10.10.0.2:3080` over the WireGuard mesh) — GitHub is retired.
|
|
|
|
```
|
|
open PR ──▶ CI (build + boot/health, on the LAN Forgejo runner)
|
|
│ green
|
|
▼
|
|
merge into dev (explicit — Forgejo does not auto-merge on its own;
|
|
see "Landing a PR" below)
|
|
│
|
|
▼
|
|
LAN runner (thermograph-lan label) runs deploy/deploy-dev.sh
|
|
│
|
|
▼
|
|
~/thermograph-dev updated ─▶ docker compose stack (backend + frontend +
|
|
Postgres 18) brought up, backend on 0.0.0.0:8137
|
|
```
|
|
|
|
Reachable at `http://<lan-ip>:8137/` from any device on your Wi-Fi.
|
|
|
|
The dev server runs the **same containerized stack as prod** (`docker-compose.yml`),
|
|
overlaid with `docker-compose.dev.yml`: **uncapped** (no CPU limits, unlike prod's
|
|
backend=4 / frontend=2 / db=2) and backend published on `0.0.0.0:8137` for the
|
|
LAN (prod binds loopback behind Caddy; frontend has no published port here
|
|
either way, no Caddy to reach it directly, so it's only reached through
|
|
backend's own reverse-proxy fallback). Bring it up by hand with `make dev-up`.
|
|
|
|
## Why it's built this way
|
|
|
|
Forgejo has native branch protection + required-status-checks + auto-merge
|
|
(unlike GitHub on a private free-tier repo, where those are paywalled). But
|
|
"auto-merge" here still means opting a PR in — either clicking **"Auto merge
|
|
when checks succeed"** in the web UI, or merging explicitly once CI is green
|
|
via the API (`POST .../pulls/{n}/merge`, `{"Do": "squash"}`). Nothing merges
|
|
a ready PR for you automatically just because checks pass, unlike GitHub's
|
|
old in-workflow `ci-cd.yml` (retired along with the rest of `.github/`).
|
|
|
|
## Moving parts
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `.forgejo/workflows/pr-build.yml` | required status check for PRs into `dev` (calls `build.yml`) |
|
|
| `.forgejo/workflows/deploy-dev.yml` | build + deploy on pushes to `dev` (direct, or a PR merge) |
|
|
| `.forgejo/workflows/build.yml` | shared build gate: deps, backend tests, JS syntax check, boot/health check |
|
|
| `deploy/deploy-dev.sh` | pull `dev` into `~/thermograph-dev`, `docker compose up` the uncapped LAN stack, health check |
|
|
| `docker-compose.dev.yml` | dev overlay: no CPU limits, backend published on `0.0.0.0:8137` (frontend unpublished) |
|
|
| `deploy/provision-dev-lan.sh` | one-time sudo-free bootstrap of the checkout + runner |
|
|
| `deploy/forgejo/register-lan-runner.sh` | registers the Forgejo Actions runner on this machine |
|
|
|
|
`deploy-dev.yml`'s `deploy` job runs on the `thermograph-lan` label — **not**
|
|
`[self-hosted, thermograph-lan]` (the array form is what the original GitHub
|
|
version used; GitHub implicitly tags every self-hosted runner `self-hosted`,
|
|
but Forgejo's runner has only the labels it was explicitly registered with,
|
|
so the array form is permanently unschedulable there — a real bug found and
|
|
fixed once, worth knowing about if a "deploy" job ever silently stops
|
|
appearing in the Actions history again).
|
|
|
|
`deploy-dev.sh`'s git auth (`GH_TOKEN`/`GITHUB_TOKEN`, Forgejo's per-job
|
|
token exposed under that name for compatibility) is scoped to `REPO_URL`'s
|
|
own host via a per-command `http.<scheme>://<host>/.extraheader`, not a
|
|
hardcoded one — important if the Forgejo host ever changes, since a stale
|
|
hardcoded host would make git silently skip the header (no error) and fall
|
|
through to whatever ambient credentials happen to be available, not fail
|
|
loudly.
|
|
|
|
## The self-hosted runner
|
|
|
|
A Forgejo Actions **runner** (`forgejo-runner`) runs on this machine, labels
|
|
`docker` (containerized jobs, node:20-bookworm, with the host's docker.sock
|
|
automounted in — `container.docker_host: automount` in
|
|
`~/forgejo-runner/config.yaml` — so `docker build`/`push` work without
|
|
privileged Docker-in-Docker) and `thermograph-lan` (host-native jobs, for
|
|
`deploy-dev.sh`'s systemctl/docker-compose calls). Installed under
|
|
`~/forgejo-runner`, runs as the `forgejo-runner` systemd `--user` service.
|
|
|
|
```bash
|
|
# status / logs
|
|
systemctl --user status forgejo-runner
|
|
journalctl --user -u forgejo-runner -f
|
|
|
|
# re-register if the token/host ever changes
|
|
bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
|
|
```
|
|
|
|
`git.thermograph.org`'s public DNS resolves to beta's **public** IP, which
|
|
the registry's mesh-only ACL (see `deploy/forgejo/README.md`) rejects for
|
|
`/v2/` paths — `docker build-push.yml` pushes run against the *host's*
|
|
automounted daemon, so it's this **host's** own resolver that needs to route
|
|
git.thermograph.org over the mesh, not anything settable from a job
|
|
container. If registry pushes ever start failing with an HTTP/TLS mismatch
|
|
or a 403 on `/v2/`, check `getent hosts git.thermograph.org` here — it needs
|
|
to resolve to beta's WireGuard IP (`10.10.0.2`), typically via a `/etc/hosts`
|
|
line, not the public one.
|
|
|
|
## The LAN dev service (Docker Compose)
|
|
|
|
Runs as a Docker Compose stack, not a bare systemd unit — `docker ps --filter
|
|
name=thermograph-dev` shows `thermograph-dev-backend-1`, `thermograph-dev-frontend-1`
|
|
(repo-split Stage 4 split the single "app" container in two — frontend has no
|
|
published port, reached only through backend's own reverse-proxy fallback) and
|
|
`thermograph-dev-db-1` (TimescaleDB). A legacy pre-container
|
|
`thermograph-dev.service` systemd --user unit may still exist from before this
|
|
cutover; it's stale and `deploy-dev.sh` stops/disables it on every run — don't
|
|
trust `systemctl --user status thermograph-dev` as a liveness check, use `docker
|
|
ps` instead.
|
|
|
|
```bash
|
|
docker ps --filter name=thermograph-dev # is it up?
|
|
docker compose -f docker-compose.yml -f docker-compose.dev.yml logs -f backend frontend # app logs (from ~/thermograph-dev)
|
|
```
|
|
|
|
Bootstrap (or rebuild) it by hand:
|
|
|
|
```bash
|
|
bash deploy/provision-dev-lan.sh
|
|
```
|
|
|
|
Config: port/base path are baked into the unit from `deploy/deploy-dev.sh`
|
|
(`PORT`, `THERMOGRAPH_BASE`). Change them there and redeploy. The dedicated
|
|
checkout at `~/thermograph-dev` is separate from your working tree, so a deploy
|
|
never touches uncommitted edits in `~/Code/Thermograph`.
|
|
|
|
## Before monitoring a PR to land
|
|
|
|
A PR needs an explicit merge once CI is green — but first confirm it's
|
|
actually set up to land, or you'll monitor a PR that can never merge:
|
|
|
|
1. **Not conflicting** — `GET /api/v1/repos/{owner}/{repo}/pulls/{n}` should
|
|
show `"mergeable": true`. If not, merge `forgejo/dev` in and resolve now —
|
|
don't monitor a PR that can't merge.
|
|
2. **Required check actually configured** — `GET
|
|
/api/v1/repos/{owner}/{repo}/branch_protections/dev` should list
|
|
`status_check_contexts` containing the *exact* string Forgejo reports for
|
|
that PR's check (workflow display name + job name + event, e.g. `"PR build
|
|
(required check) / build (pull_request)"` — not just the job name
|
|
`"build"`). A mismatch here means no PR can ever satisfy the check even
|
|
with fully green CI; this bit us once already.
|
|
3. **CI actually running** — `GET
|
|
/api/v1/repos/{owner}/{repo}/commits/{sha}/status` shows a status, not an
|
|
empty list. If empty, check the runner is up
|
|
(`systemctl --user status forgejo-runner`).
|
|
|
|
Only start monitoring once all three pass. Once CI is green and mergeable is
|
|
true, merge explicitly: `POST /api/v1/repos/{owner}/{repo}/pulls/{n}/merge`
|
|
with `{"Do": "squash", "delete_branch_after_merge": true}`.
|
|
|
|
## Day-to-day
|
|
|
|
- **Ship:** open a PR into `dev`, confirm CI is green, merge it explicitly —
|
|
the merge triggers the deploy here.
|
|
- **Manual redeploy:** trigger `deploy-dev.yml` via `workflow_dispatch`
|
|
(`POST /api/v1/repos/{owner}/{repo}/actions/workflows/deploy-dev.yml/dispatches`,
|
|
`{"ref": "dev"}`), or locally `APP_DIR=~/thermograph-dev bash deploy/deploy-dev.sh`.
|
|
- **Promote to prod:** merge/fast-forward `dev` into `main` and push — that
|
|
triggers the VPS pipeline in `DEPLOY.md`.
|
|
|
|
## Monitoring
|
|
|
|
Logs and dashboards moved to a **separate project** — a central Grafana + Loki
|
|
stack fed by a Grafana Alloy agent on every node (including this LAN dev box),
|
|
over the WireGuard mesh (`thermograph-observability` on Forgejo; UI at
|
|
`https://dashboard.thermograph.org`). The dev node's Alloy agent ships its
|
|
container/app logs under `host="dev"`, so LAN dev shows up alongside prod and
|
|
beta in the same dashboards. This replaced the old `scripts/dashboard.py`.
|
|
|
|
For a quick terminal check without Grafana, the app still serves raw counters at
|
|
the gated `GET /api/v2/metrics` route (loopback-only; set
|
|
`THERMOGRAPH_METRICS_TOKEN` to read it over an SSH tunnel).
|