infra/forgejo: add resource limits and a leaner CI job image (#38)

infra/deploy/forgejo/README.md

@@ -41,6 +41,15 @@ Do this **before** the DNS + Caddy step below — Caddy's reverse_proxy target
 (`127.0.0.1:3080`) needs the `forgejo` service actually listening first, or
 its first health check just fails harmlessly until it is.
 
+This stack has **no auto-deploy trigger** — nothing in `.forgejo/workflows/`
+redeploys it on push. A change to `docker-stack.yml` only takes effect once
+someone re-runs `docker stack deploy` by hand on the manager (prod).
+
+`db`/`forgejo` both carry `resources.limits` (defaults: db 1 CPU/1g, forgejo 2
+CPU/2g — several times observed steady-state usage), overridable with
+`FORGEJO_DB_CPUS`/`FORGEJO_DB_MEMORY`/`FORGEJO_CPUS`/`FORGEJO_MEMORY` env vars
+before `docker stack deploy`, same convention as the app stack.
+
 ## DNS + TLS: reusing beta's existing Caddy, not a second reverse proxy
 
 Forgejo is pinned to beta (`role=forge`) — but beta is also **today's live
@@ -98,6 +107,43 @@ See that script's header for exactly what it replaces (the pre-Forgejo GitHub
 self-hosted runner on this same machine) and why it registers with two
 labels where there used to be two separate runners.
 
+## Custom CI job image (`ci-runner/`)
+
+`ci-runner/Dockerfile` still bases on `node:20-bookworm` — Node is a hard
+requirement, not leftover: Forgejo's runner executes `actions/checkout@v4`
+(used by every workflow) as `node dist/index.js` *inside the job container*,
+regardless of whether the workflow itself uses npm/node. (v1 of this image
+tried a Node-free Debian-slim base and broke every job's checkout step —
+`node: executable file not found` — within a minute of going live; reverted
+immediately.) What it actually fixes: every `docker`-labeled build-push job
+currently re-installs the Docker CLI on each run (`apt-get install
+docker.io`), which pulls in the classic builder rather than BuildKit (the
+classic builder mishandles `COPY --chown=<name>` group resolution — a real
+bug hit during the frontend Go rewrite). `ci-runner` adds `docker-ce-cli` +
+`docker-buildx-plugin` (BuildKit) on top of the same Node base, plus
+`git`/`python3`/`python3-yaml` for the other jobs that need them
+(`shell-lint`, `observability-validate`).
+
+Current tag: `git.thermograph.org/emi/thermograph/ci-runner:v2` (`v1` is
+broken — do not register any runner against it). Rebuild/push (requires a PAT
+with `write:package` scope — the embedded git-remote token lacks it, same
+requirement documented in `backend-build-push.yml`):
+
+```bash
+docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN deploy/forgejo/ci-runner
+docker push git.thermograph.org/emi/thermograph/ci-runner:vN
+```
+
+`register-lan-runner.sh`'s `LABELS` default points at the current tag, so
+fresh registrations pick it up automatically. The live runner is cut over by
+editing the `labels` array in `~/forgejo-runner/.runner` on the desktop (same
+runner id/token, no re-registration needed) and restarting the service —
+**verify a real job runs green under the new image before relying on it**,
+same way v1's break was caught. Only after that verification should the
+now-redundant `apt-get install docker.io` / `python3-yaml` steps be removed
+from the workflows that had them — removing them first would break every job
+still running on the stock `node:20-bookworm` image.
+
 ## Why Postgres here and not the Thermograph app's TimescaleDB
 
 Separate instance, separate network (`forgejo_net`, not the app's compose

infra/deploy/forgejo/ci-runner/Dockerfile

@@ -0,0 +1,46 @@
+# Job-container image for the LAN Forgejo Actions runner's `docker`-labeled
+# jobs (see ../register-lan-runner.sh) — used by every backend/frontend
+# build-push, deploy, and validation workflow that needs `docker build`.
+#
+# Base stays node:20-bookworm, NOT a Node-free slim image (v1 of this file
+# tried that and broke every job: `actions/checkout@v4` is a JS action that
+# Forgejo's runner execs as `node dist/index.js` *inside the job container*,
+# so a Node runtime is a hard requirement regardless of whether the workflow
+# itself uses npm/node — this repo's own workflows don't, but the checkout
+# step every one of them starts with does).
+#
+# What this image actually fixes: every workflow using the `docker` label
+# currently re-provisions the Docker CLI on each run via `apt-get install
+# docker.io` — that step costs ~15-20s per job and, more importantly,
+# installs the CLASSIC Docker builder rather than BuildKit, which mishandles
+# `COPY --chown=<name>` group resolution on images without a matching system
+# group (a real bug hit during the frontend Go rewrite). Installing
+# docker-buildx-plugin here makes BuildKit the default, closing that bug
+# class, and skips the per-job install entirely.
+#
+# Build/push (manual — see ../README.md for when to rebuild):
+#   docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN \
+#     infra/deploy/forgejo/ci-runner
+#   docker push git.thermograph.org/emi/thermograph/ci-runner:vN
+#
+# Cutting over the live runner to a new tag means editing the `labels` array
+# in ~/forgejo-runner/.runner on the runner host directly (same runner
+# id/token, no re-registration needed) and updating register-lan-runner.sh's
+# LABELS default so future re-provisioning picks it up too.
+FROM node:20-bookworm
+
+RUN apt-get update -qq \
+    && apt-get install -y -qq --no-install-recommends \
+        ca-certificates curl gnupg git python3 python3-yaml \
+    && install -m 0755 -d /etc/apt/keyrings \
+    && curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc \
+    && chmod a+r /etc/apt/keyrings/docker.asc \
+    && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian bookworm stable" \
+        > /etc/apt/sources.list.d/docker.list \
+    && apt-get update -qq \
+    && apt-get install -y -qq --no-install-recommends docker-ce-cli docker-buildx-plugin \
+    && rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/docker.list
+
+RUN docker --version && docker buildx version && git --version \
+    && node --version \
+    && python3 -c "import yaml; print('PyYAML', yaml.__version__)"

infra/deploy/forgejo/docker-stack.yml

@@ -48,6 +48,10 @@ services:
     deploy:
       placement:
         constraints: [node.labels.role == forge]
+      resources:
+        limits:
+          cpus: "${FORGEJO_DB_CPUS:-1}"
+          memory: ${FORGEJO_DB_MEMORY:-1g}
       restart_policy:
         condition: on-failure
 
@@ -131,6 +135,10 @@ services:
     deploy:
       placement:
         constraints: [node.labels.role == forge]
+      resources:
+        limits:
+          cpus: "${FORGEJO_CPUS:-2}"
+          memory: ${FORGEJO_MEMORY:-2g}
       restart_policy:
         condition: on-failure
 

infra/deploy/forgejo/register-lan-runner.sh

@@ -26,7 +26,7 @@ set -euo pipefail
 FORGEJO_URL="${1:?usage: $0 <forgejo_url> <registration_token>}"
 TOKEN="${2:?}"
 RUNNER_DIR="${RUNNER_DIR:-$HOME/forgejo-runner}"
-LABELS="${LABELS:-docker:docker://node:20-bookworm,thermograph-lan}"
+LABELS="${LABELS:-docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v2,thermograph-lan}"
 
 echo "==> Stopping and disabling the old GitHub Actions runner service, if present"
 systemctl --user stop github-actions-runner 2>/dev/null || true