infra/forgejo: add resource limits and a leaner CI job image #38

Merged
admin_emi merged 4 commits from infra/forgejo-reliability-ci-perf into main 2026-07-24 18:56:47 +00:00
3 changed files with 44 additions and 35 deletions
Showing only changes of commit 3678f643ef - Show all commits

View file

@ -109,36 +109,40 @@ labels where there used to be two separate runners.
## Custom CI job image (`ci-runner/`)
`ci-runner/Dockerfile` replaces the stock `node:20-bookworm` image most
`docker`-labeled workflow jobs run in. That base is leftover from when the
frontend was Python/Jinja and the old GitHub runner also built JS assets for
it — the frontend is Go now, nothing in `.forgejo/workflows/` uses npm/node.
Every `docker`-labeled build-push job also currently re-installs the Docker
CLI on each run (`apt-get install docker.io`), which pulls in the classic
builder rather than BuildKit (the classic builder mishandles `COPY
--chown=<name>` group resolution — a real bug hit during the frontend Go
rewrite). `ci-runner` is a slim Debian base with `docker-ce-cli` +
`docker-buildx-plugin` (BuildKit) preinstalled, plus `git`/`python3`/
`python3-yaml` for the other jobs that need them (`shell-lint`,
`observability-validate`).
`ci-runner/Dockerfile` still bases on `node:20-bookworm` — Node is a hard
requirement, not leftover: Forgejo's runner executes `actions/checkout@v4`
(used by every workflow) as `node dist/index.js` *inside the job container*,
regardless of whether the workflow itself uses npm/node. (v1 of this image
tried a Node-free Debian-slim base and broke every job's checkout step —
`node: executable file not found` — within a minute of going live; reverted
immediately.) What it actually fixes: every `docker`-labeled build-push job
currently re-installs the Docker CLI on each run (`apt-get install
docker.io`), which pulls in the classic builder rather than BuildKit (the
classic builder mishandles `COPY --chown=<name>` group resolution — a real
bug hit during the frontend Go rewrite). `ci-runner` adds `docker-ce-cli` +
`docker-buildx-plugin` (BuildKit) on top of the same Node base, plus
`git`/`python3`/`python3-yaml` for the other jobs that need them
(`shell-lint`, `observability-validate`).
Pushed as `git.thermograph.org/emi/thermograph/ci-runner:v1`. Rebuild/push
(requires a PAT with `write:package` scope — the embedded git-remote token
lacks it, same requirement documented in `backend-build-push.yml`):
Current tag: `git.thermograph.org/emi/thermograph/ci-runner:v2` (`v1` is
broken — do not register any runner against it). Rebuild/push (requires a PAT
with `write:package` scope — the embedded git-remote token lacks it, same
requirement documented in `backend-build-push.yml`):
```bash
docker build -t git.thermograph.org/emi/thermograph/ci-runner:v1 deploy/forgejo/ci-runner
docker push git.thermograph.org/emi/thermograph/ci-runner:v1
docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN deploy/forgejo/ci-runner
docker push git.thermograph.org/emi/thermograph/ci-runner:vN
```
`register-lan-runner.sh`'s `LABELS` default now points at this image, so
`register-lan-runner.sh`'s `LABELS` default points at the current tag, so
fresh registrations pick it up automatically. The live runner is cut over by
editing the `labels` array in `~/forgejo-runner/.runner` on the desktop (same
runner id/token, no re-registration needed) and restarting the service. Only
after a real job has run successfully under the new image should the
runner id/token, no re-registration needed) and restarting the service —
**verify a real job runs green under the new image before relying on it**,
same way v1's break was caught. Only after that verification should the
now-redundant `apt-get install docker.io` / `python3-yaml` steps be removed
from the workflows that had them — removing them first would break
every job still running on `node:20-bookworm`.
from the workflows that had them — removing them first would break every job
still running on the stock `node:20-bookworm` image.
## Why Postgres here and not the Thermograph app's TimescaleDB

View file

@ -2,17 +2,21 @@
# jobs (see ../register-lan-runner.sh) — used by every backend/frontend
# build-push, deploy, and validation workflow that needs `docker build`.
#
# Replaces the stock node:20-bookworm image. That base is pure leftover from
# when the frontend was a Python/Jinja app the old GitHub runner also built
# JS assets for; the frontend is Go now (no npm/node anywhere in
# .forgejo/workflows/) so carrying a full Node runtime here just adds weight.
# Every workflow using the `docker` label also currently re-provisions the
# Docker CLI itself on each run via `apt-get install docker.io` — that step
# costs ~15-20s per job and, more importantly, installs the CLASSIC Docker
# builder rather than BuildKit, which mishandles `COPY --chown=<name>` group
# resolution on images without a matching system group (a real bug hit during
# the frontend Go rewrite). Installing docker-buildx-plugin here makes
# BuildKit the default, closing that bug class, on a much smaller base.
# Base stays node:20-bookworm, NOT a Node-free slim image (v1 of this file
# tried that and broke every job: `actions/checkout@v4` is a JS action that
# Forgejo's runner execs as `node dist/index.js` *inside the job container*,
# so a Node runtime is a hard requirement regardless of whether the workflow
# itself uses npm/node — this repo's own workflows don't, but the checkout
# step every one of them starts with does).
#
# What this image actually fixes: every workflow using the `docker` label
# currently re-provisions the Docker CLI on each run via `apt-get install
# docker.io` — that step costs ~15-20s per job and, more importantly,
# installs the CLASSIC Docker builder rather than BuildKit, which mishandles
# `COPY --chown=<name>` group resolution on images without a matching system
# group (a real bug hit during the frontend Go rewrite). Installing
# docker-buildx-plugin here makes BuildKit the default, closing that bug
# class, and skips the per-job install entirely.
#
# Build/push (manual — see ../README.md for when to rebuild):
# docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN \
@ -23,7 +27,7 @@
# in ~/forgejo-runner/.runner on the runner host directly (same runner
# id/token, no re-registration needed) and updating register-lan-runner.sh's
# LABELS default so future re-provisioning picks it up too.
FROM debian:bookworm-slim
FROM node:20-bookworm
RUN apt-get update -qq \
&& apt-get install -y -qq --no-install-recommends \
@ -38,4 +42,5 @@ RUN apt-get update -qq \
&& rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/docker.list
RUN docker --version && docker buildx version && git --version \
&& node --version \
&& python3 -c "import yaml; print('PyYAML', yaml.__version__)"

View file

@ -26,7 +26,7 @@ set -euo pipefail
FORGEJO_URL="${1:?usage: $0 <forgejo_url> <registration_token>}"
TOKEN="${2:?}"
RUNNER_DIR="${RUNNER_DIR:-$HOME/forgejo-runner}"
LABELS="${LABELS:-docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v1,thermograph-lan}"
LABELS="${LABELS:-docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v2,thermograph-lan}"
echo "==> Stopping and disabling the old GitHub Actions runner service, if present"
systemctl --user stop github-actions-runner 2>/dev/null || true