infra/forgejo: add resource limits and a leaner CI job image #38

Merged
admin_emi merged 4 commits from infra/forgejo-reliability-ci-perf into main 2026-07-24 18:56:47 +00:00
3 changed files with 44 additions and 35 deletions
Showing only changes of commit 3678f643ef - Show all commits

View file

@ -109,36 +109,40 @@ labels where there used to be two separate runners.
## Custom CI job image (`ci-runner/`) ## Custom CI job image (`ci-runner/`)
`ci-runner/Dockerfile` replaces the stock `node:20-bookworm` image most `ci-runner/Dockerfile` still bases on `node:20-bookworm` — Node is a hard
`docker`-labeled workflow jobs run in. That base is leftover from when the requirement, not leftover: Forgejo's runner executes `actions/checkout@v4`
frontend was Python/Jinja and the old GitHub runner also built JS assets for (used by every workflow) as `node dist/index.js` *inside the job container*,
it — the frontend is Go now, nothing in `.forgejo/workflows/` uses npm/node. regardless of whether the workflow itself uses npm/node. (v1 of this image
Every `docker`-labeled build-push job also currently re-installs the Docker tried a Node-free Debian-slim base and broke every job's checkout step —
CLI on each run (`apt-get install docker.io`), which pulls in the classic `node: executable file not found` — within a minute of going live; reverted
builder rather than BuildKit (the classic builder mishandles `COPY immediately.) What it actually fixes: every `docker`-labeled build-push job
--chown=<name>` group resolution — a real bug hit during the frontend Go currently re-installs the Docker CLI on each run (`apt-get install
rewrite). `ci-runner` is a slim Debian base with `docker-ce-cli` + docker.io`), which pulls in the classic builder rather than BuildKit (the
`docker-buildx-plugin` (BuildKit) preinstalled, plus `git`/`python3`/ classic builder mishandles `COPY --chown=<name>` group resolution — a real
`python3-yaml` for the other jobs that need them (`shell-lint`, bug hit during the frontend Go rewrite). `ci-runner` adds `docker-ce-cli` +
`observability-validate`). `docker-buildx-plugin` (BuildKit) on top of the same Node base, plus
`git`/`python3`/`python3-yaml` for the other jobs that need them
(`shell-lint`, `observability-validate`).
Pushed as `git.thermograph.org/emi/thermograph/ci-runner:v1`. Rebuild/push Current tag: `git.thermograph.org/emi/thermograph/ci-runner:v2` (`v1` is
(requires a PAT with `write:package` scope — the embedded git-remote token broken — do not register any runner against it). Rebuild/push (requires a PAT
lacks it, same requirement documented in `backend-build-push.yml`): with `write:package` scope — the embedded git-remote token lacks it, same
requirement documented in `backend-build-push.yml`):
```bash ```bash
docker build -t git.thermograph.org/emi/thermograph/ci-runner:v1 deploy/forgejo/ci-runner docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN deploy/forgejo/ci-runner
docker push git.thermograph.org/emi/thermograph/ci-runner:v1 docker push git.thermograph.org/emi/thermograph/ci-runner:vN
``` ```
`register-lan-runner.sh`'s `LABELS` default now points at this image, so `register-lan-runner.sh`'s `LABELS` default points at the current tag, so
fresh registrations pick it up automatically. The live runner is cut over by fresh registrations pick it up automatically. The live runner is cut over by
editing the `labels` array in `~/forgejo-runner/.runner` on the desktop (same editing the `labels` array in `~/forgejo-runner/.runner` on the desktop (same
runner id/token, no re-registration needed) and restarting the service. Only runner id/token, no re-registration needed) and restarting the service —
after a real job has run successfully under the new image should the **verify a real job runs green under the new image before relying on it**,
same way v1's break was caught. Only after that verification should the
now-redundant `apt-get install docker.io` / `python3-yaml` steps be removed now-redundant `apt-get install docker.io` / `python3-yaml` steps be removed
from the workflows that had them — removing them first would break from the workflows that had them — removing them first would break every job
every job still running on `node:20-bookworm`. still running on the stock `node:20-bookworm` image.
## Why Postgres here and not the Thermograph app's TimescaleDB ## Why Postgres here and not the Thermograph app's TimescaleDB

View file

@ -2,17 +2,21 @@
# jobs (see ../register-lan-runner.sh) — used by every backend/frontend # jobs (see ../register-lan-runner.sh) — used by every backend/frontend
# build-push, deploy, and validation workflow that needs `docker build`. # build-push, deploy, and validation workflow that needs `docker build`.
# #
# Replaces the stock node:20-bookworm image. That base is pure leftover from # Base stays node:20-bookworm, NOT a Node-free slim image (v1 of this file
# when the frontend was a Python/Jinja app the old GitHub runner also built # tried that and broke every job: `actions/checkout@v4` is a JS action that
# JS assets for; the frontend is Go now (no npm/node anywhere in # Forgejo's runner execs as `node dist/index.js` *inside the job container*,
# .forgejo/workflows/) so carrying a full Node runtime here just adds weight. # so a Node runtime is a hard requirement regardless of whether the workflow
# Every workflow using the `docker` label also currently re-provisions the # itself uses npm/node — this repo's own workflows don't, but the checkout
# Docker CLI itself on each run via `apt-get install docker.io` — that step # step every one of them starts with does).
# costs ~15-20s per job and, more importantly, installs the CLASSIC Docker #
# builder rather than BuildKit, which mishandles `COPY --chown=<name>` group # What this image actually fixes: every workflow using the `docker` label
# resolution on images without a matching system group (a real bug hit during # currently re-provisions the Docker CLI on each run via `apt-get install
# the frontend Go rewrite). Installing docker-buildx-plugin here makes # docker.io` — that step costs ~15-20s per job and, more importantly,
# BuildKit the default, closing that bug class, on a much smaller base. # installs the CLASSIC Docker builder rather than BuildKit, which mishandles
# `COPY --chown=<name>` group resolution on images without a matching system
# group (a real bug hit during the frontend Go rewrite). Installing
# docker-buildx-plugin here makes BuildKit the default, closing that bug
# class, and skips the per-job install entirely.
# #
# Build/push (manual — see ../README.md for when to rebuild): # Build/push (manual — see ../README.md for when to rebuild):
# docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN \ # docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN \
@ -23,7 +27,7 @@
# in ~/forgejo-runner/.runner on the runner host directly (same runner # in ~/forgejo-runner/.runner on the runner host directly (same runner
# id/token, no re-registration needed) and updating register-lan-runner.sh's # id/token, no re-registration needed) and updating register-lan-runner.sh's
# LABELS default so future re-provisioning picks it up too. # LABELS default so future re-provisioning picks it up too.
FROM debian:bookworm-slim FROM node:20-bookworm
RUN apt-get update -qq \ RUN apt-get update -qq \
&& apt-get install -y -qq --no-install-recommends \ && apt-get install -y -qq --no-install-recommends \
@ -38,4 +42,5 @@ RUN apt-get update -qq \
&& rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/docker.list && rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/docker.list
RUN docker --version && docker buildx version && git --version \ RUN docker --version && docker buildx version && git --version \
&& node --version \
&& python3 -c "import yaml; print('PyYAML', yaml.__version__)" && python3 -c "import yaml; print('PyYAML', yaml.__version__)"

View file

@ -26,7 +26,7 @@ set -euo pipefail
FORGEJO_URL="${1:?usage: $0 <forgejo_url> <registration_token>}" FORGEJO_URL="${1:?usage: $0 <forgejo_url> <registration_token>}"
TOKEN="${2:?}" TOKEN="${2:?}"
RUNNER_DIR="${RUNNER_DIR:-$HOME/forgejo-runner}" RUNNER_DIR="${RUNNER_DIR:-$HOME/forgejo-runner}"
LABELS="${LABELS:-docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v1,thermograph-lan}" LABELS="${LABELS:-docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v2,thermograph-lan}"
echo "==> Stopping and disabling the old GitHub Actions runner service, if present" echo "==> Stopping and disabling the old GitHub Actions runner service, if present"
systemctl --user stop github-actions-runner 2>/dev/null || true systemctl --user stop github-actions-runner 2>/dev/null || true