From 3678f643efe23e4c5db667e4f908c18fb2176e23 Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Fri, 24 Jul 2026 11:47:49 -0700 Subject: [PATCH] =?UTF-8?q?infra/forgejo:=20fix=20ci-runner=20=E2=80=94=20?= =?UTF-8?q?node=20is=20required=20by=20actions/checkout?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit v1 dropped node:20-bookworm for a Node-free Debian base, on the wrong assumption that nothing in CI needs Node since the frontend is Go now. Forgejo's runner executes actions/checkout@v4 as `node dist/index.js` inside the job container regardless of what the workflow itself runs, so every job's checkout step failed. Caught from the live run within a minute; runner was reverted to node:20-bookworm immediately. v2 keeps the node:20-bookworm base and only adds docker-ce-cli + docker-buildx-plugin on top, preserving the actual fix (BuildKit instead of the classic builder, no per-job docker.io install) without removing a hard dependency. --- infra/deploy/forgejo/README.md | 48 +++++++++++---------- infra/deploy/forgejo/ci-runner/Dockerfile | 29 +++++++------ infra/deploy/forgejo/register-lan-runner.sh | 2 +- 3 files changed, 44 insertions(+), 35 deletions(-) diff --git a/infra/deploy/forgejo/README.md b/infra/deploy/forgejo/README.md index caf2ffe..8ad737d 100644 --- a/infra/deploy/forgejo/README.md +++ b/infra/deploy/forgejo/README.md @@ -109,36 +109,40 @@ labels where there used to be two separate runners. ## Custom CI job image (`ci-runner/`) -`ci-runner/Dockerfile` replaces the stock `node:20-bookworm` image most -`docker`-labeled workflow jobs run in. That base is leftover from when the -frontend was Python/Jinja and the old GitHub runner also built JS assets for -it — the frontend is Go now, nothing in `.forgejo/workflows/` uses npm/node. -Every `docker`-labeled build-push job also currently re-installs the Docker -CLI on each run (`apt-get install docker.io`), which pulls in the classic -builder rather than BuildKit (the classic builder mishandles `COPY ---chown=` group resolution — a real bug hit during the frontend Go -rewrite). `ci-runner` is a slim Debian base with `docker-ce-cli` + -`docker-buildx-plugin` (BuildKit) preinstalled, plus `git`/`python3`/ -`python3-yaml` for the other jobs that need them (`shell-lint`, -`observability-validate`). +`ci-runner/Dockerfile` still bases on `node:20-bookworm` — Node is a hard +requirement, not leftover: Forgejo's runner executes `actions/checkout@v4` +(used by every workflow) as `node dist/index.js` *inside the job container*, +regardless of whether the workflow itself uses npm/node. (v1 of this image +tried a Node-free Debian-slim base and broke every job's checkout step — +`node: executable file not found` — within a minute of going live; reverted +immediately.) What it actually fixes: every `docker`-labeled build-push job +currently re-installs the Docker CLI on each run (`apt-get install +docker.io`), which pulls in the classic builder rather than BuildKit (the +classic builder mishandles `COPY --chown=` group resolution — a real +bug hit during the frontend Go rewrite). `ci-runner` adds `docker-ce-cli` + +`docker-buildx-plugin` (BuildKit) on top of the same Node base, plus +`git`/`python3`/`python3-yaml` for the other jobs that need them +(`shell-lint`, `observability-validate`). -Pushed as `git.thermograph.org/emi/thermograph/ci-runner:v1`. Rebuild/push -(requires a PAT with `write:package` scope — the embedded git-remote token -lacks it, same requirement documented in `backend-build-push.yml`): +Current tag: `git.thermograph.org/emi/thermograph/ci-runner:v2` (`v1` is +broken — do not register any runner against it). Rebuild/push (requires a PAT +with `write:package` scope — the embedded git-remote token lacks it, same +requirement documented in `backend-build-push.yml`): ```bash -docker build -t git.thermograph.org/emi/thermograph/ci-runner:v1 deploy/forgejo/ci-runner -docker push git.thermograph.org/emi/thermograph/ci-runner:v1 +docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN deploy/forgejo/ci-runner +docker push git.thermograph.org/emi/thermograph/ci-runner:vN ``` -`register-lan-runner.sh`'s `LABELS` default now points at this image, so +`register-lan-runner.sh`'s `LABELS` default points at the current tag, so fresh registrations pick it up automatically. The live runner is cut over by editing the `labels` array in `~/forgejo-runner/.runner` on the desktop (same -runner id/token, no re-registration needed) and restarting the service. Only -after a real job has run successfully under the new image should the +runner id/token, no re-registration needed) and restarting the service — +**verify a real job runs green under the new image before relying on it**, +same way v1's break was caught. Only after that verification should the now-redundant `apt-get install docker.io` / `python3-yaml` steps be removed -from the workflows that had them — removing them first would break -every job still running on `node:20-bookworm`. +from the workflows that had them — removing them first would break every job +still running on the stock `node:20-bookworm` image. ## Why Postgres here and not the Thermograph app's TimescaleDB diff --git a/infra/deploy/forgejo/ci-runner/Dockerfile b/infra/deploy/forgejo/ci-runner/Dockerfile index 4db36b6..e4259e0 100644 --- a/infra/deploy/forgejo/ci-runner/Dockerfile +++ b/infra/deploy/forgejo/ci-runner/Dockerfile @@ -2,17 +2,21 @@ # jobs (see ../register-lan-runner.sh) — used by every backend/frontend # build-push, deploy, and validation workflow that needs `docker build`. # -# Replaces the stock node:20-bookworm image. That base is pure leftover from -# when the frontend was a Python/Jinja app the old GitHub runner also built -# JS assets for; the frontend is Go now (no npm/node anywhere in -# .forgejo/workflows/) so carrying a full Node runtime here just adds weight. -# Every workflow using the `docker` label also currently re-provisions the -# Docker CLI itself on each run via `apt-get install docker.io` — that step -# costs ~15-20s per job and, more importantly, installs the CLASSIC Docker -# builder rather than BuildKit, which mishandles `COPY --chown=` group -# resolution on images without a matching system group (a real bug hit during -# the frontend Go rewrite). Installing docker-buildx-plugin here makes -# BuildKit the default, closing that bug class, on a much smaller base. +# Base stays node:20-bookworm, NOT a Node-free slim image (v1 of this file +# tried that and broke every job: `actions/checkout@v4` is a JS action that +# Forgejo's runner execs as `node dist/index.js` *inside the job container*, +# so a Node runtime is a hard requirement regardless of whether the workflow +# itself uses npm/node — this repo's own workflows don't, but the checkout +# step every one of them starts with does). +# +# What this image actually fixes: every workflow using the `docker` label +# currently re-provisions the Docker CLI on each run via `apt-get install +# docker.io` — that step costs ~15-20s per job and, more importantly, +# installs the CLASSIC Docker builder rather than BuildKit, which mishandles +# `COPY --chown=` group resolution on images without a matching system +# group (a real bug hit during the frontend Go rewrite). Installing +# docker-buildx-plugin here makes BuildKit the default, closing that bug +# class, and skips the per-job install entirely. # # Build/push (manual — see ../README.md for when to rebuild): # docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN \ @@ -23,7 +27,7 @@ # in ~/forgejo-runner/.runner on the runner host directly (same runner # id/token, no re-registration needed) and updating register-lan-runner.sh's # LABELS default so future re-provisioning picks it up too. -FROM debian:bookworm-slim +FROM node:20-bookworm RUN apt-get update -qq \ && apt-get install -y -qq --no-install-recommends \ @@ -38,4 +42,5 @@ RUN apt-get update -qq \ && rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/docker.list RUN docker --version && docker buildx version && git --version \ + && node --version \ && python3 -c "import yaml; print('PyYAML', yaml.__version__)" diff --git a/infra/deploy/forgejo/register-lan-runner.sh b/infra/deploy/forgejo/register-lan-runner.sh index 6edb92b..c35660d 100755 --- a/infra/deploy/forgejo/register-lan-runner.sh +++ b/infra/deploy/forgejo/register-lan-runner.sh @@ -26,7 +26,7 @@ set -euo pipefail FORGEJO_URL="${1:?usage: $0 }" TOKEN="${2:?}" RUNNER_DIR="${RUNNER_DIR:-$HOME/forgejo-runner}" -LABELS="${LABELS:-docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v1,thermograph-lan}" +LABELS="${LABELS:-docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v2,thermograph-lan}" echo "==> Stopping and disabling the old GitHub Actions runner service, if present" systemctl --user stop github-actions-runner 2>/dev/null || true