diff --git a/infra/deploy/forgejo/README.md b/infra/deploy/forgejo/README.md index caf2ffe..8ad737d 100644 --- a/infra/deploy/forgejo/README.md +++ b/infra/deploy/forgejo/README.md @@ -109,36 +109,40 @@ labels where there used to be two separate runners. ## Custom CI job image (`ci-runner/`) -`ci-runner/Dockerfile` replaces the stock `node:20-bookworm` image most -`docker`-labeled workflow jobs run in. That base is leftover from when the -frontend was Python/Jinja and the old GitHub runner also built JS assets for -it — the frontend is Go now, nothing in `.forgejo/workflows/` uses npm/node. -Every `docker`-labeled build-push job also currently re-installs the Docker -CLI on each run (`apt-get install docker.io`), which pulls in the classic -builder rather than BuildKit (the classic builder mishandles `COPY ---chown=` group resolution — a real bug hit during the frontend Go -rewrite). `ci-runner` is a slim Debian base with `docker-ce-cli` + -`docker-buildx-plugin` (BuildKit) preinstalled, plus `git`/`python3`/ -`python3-yaml` for the other jobs that need them (`shell-lint`, -`observability-validate`). +`ci-runner/Dockerfile` still bases on `node:20-bookworm` — Node is a hard +requirement, not leftover: Forgejo's runner executes `actions/checkout@v4` +(used by every workflow) as `node dist/index.js` *inside the job container*, +regardless of whether the workflow itself uses npm/node. (v1 of this image +tried a Node-free Debian-slim base and broke every job's checkout step — +`node: executable file not found` — within a minute of going live; reverted +immediately.) What it actually fixes: every `docker`-labeled build-push job +currently re-installs the Docker CLI on each run (`apt-get install +docker.io`), which pulls in the classic builder rather than BuildKit (the +classic builder mishandles `COPY --chown=` group resolution — a real +bug hit during the frontend Go rewrite). `ci-runner` adds `docker-ce-cli` + +`docker-buildx-plugin` (BuildKit) on top of the same Node base, plus +`git`/`python3`/`python3-yaml` for the other jobs that need them +(`shell-lint`, `observability-validate`). -Pushed as `git.thermograph.org/emi/thermograph/ci-runner:v1`. Rebuild/push -(requires a PAT with `write:package` scope — the embedded git-remote token -lacks it, same requirement documented in `backend-build-push.yml`): +Current tag: `git.thermograph.org/emi/thermograph/ci-runner:v2` (`v1` is +broken — do not register any runner against it). Rebuild/push (requires a PAT +with `write:package` scope — the embedded git-remote token lacks it, same +requirement documented in `backend-build-push.yml`): ```bash -docker build -t git.thermograph.org/emi/thermograph/ci-runner:v1 deploy/forgejo/ci-runner -docker push git.thermograph.org/emi/thermograph/ci-runner:v1 +docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN deploy/forgejo/ci-runner +docker push git.thermograph.org/emi/thermograph/ci-runner:vN ``` -`register-lan-runner.sh`'s `LABELS` default now points at this image, so +`register-lan-runner.sh`'s `LABELS` default points at the current tag, so fresh registrations pick it up automatically. The live runner is cut over by editing the `labels` array in `~/forgejo-runner/.runner` on the desktop (same -runner id/token, no re-registration needed) and restarting the service. Only -after a real job has run successfully under the new image should the +runner id/token, no re-registration needed) and restarting the service — +**verify a real job runs green under the new image before relying on it**, +same way v1's break was caught. Only after that verification should the now-redundant `apt-get install docker.io` / `python3-yaml` steps be removed -from the workflows that had them — removing them first would break -every job still running on `node:20-bookworm`. +from the workflows that had them — removing them first would break every job +still running on the stock `node:20-bookworm` image. ## Why Postgres here and not the Thermograph app's TimescaleDB diff --git a/infra/deploy/forgejo/ci-runner/Dockerfile b/infra/deploy/forgejo/ci-runner/Dockerfile index 4db36b6..e4259e0 100644 --- a/infra/deploy/forgejo/ci-runner/Dockerfile +++ b/infra/deploy/forgejo/ci-runner/Dockerfile @@ -2,17 +2,21 @@ # jobs (see ../register-lan-runner.sh) — used by every backend/frontend # build-push, deploy, and validation workflow that needs `docker build`. # -# Replaces the stock node:20-bookworm image. That base is pure leftover from -# when the frontend was a Python/Jinja app the old GitHub runner also built -# JS assets for; the frontend is Go now (no npm/node anywhere in -# .forgejo/workflows/) so carrying a full Node runtime here just adds weight. -# Every workflow using the `docker` label also currently re-provisions the -# Docker CLI itself on each run via `apt-get install docker.io` — that step -# costs ~15-20s per job and, more importantly, installs the CLASSIC Docker -# builder rather than BuildKit, which mishandles `COPY --chown=` group -# resolution on images without a matching system group (a real bug hit during -# the frontend Go rewrite). Installing docker-buildx-plugin here makes -# BuildKit the default, closing that bug class, on a much smaller base. +# Base stays node:20-bookworm, NOT a Node-free slim image (v1 of this file +# tried that and broke every job: `actions/checkout@v4` is a JS action that +# Forgejo's runner execs as `node dist/index.js` *inside the job container*, +# so a Node runtime is a hard requirement regardless of whether the workflow +# itself uses npm/node — this repo's own workflows don't, but the checkout +# step every one of them starts with does). +# +# What this image actually fixes: every workflow using the `docker` label +# currently re-provisions the Docker CLI on each run via `apt-get install +# docker.io` — that step costs ~15-20s per job and, more importantly, +# installs the CLASSIC Docker builder rather than BuildKit, which mishandles +# `COPY --chown=` group resolution on images without a matching system +# group (a real bug hit during the frontend Go rewrite). Installing +# docker-buildx-plugin here makes BuildKit the default, closing that bug +# class, and skips the per-job install entirely. # # Build/push (manual — see ../README.md for when to rebuild): # docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN \ @@ -23,7 +27,7 @@ # in ~/forgejo-runner/.runner on the runner host directly (same runner # id/token, no re-registration needed) and updating register-lan-runner.sh's # LABELS default so future re-provisioning picks it up too. -FROM debian:bookworm-slim +FROM node:20-bookworm RUN apt-get update -qq \ && apt-get install -y -qq --no-install-recommends \ @@ -38,4 +42,5 @@ RUN apt-get update -qq \ && rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/docker.list RUN docker --version && docker buildx version && git --version \ + && node --version \ && python3 -c "import yaml; print('PyYAML', yaml.__version__)" diff --git a/infra/deploy/forgejo/register-lan-runner.sh b/infra/deploy/forgejo/register-lan-runner.sh index 6edb92b..c35660d 100755 --- a/infra/deploy/forgejo/register-lan-runner.sh +++ b/infra/deploy/forgejo/register-lan-runner.sh @@ -26,7 +26,7 @@ set -euo pipefail FORGEJO_URL="${1:?usage: $0 }" TOKEN="${2:?}" RUNNER_DIR="${RUNNER_DIR:-$HOME/forgejo-runner}" -LABELS="${LABELS:-docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v1,thermograph-lan}" +LABELS="${LABELS:-docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v2,thermograph-lan}" echo "==> Stopping and disabling the old GitHub Actions runner service, if present" systemctl --user stop github-actions-runner 2>/dev/null || true