diff --git a/infra/deploy/forgejo/README.md b/infra/deploy/forgejo/README.md index 0325386..8ad737d 100644 --- a/infra/deploy/forgejo/README.md +++ b/infra/deploy/forgejo/README.md @@ -41,6 +41,15 @@ Do this **before** the DNS + Caddy step below — Caddy's reverse_proxy target (`127.0.0.1:3080`) needs the `forgejo` service actually listening first, or its first health check just fails harmlessly until it is. +This stack has **no auto-deploy trigger** — nothing in `.forgejo/workflows/` +redeploys it on push. A change to `docker-stack.yml` only takes effect once +someone re-runs `docker stack deploy` by hand on the manager (prod). + +`db`/`forgejo` both carry `resources.limits` (defaults: db 1 CPU/1g, forgejo 2 +CPU/2g — several times observed steady-state usage), overridable with +`FORGEJO_DB_CPUS`/`FORGEJO_DB_MEMORY`/`FORGEJO_CPUS`/`FORGEJO_MEMORY` env vars +before `docker stack deploy`, same convention as the app stack. + ## DNS + TLS: reusing beta's existing Caddy, not a second reverse proxy Forgejo is pinned to beta (`role=forge`) — but beta is also **today's live @@ -98,6 +107,43 @@ See that script's header for exactly what it replaces (the pre-Forgejo GitHub self-hosted runner on this same machine) and why it registers with two labels where there used to be two separate runners. +## Custom CI job image (`ci-runner/`) + +`ci-runner/Dockerfile` still bases on `node:20-bookworm` — Node is a hard +requirement, not leftover: Forgejo's runner executes `actions/checkout@v4` +(used by every workflow) as `node dist/index.js` *inside the job container*, +regardless of whether the workflow itself uses npm/node. (v1 of this image +tried a Node-free Debian-slim base and broke every job's checkout step — +`node: executable file not found` — within a minute of going live; reverted +immediately.) What it actually fixes: every `docker`-labeled build-push job +currently re-installs the Docker CLI on each run (`apt-get install +docker.io`), which pulls in the classic builder rather than BuildKit (the +classic builder mishandles `COPY --chown=` group resolution — a real +bug hit during the frontend Go rewrite). `ci-runner` adds `docker-ce-cli` + +`docker-buildx-plugin` (BuildKit) on top of the same Node base, plus +`git`/`python3`/`python3-yaml` for the other jobs that need them +(`shell-lint`, `observability-validate`). + +Current tag: `git.thermograph.org/emi/thermograph/ci-runner:v2` (`v1` is +broken — do not register any runner against it). Rebuild/push (requires a PAT +with `write:package` scope — the embedded git-remote token lacks it, same +requirement documented in `backend-build-push.yml`): + +```bash +docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN deploy/forgejo/ci-runner +docker push git.thermograph.org/emi/thermograph/ci-runner:vN +``` + +`register-lan-runner.sh`'s `LABELS` default points at the current tag, so +fresh registrations pick it up automatically. The live runner is cut over by +editing the `labels` array in `~/forgejo-runner/.runner` on the desktop (same +runner id/token, no re-registration needed) and restarting the service — +**verify a real job runs green under the new image before relying on it**, +same way v1's break was caught. Only after that verification should the +now-redundant `apt-get install docker.io` / `python3-yaml` steps be removed +from the workflows that had them — removing them first would break every job +still running on the stock `node:20-bookworm` image. + ## Why Postgres here and not the Thermograph app's TimescaleDB Separate instance, separate network (`forgejo_net`, not the app's compose diff --git a/infra/deploy/forgejo/ci-runner/Dockerfile b/infra/deploy/forgejo/ci-runner/Dockerfile new file mode 100644 index 0000000..e4259e0 --- /dev/null +++ b/infra/deploy/forgejo/ci-runner/Dockerfile @@ -0,0 +1,46 @@ +# Job-container image for the LAN Forgejo Actions runner's `docker`-labeled +# jobs (see ../register-lan-runner.sh) — used by every backend/frontend +# build-push, deploy, and validation workflow that needs `docker build`. +# +# Base stays node:20-bookworm, NOT a Node-free slim image (v1 of this file +# tried that and broke every job: `actions/checkout@v4` is a JS action that +# Forgejo's runner execs as `node dist/index.js` *inside the job container*, +# so a Node runtime is a hard requirement regardless of whether the workflow +# itself uses npm/node — this repo's own workflows don't, but the checkout +# step every one of them starts with does). +# +# What this image actually fixes: every workflow using the `docker` label +# currently re-provisions the Docker CLI on each run via `apt-get install +# docker.io` — that step costs ~15-20s per job and, more importantly, +# installs the CLASSIC Docker builder rather than BuildKit, which mishandles +# `COPY --chown=` group resolution on images without a matching system +# group (a real bug hit during the frontend Go rewrite). Installing +# docker-buildx-plugin here makes BuildKit the default, closing that bug +# class, and skips the per-job install entirely. +# +# Build/push (manual — see ../README.md for when to rebuild): +# docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN \ +# infra/deploy/forgejo/ci-runner +# docker push git.thermograph.org/emi/thermograph/ci-runner:vN +# +# Cutting over the live runner to a new tag means editing the `labels` array +# in ~/forgejo-runner/.runner on the runner host directly (same runner +# id/token, no re-registration needed) and updating register-lan-runner.sh's +# LABELS default so future re-provisioning picks it up too. +FROM node:20-bookworm + +RUN apt-get update -qq \ + && apt-get install -y -qq --no-install-recommends \ + ca-certificates curl gnupg git python3 python3-yaml \ + && install -m 0755 -d /etc/apt/keyrings \ + && curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc \ + && chmod a+r /etc/apt/keyrings/docker.asc \ + && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian bookworm stable" \ + > /etc/apt/sources.list.d/docker.list \ + && apt-get update -qq \ + && apt-get install -y -qq --no-install-recommends docker-ce-cli docker-buildx-plugin \ + && rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/docker.list + +RUN docker --version && docker buildx version && git --version \ + && node --version \ + && python3 -c "import yaml; print('PyYAML', yaml.__version__)" diff --git a/infra/deploy/forgejo/docker-stack.yml b/infra/deploy/forgejo/docker-stack.yml index 9d94cbd..9891c3b 100644 --- a/infra/deploy/forgejo/docker-stack.yml +++ b/infra/deploy/forgejo/docker-stack.yml @@ -48,6 +48,10 @@ services: deploy: placement: constraints: [node.labels.role == forge] + resources: + limits: + cpus: "${FORGEJO_DB_CPUS:-1}" + memory: ${FORGEJO_DB_MEMORY:-1g} restart_policy: condition: on-failure @@ -131,6 +135,10 @@ services: deploy: placement: constraints: [node.labels.role == forge] + resources: + limits: + cpus: "${FORGEJO_CPUS:-2}" + memory: ${FORGEJO_MEMORY:-2g} restart_policy: condition: on-failure diff --git a/infra/deploy/forgejo/register-lan-runner.sh b/infra/deploy/forgejo/register-lan-runner.sh index 71f1c90..c35660d 100755 --- a/infra/deploy/forgejo/register-lan-runner.sh +++ b/infra/deploy/forgejo/register-lan-runner.sh @@ -26,7 +26,7 @@ set -euo pipefail FORGEJO_URL="${1:?usage: $0 }" TOKEN="${2:?}" RUNNER_DIR="${RUNNER_DIR:-$HOME/forgejo-runner}" -LABELS="${LABELS:-docker:docker://node:20-bookworm,thermograph-lan}" +LABELS="${LABELS:-docker:docker://git.thermograph.org/emi/thermograph/ci-runner:v2,thermograph-lan}" echo "==> Stopping and disabling the old GitHub Actions runner service, if present" systemctl --user stop github-actions-runner 2>/dev/null || true