From c966ac4801022b04d8545d7957416f7119f6a6ae Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Fri, 24 Jul 2026 00:21:29 -0700 Subject: [PATCH] infra/forgejo: add resource limits and a leaner CI job image docker-stack.yml previously had no cpu/memory limits on db or forgejo, unlike every service in the app stack. Add generous limits (several times observed steady-state usage) as a backstop, overridable via env vars matching the app stack's convention. Also add ci-runner/Dockerfile: a slim Debian base with the Docker CLI + buildx preinstalled, replacing node:20-bookworm (leftover from when the frontend was Python/Jinja; nothing in CI uses npm/node anymore). Closes a COPY --chown group-resolution bug in the classic Docker builder that apt-get install docker.io currently pulls in on every build-push job, and drops that install step's cost once cut over. Built and verified locally; not yet pushed to the registry (needs a write:package-scoped token) or wired into the live runner. --- infra/deploy/forgejo/README.md | 42 +++++++++++++++++++++++ infra/deploy/forgejo/ci-runner/Dockerfile | 41 ++++++++++++++++++++++ infra/deploy/forgejo/docker-stack.yml | 8 +++++ 3 files changed, 91 insertions(+) create mode 100644 infra/deploy/forgejo/ci-runner/Dockerfile diff --git a/infra/deploy/forgejo/README.md b/infra/deploy/forgejo/README.md index 0325386..03f6957 100644 --- a/infra/deploy/forgejo/README.md +++ b/infra/deploy/forgejo/README.md @@ -41,6 +41,15 @@ Do this **before** the DNS + Caddy step below — Caddy's reverse_proxy target (`127.0.0.1:3080`) needs the `forgejo` service actually listening first, or its first health check just fails harmlessly until it is. +This stack has **no auto-deploy trigger** — nothing in `.forgejo/workflows/` +redeploys it on push. A change to `docker-stack.yml` only takes effect once +someone re-runs `docker stack deploy` by hand on the manager (prod). + +`db`/`forgejo` both carry `resources.limits` (defaults: db 1 CPU/1g, forgejo 2 +CPU/2g — several times observed steady-state usage), overridable with +`FORGEJO_DB_CPUS`/`FORGEJO_DB_MEMORY`/`FORGEJO_CPUS`/`FORGEJO_MEMORY` env vars +before `docker stack deploy`, same convention as the app stack. + ## DNS + TLS: reusing beta's existing Caddy, not a second reverse proxy Forgejo is pinned to beta (`role=forge`) — but beta is also **today's live @@ -98,6 +107,39 @@ See that script's header for exactly what it replaces (the pre-Forgejo GitHub self-hosted runner on this same machine) and why it registers with two labels where there used to be two separate runners. +## Custom CI job image (`ci-runner/`) + +`ci-runner/Dockerfile` replaces the stock `node:20-bookworm` image most +`docker`-labeled workflow jobs run in. That base is leftover from when the +frontend was Python/Jinja and the old GitHub runner also built JS assets for +it — the frontend is Go now, nothing in `.forgejo/workflows/` uses npm/node. +Every `docker`-labeled build-push job also currently re-installs the Docker +CLI on each run (`apt-get install docker.io`), which pulls in the classic +builder rather than BuildKit (the classic builder mishandles `COPY +--chown=` group resolution — a real bug hit during the frontend Go +rewrite). `ci-runner` is a slim Debian base with `docker-ce-cli` + +`docker-buildx-plugin` (BuildKit) preinstalled, plus `git`/`python3`/ +`python3-yaml` for the other jobs that need them (`shell-lint`, +`observability-validate`). + +Built and verified locally; **not yet pushed** — pushing to +`git.thermograph.org/emi/thermograph/ci-runner` needs a PAT with +`write:package` scope (the embedded git-remote token lacks it, same +requirement documented in `backend-build-push.yml`). Once pushed: + +```bash +docker build -t git.thermograph.org/emi/thermograph/ci-runner:v1 deploy/forgejo/ci-runner +docker push git.thermograph.org/emi/thermograph/ci-runner:v1 +``` + +then cut the live runner over by editing the `labels` array in +`~/forgejo-runner/.runner` on the desktop (same runner id/token, no +re-registration) and updating `register-lan-runner.sh`'s `LABELS` default so +future re-provisioning matches. Only after that cutover is verified working +should the now-redundant `apt-get install docker.io` / `python3-yaml` steps be +removed from the workflows that had them — removing them first would break +every job still running on `node:20-bookworm`. + ## Why Postgres here and not the Thermograph app's TimescaleDB Separate instance, separate network (`forgejo_net`, not the app's compose diff --git a/infra/deploy/forgejo/ci-runner/Dockerfile b/infra/deploy/forgejo/ci-runner/Dockerfile new file mode 100644 index 0000000..4db36b6 --- /dev/null +++ b/infra/deploy/forgejo/ci-runner/Dockerfile @@ -0,0 +1,41 @@ +# Job-container image for the LAN Forgejo Actions runner's `docker`-labeled +# jobs (see ../register-lan-runner.sh) — used by every backend/frontend +# build-push, deploy, and validation workflow that needs `docker build`. +# +# Replaces the stock node:20-bookworm image. That base is pure leftover from +# when the frontend was a Python/Jinja app the old GitHub runner also built +# JS assets for; the frontend is Go now (no npm/node anywhere in +# .forgejo/workflows/) so carrying a full Node runtime here just adds weight. +# Every workflow using the `docker` label also currently re-provisions the +# Docker CLI itself on each run via `apt-get install docker.io` — that step +# costs ~15-20s per job and, more importantly, installs the CLASSIC Docker +# builder rather than BuildKit, which mishandles `COPY --chown=` group +# resolution on images without a matching system group (a real bug hit during +# the frontend Go rewrite). Installing docker-buildx-plugin here makes +# BuildKit the default, closing that bug class, on a much smaller base. +# +# Build/push (manual — see ../README.md for when to rebuild): +# docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN \ +# infra/deploy/forgejo/ci-runner +# docker push git.thermograph.org/emi/thermograph/ci-runner:vN +# +# Cutting over the live runner to a new tag means editing the `labels` array +# in ~/forgejo-runner/.runner on the runner host directly (same runner +# id/token, no re-registration needed) and updating register-lan-runner.sh's +# LABELS default so future re-provisioning picks it up too. +FROM debian:bookworm-slim + +RUN apt-get update -qq \ + && apt-get install -y -qq --no-install-recommends \ + ca-certificates curl gnupg git python3 python3-yaml \ + && install -m 0755 -d /etc/apt/keyrings \ + && curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc \ + && chmod a+r /etc/apt/keyrings/docker.asc \ + && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian bookworm stable" \ + > /etc/apt/sources.list.d/docker.list \ + && apt-get update -qq \ + && apt-get install -y -qq --no-install-recommends docker-ce-cli docker-buildx-plugin \ + && rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/docker.list + +RUN docker --version && docker buildx version && git --version \ + && python3 -c "import yaml; print('PyYAML', yaml.__version__)" diff --git a/infra/deploy/forgejo/docker-stack.yml b/infra/deploy/forgejo/docker-stack.yml index 9d94cbd..9891c3b 100644 --- a/infra/deploy/forgejo/docker-stack.yml +++ b/infra/deploy/forgejo/docker-stack.yml @@ -48,6 +48,10 @@ services: deploy: placement: constraints: [node.labels.role == forge] + resources: + limits: + cpus: "${FORGEJO_DB_CPUS:-1}" + memory: ${FORGEJO_DB_MEMORY:-1g} restart_policy: condition: on-failure @@ -131,6 +135,10 @@ services: deploy: placement: constraints: [node.labels.role == forge] + resources: + limits: + cpus: "${FORGEJO_CPUS:-2}" + memory: ${FORGEJO_MEMORY:-2g} restart_policy: condition: on-failure