infra/forgejo: add resource limits and a leaner CI job image
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / changes (pull_request) Successful in 9s
shell-lint / shellcheck (pull_request) Successful in 8s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / changes (pull_request) Successful in 9s
shell-lint / shellcheck (pull_request) Successful in 8s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
docker-stack.yml previously had no cpu/memory limits on db or forgejo, unlike every service in the app stack. Add generous limits (several times observed steady-state usage) as a backstop, overridable via env vars matching the app stack's convention. Also add ci-runner/Dockerfile: a slim Debian base with the Docker CLI + buildx preinstalled, replacing node:20-bookworm (leftover from when the frontend was Python/Jinja; nothing in CI uses npm/node anymore). Closes a COPY --chown group-resolution bug in the classic Docker builder that apt-get install docker.io currently pulls in on every build-push job, and drops that install step's cost once cut over. Built and verified locally; not yet pushed to the registry (needs a write:package-scoped token) or wired into the live runner.
This commit is contained in:
parent
da82abda27
commit
c966ac4801
3 changed files with 91 additions and 0 deletions
|
|
@ -41,6 +41,15 @@ Do this **before** the DNS + Caddy step below — Caddy's reverse_proxy target
|
||||||
(`127.0.0.1:3080`) needs the `forgejo` service actually listening first, or
|
(`127.0.0.1:3080`) needs the `forgejo` service actually listening first, or
|
||||||
its first health check just fails harmlessly until it is.
|
its first health check just fails harmlessly until it is.
|
||||||
|
|
||||||
|
This stack has **no auto-deploy trigger** — nothing in `.forgejo/workflows/`
|
||||||
|
redeploys it on push. A change to `docker-stack.yml` only takes effect once
|
||||||
|
someone re-runs `docker stack deploy` by hand on the manager (prod).
|
||||||
|
|
||||||
|
`db`/`forgejo` both carry `resources.limits` (defaults: db 1 CPU/1g, forgejo 2
|
||||||
|
CPU/2g — several times observed steady-state usage), overridable with
|
||||||
|
`FORGEJO_DB_CPUS`/`FORGEJO_DB_MEMORY`/`FORGEJO_CPUS`/`FORGEJO_MEMORY` env vars
|
||||||
|
before `docker stack deploy`, same convention as the app stack.
|
||||||
|
|
||||||
## DNS + TLS: reusing beta's existing Caddy, not a second reverse proxy
|
## DNS + TLS: reusing beta's existing Caddy, not a second reverse proxy
|
||||||
|
|
||||||
Forgejo is pinned to beta (`role=forge`) — but beta is also **today's live
|
Forgejo is pinned to beta (`role=forge`) — but beta is also **today's live
|
||||||
|
|
@ -98,6 +107,39 @@ See that script's header for exactly what it replaces (the pre-Forgejo GitHub
|
||||||
self-hosted runner on this same machine) and why it registers with two
|
self-hosted runner on this same machine) and why it registers with two
|
||||||
labels where there used to be two separate runners.
|
labels where there used to be two separate runners.
|
||||||
|
|
||||||
|
## Custom CI job image (`ci-runner/`)
|
||||||
|
|
||||||
|
`ci-runner/Dockerfile` replaces the stock `node:20-bookworm` image most
|
||||||
|
`docker`-labeled workflow jobs run in. That base is leftover from when the
|
||||||
|
frontend was Python/Jinja and the old GitHub runner also built JS assets for
|
||||||
|
it — the frontend is Go now, nothing in `.forgejo/workflows/` uses npm/node.
|
||||||
|
Every `docker`-labeled build-push job also currently re-installs the Docker
|
||||||
|
CLI on each run (`apt-get install docker.io`), which pulls in the classic
|
||||||
|
builder rather than BuildKit (the classic builder mishandles `COPY
|
||||||
|
--chown=<name>` group resolution — a real bug hit during the frontend Go
|
||||||
|
rewrite). `ci-runner` is a slim Debian base with `docker-ce-cli` +
|
||||||
|
`docker-buildx-plugin` (BuildKit) preinstalled, plus `git`/`python3`/
|
||||||
|
`python3-yaml` for the other jobs that need them (`shell-lint`,
|
||||||
|
`observability-validate`).
|
||||||
|
|
||||||
|
Built and verified locally; **not yet pushed** — pushing to
|
||||||
|
`git.thermograph.org/emi/thermograph/ci-runner` needs a PAT with
|
||||||
|
`write:package` scope (the embedded git-remote token lacks it, same
|
||||||
|
requirement documented in `backend-build-push.yml`). Once pushed:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker build -t git.thermograph.org/emi/thermograph/ci-runner:v1 deploy/forgejo/ci-runner
|
||||||
|
docker push git.thermograph.org/emi/thermograph/ci-runner:v1
|
||||||
|
```
|
||||||
|
|
||||||
|
then cut the live runner over by editing the `labels` array in
|
||||||
|
`~/forgejo-runner/.runner` on the desktop (same runner id/token, no
|
||||||
|
re-registration) and updating `register-lan-runner.sh`'s `LABELS` default so
|
||||||
|
future re-provisioning matches. Only after that cutover is verified working
|
||||||
|
should the now-redundant `apt-get install docker.io` / `python3-yaml` steps be
|
||||||
|
removed from the workflows that had them — removing them first would break
|
||||||
|
every job still running on `node:20-bookworm`.
|
||||||
|
|
||||||
## Why Postgres here and not the Thermograph app's TimescaleDB
|
## Why Postgres here and not the Thermograph app's TimescaleDB
|
||||||
|
|
||||||
Separate instance, separate network (`forgejo_net`, not the app's compose
|
Separate instance, separate network (`forgejo_net`, not the app's compose
|
||||||
|
|
|
||||||
41
infra/deploy/forgejo/ci-runner/Dockerfile
Normal file
41
infra/deploy/forgejo/ci-runner/Dockerfile
Normal file
|
|
@ -0,0 +1,41 @@
|
||||||
|
# Job-container image for the LAN Forgejo Actions runner's `docker`-labeled
|
||||||
|
# jobs (see ../register-lan-runner.sh) — used by every backend/frontend
|
||||||
|
# build-push, deploy, and validation workflow that needs `docker build`.
|
||||||
|
#
|
||||||
|
# Replaces the stock node:20-bookworm image. That base is pure leftover from
|
||||||
|
# when the frontend was a Python/Jinja app the old GitHub runner also built
|
||||||
|
# JS assets for; the frontend is Go now (no npm/node anywhere in
|
||||||
|
# .forgejo/workflows/) so carrying a full Node runtime here just adds weight.
|
||||||
|
# Every workflow using the `docker` label also currently re-provisions the
|
||||||
|
# Docker CLI itself on each run via `apt-get install docker.io` — that step
|
||||||
|
# costs ~15-20s per job and, more importantly, installs the CLASSIC Docker
|
||||||
|
# builder rather than BuildKit, which mishandles `COPY --chown=<name>` group
|
||||||
|
# resolution on images without a matching system group (a real bug hit during
|
||||||
|
# the frontend Go rewrite). Installing docker-buildx-plugin here makes
|
||||||
|
# BuildKit the default, closing that bug class, on a much smaller base.
|
||||||
|
#
|
||||||
|
# Build/push (manual — see ../README.md for when to rebuild):
|
||||||
|
# docker build -t git.thermograph.org/emi/thermograph/ci-runner:vN \
|
||||||
|
# infra/deploy/forgejo/ci-runner
|
||||||
|
# docker push git.thermograph.org/emi/thermograph/ci-runner:vN
|
||||||
|
#
|
||||||
|
# Cutting over the live runner to a new tag means editing the `labels` array
|
||||||
|
# in ~/forgejo-runner/.runner on the runner host directly (same runner
|
||||||
|
# id/token, no re-registration needed) and updating register-lan-runner.sh's
|
||||||
|
# LABELS default so future re-provisioning picks it up too.
|
||||||
|
FROM debian:bookworm-slim
|
||||||
|
|
||||||
|
RUN apt-get update -qq \
|
||||||
|
&& apt-get install -y -qq --no-install-recommends \
|
||||||
|
ca-certificates curl gnupg git python3 python3-yaml \
|
||||||
|
&& install -m 0755 -d /etc/apt/keyrings \
|
||||||
|
&& curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc \
|
||||||
|
&& chmod a+r /etc/apt/keyrings/docker.asc \
|
||||||
|
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian bookworm stable" \
|
||||||
|
> /etc/apt/sources.list.d/docker.list \
|
||||||
|
&& apt-get update -qq \
|
||||||
|
&& apt-get install -y -qq --no-install-recommends docker-ce-cli docker-buildx-plugin \
|
||||||
|
&& rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/docker.list
|
||||||
|
|
||||||
|
RUN docker --version && docker buildx version && git --version \
|
||||||
|
&& python3 -c "import yaml; print('PyYAML', yaml.__version__)"
|
||||||
|
|
@ -48,6 +48,10 @@ services:
|
||||||
deploy:
|
deploy:
|
||||||
placement:
|
placement:
|
||||||
constraints: [node.labels.role == forge]
|
constraints: [node.labels.role == forge]
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
cpus: "${FORGEJO_DB_CPUS:-1}"
|
||||||
|
memory: ${FORGEJO_DB_MEMORY:-1g}
|
||||||
restart_policy:
|
restart_policy:
|
||||||
condition: on-failure
|
condition: on-failure
|
||||||
|
|
||||||
|
|
@ -131,6 +135,10 @@ services:
|
||||||
deploy:
|
deploy:
|
||||||
placement:
|
placement:
|
||||||
constraints: [node.labels.role == forge]
|
constraints: [node.labels.role == forge]
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
cpus: "${FORGEJO_CPUS:-2}"
|
||||||
|
memory: ${FORGEJO_MEMORY:-2g}
|
||||||
restart_policy:
|
restart_policy:
|
||||||
condition: on-failure
|
condition: on-failure
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue