Route Forgejo through beta's existing Caddy instead of Traefik (#244)

beta (Forgejo's pinned node) is also today's live thermograph.org host,
with Caddy already bound to ports 80/443. A second reverse proxy
binding those same ports would collide with it. Drop Traefik entirely:
forgejo's web port now publishes to 127.0.0.1:3080 only, and a new
Caddy site block (deploy/forgejo/caddy-git.conf) reverse-proxies
git.thermograph.org to it, reusing Caddy's existing automatic-HTTPS
instead of a second ACME flow.

That same Caddy block resolves the registry-exposure question (hazard
#15 in the hop1 runbook) as its first listed option: /v2/* (the
registry API) is blocked to everything outside the WireGuard mesh
CIDR, while the git/web UI stays public. README documents the
/etc/hosts override mesh clients need so registry traffic actually
routes over the tunnel rather than the public IP.

deploy/forgejo/README.md

@@ -37,6 +37,50 @@ docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo
 ```
 
 Re-running is safe — Swarm only touches services whose spec actually changed.
+Do this **before** the DNS + Caddy step below — Caddy's reverse_proxy target
+(`127.0.0.1:3080`) needs the `forgejo` service actually listening first, or
+its first health check just fails harmlessly until it is.
+
+## DNS + TLS: reusing beta's existing Caddy, not a second reverse proxy
+
+Forgejo is pinned to beta (`role=forge`) — but beta is also **today's live
+thermograph.org host**, and its Caddy already owns ports 80/443
+(`/etc/caddy/Caddyfile` on that box). A second ingress (Traefik) trying to
+bind the same ports would collide with it. So there's no Traefik in this
+stack: `forgejo`'s web port publishes to `127.0.0.1:3080` only (host-local),
+and beta's *existing* Caddy gets one more site block reverse-proxying to it —
+same pattern as its `thermograph.org` block, same automatic-HTTPS.
+
+1. Point the Forgejo domain (default `git.thermograph.org`; override with
+   `FORGEJO_DOMAIN=...` before `docker stack deploy`) at **beta's** public IP
+   — that's where the task actually runs, not prod's or the desktop's.
+2. Append `deploy/forgejo/caddy-git.conf` to beta's `/etc/caddy/Caddyfile`,
+   adjusting the domain if you didn't use the default, then `systemctl reload
+   caddy`.
+3. That file also resolves the registry-exposure hazard (#15 in
+   `docs/runbooks/hop1-forgejo-registry-cutover.md`): `/v2/*` (the registry
+   API) is blocked to everything except the WireGuard mesh CIDR
+   (`10.10.0.0/24`); the git/web UI stays public. CI runners and Swarm nodes
+   reach the registry over the mesh, not the public internet — see "Registry
+   access from mesh clients" below.
+
+## Registry access from mesh clients
+
+Any node that needs `docker login`/push/pull against the registry (the
+desktop's CI runner building/pushing images, later any Swarm node pulling
+them) must reach `git.thermograph.org` **over the WireGuard tunnel**, not
+beta's public IP — otherwise Caddy's `/v2/*` block above refuses the
+connection. Public DNS resolves the domain to beta's public IP, so add a
+`/etc/hosts` override on each such node pinning it to beta's WireGuard
+address instead:
+
+```
+echo "10.10.0.2 git.thermograph.org" | sudo tee -a /etc/hosts
+```
+
+(`10.10.0.2` is beta's WG address per `deploy/swarm/README.md`'s peer
+numbering — adjust if you assigned it differently.) The git/web UI keeps
+working normally for everyone else since only `/v2/*` is restricted.
 
 ## Register the Actions runner (on the desktop, not through Swarm)
 
@@ -54,15 +98,6 @@ See that script's header for exactly what it replaces (the pre-Forgejo GitHub
 self-hosted runner on this same machine) and why it registers with two
 labels where there used to be two separate runners.
 
-## DNS
-
-Point the Forgejo domain (default `git.thermograph.org`; override with
-`FORGEJO_DOMAIN=...` before `docker stack deploy`, Swarm reads it from the
-deploying shell's environment) at **prod's or beta's** public IP — not the
-desktop's, which isn't a stable publicly-reachable address. The routing mesh
-forwards published ports to wherever the task actually landed, so either VPS
-IP works.
-
 ## Why Postgres here and not the Thermograph app's TimescaleDB
 
 Separate instance, separate network (`forgejo_net`, not the app's compose
@@ -74,8 +109,9 @@ independently.
 ## Verifying
 
 ```bash
-docker service ls                       # forgejo_db, forgejo_forgejo, forgejo_traefik all Running, 1/1
+docker service ls                       # forgejo_db, forgejo_forgejo both Running, 1/1
-curl -I https://git.thermograph.org/    # 200, valid cert
+curl -I https://git.thermograph.org/    # 200, valid cert (Caddy's, not a new one)
+curl -I https://git.thermograph.org/v2/ # 403 from anywhere off the WireGuard mesh
 # On the desktop, after registering the runner:
 systemctl --user status forgejo-runner   # active, both labels registered
 ```

deploy/forgejo/caddy-git.conf

@@ -0,0 +1,31 @@
+# Append this block to /etc/caddy/Caddyfile on beta (the box this stack's
+# `forgejo` service is pinned to — see docker-stack.yml). Reuses beta's
+# existing Caddy instead of a second reverse proxy/ACME flow: Forgejo
+# publishes its web UI to 127.0.0.1:3080 only (host-local), and this block is
+# the only thing that ever talks to it.
+#
+# Registry exposure (hazard #15, see docker-stack.yml's header comment):
+# git.thermograph.org/v2/* is the built-in OCI registry API. It's blocked
+# from anywhere except the WireGuard mesh (10.10.0.0/24) — reachable to CI
+# runners and Swarm nodes over wg0, not from the public internet. Everything
+# else (web UI, git-over-HTTP, PR pages) stays public like the rest of the
+# site.
+#
+# Update FORGEJO_DOMAIN below to match whatever domain you actually pointed
+# at beta's IP if it's not git.thermograph.org.
+
+git.thermograph.org {
+	encode zstd gzip
+
+	@registry_external {
+		path /v2/*
+		not remote_ip 10.10.0.0/24
+	}
+	respond @registry_external 403
+
+	reverse_proxy 127.0.0.1:3080
+
+	log {
+		output file /var/log/caddy/git.log
+	}
+}

deploy/forgejo/docker-stack.yml

@@ -1,17 +1,23 @@
 # Forgejo (self-hosted Git + CI/CD) as a Docker Swarm stack — the only workload
 # this Swarm cluster runs (see deploy/swarm/README.md). Deliberately separate
 # from the Terraform-managed docker-compose.yml that runs the Thermograph app
-# itself: this stack's only job is Forgejo, its container registry, and the
+# itself: this stack's only job is Forgejo and its container registry.
-# ingress in front of it.
 #
 # The Actions RUNNER is deliberately NOT a service in this stack. Per
 # docs/runbooks/implementation-handoff.md (Track B step 5), the canonical
 # design registers the runner on the desktop node as a plain systemd service
 # (deploy/forgejo/register-lan-runner.sh) — the same place the pre-Forgejo
-# GitHub self-hosted runner already lived, not a Swarm-scheduled container. An
+# GitHub self-hosted runner already lived, not a Swarm-scheduled container.
-# earlier revision of this file ran the runner + a Docker-in-Docker sidecar as
+#
-# Swarm services pinned to beta; that's gone now in favor of the canonical
+# No Traefik here. Forgejo is pinned to beta (node.labels.role == forge)
-# placement.
+# because that's what was chosen, but beta is ALSO today's live thermograph.org
+# host — Caddy already owns its ports 80/443 (see /etc/caddy/Caddyfile on that
+# box). A second reverse proxy binding those same ports would either fail to
+# start or fight Caddy. Instead: forgejo's web port publishes to
+# 127.0.0.1:3080 only (host-local, mode: host), and beta's existing Caddy gets
+# a new site block reverse-proxying git.thermograph.org -> 127.0.0.1:3080,
+# same pattern as its thermograph.org block. TLS is Caddy's existing
+# automatic-HTTPS (HTTP-01), not a second ACME flow.
 #
 # Deploy from the manager node (prod), after all three nodes (prod, beta,
 # desktop) have joined the swarm and beta is labeled role=forge:
@@ -20,19 +26,13 @@
 #
 # Requires this Swarm secret to exist first (see deploy/forgejo/README.md):
 #   forgejo_db_password
-# (forgejo_runner_token is no longer a Swarm secret — see register-lan-runner.sh,
-# which takes the registration token as a plain argument on the desktop instead.)
 #
-# DNS: point the Forgejo domain at prod's or beta's public IP (not the
+# Registry exposure (hazard #15 in docs/runbooks/hop1-forgejo-registry-cutover.md):
-# desktop's — it isn't a stable, publicly-reachable address). Swarm's routing
+# resolved as the runbook's first listed option — serve Git/UI publicly but
-# mesh accepts published ports on every node and forwards to wherever the
+# firewall the /v2/ registry API paths so only WireGuard-mesh clients can
-# task actually runs, so either of the two VPS IPs works.
+# reach them. That's implemented in the Caddy site block (see
-#
+# deploy/forgejo/README.md), not here — nothing in this stack file is
-# Registry exposure (hazard #15 in docs/runbooks/hop1-forgejo-registry-cutover.md)
+# registry-specific, the restriction lives entirely in Caddy's config on beta.
-# is still an open call, not resolved here: this file exposes Forgejo (and its
-# built-in registry) publicly via Traefik/Let's Encrypt. The alternative —
-# mesh-only registry + DNS-01 ACME + an outbound pull-mirror — is a deliberate
-# design decision for whoever executes Track B, not something to guess at here.
 
 services:
   db:
@@ -76,54 +76,25 @@ services:
       - forgejo_data:/var/lib/gitea
     networks: [forgejo_net]
     ports:
-      # SSH for git@ clones — Swarm's routing mesh publishes this on both nodes.
+      # SSH for git@ clones — published on whichever node the task lands on
+      # (pinned to beta by the placement constraint below, so effectively
+      # always beta's public IP, port 2222).
       - target: 2222
         published: 2222
         protocol: tcp
         mode: host
+      # Web UI/API — host-local only. Not reachable from outside beta itself;
+      # beta's own Caddy reverse-proxies to it (see deploy/forgejo/README.md).
+      - target: 3000
+        published: 3080
+        protocol: tcp
+        mode: host
+        host_ip: 127.0.0.1
     deploy:
       placement:
         constraints: [node.labels.role == forge]
       restart_policy:
         condition: on-failure
-      labels:
-        - traefik.enable=true
-        - traefik.http.routers.forgejo.rule=Host(`${FORGEJO_DOMAIN:-git.thermograph.org}`)
-        - traefik.http.routers.forgejo.entrypoints=websecure
-        - traefik.http.routers.forgejo.tls.certresolver=le
-        - traefik.http.services.forgejo.loadbalancer.server.port=3000
-
-  traefik:
-    image: traefik:v3.1
-    command:
-      - --providers.swarm.endpoint=unix:///var/run/docker.sock
-      - --providers.swarm.exposedbydefault=false
-      - --entrypoints.web.address=:80
-      - --entrypoints.websecure.address=:443
-      - --entrypoints.web.http.redirections.entrypoint.to=websecure
-      - --entrypoints.web.http.redirections.entrypoint.scheme=https
-      - --certificatesresolvers.le.acme.email=${ACME_EMAIL:-admin@thermograph.org}
-      - --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
-      - --certificatesresolvers.le.acme.httpchallenge.entrypoint=web
-    ports:
-      - target: 80
-        published: 80
-        mode: host
-      - target: 443
-        published: 443
-        mode: host
-    volumes:
-      - /var/run/docker.sock:/var/run/docker.sock:ro
-      - traefik_certs:/letsencrypt
-    networks: [forgejo_net]
-    deploy:
-      placement:
-        # Pinned to the same node as the ACME cert store volume — a floating
-        # Traefik would re-request certs from Let's Encrypt on every
-        # reschedule and hit its rate limit.
-        constraints: [node.labels.role == forge]
-      restart_policy:
-        condition: on-failure
 
 networks:
   forgejo_net:
@@ -133,7 +104,6 @@ networks:
 volumes:
   forgejo_db:
   forgejo_data:
-  traefik_certs:
 
 secrets:
   forgejo_db_password: