diff --git a/INFRA.md b/INFRA.md new file mode 100644 index 0000000..fc2815b --- /dev/null +++ b/INFRA.md @@ -0,0 +1,96 @@ +# Agent access, Docker Swarm, and Forgejo CI/CD + +This covers three additive infrastructure layers on top of the two VPS boxes +Terraform already provisions (see `terraform/README.md` and +`terraform.tfvars.example`, where they're named **prod** — the new 48 GB / +12-core box, `thermograph.org` — and **beta** — the old VPS, +`75.119.132.91`). **None of this touches `backend/`, `Dockerfile`, +`docker-compose*.yml`, `terraform/`, or `deploy/db/`** — those stay owned by +the app-containerization work; this is strictly the layer above it. + +``` +Track 1: deploy/provision-agent-access.sh — a dedicated full-root login for me +Track 2: deploy/swarm/ — Swarm cluster spanning both boxes +Track 3: deploy/forgejo/ + .forgejo/ — Forgejo + Forgejo Actions, replacing GitHub +``` + +## Track 1 — Agent access + +Run `sudo bash deploy/provision-agent-access.sh` on **both** boxes. Creates a +dedicated `agent` user (not raw root login — a distinct name gives a clean +audit trail) with passwordless sudo, installs the agent's public key, disables +SSH password auth fleet-wide, and turns on `auditd` logging of every +root-effective command. See the script's header comment for the full +rationale and revocation steps (one line to delete, or delete the whole +account — your own access is never affected). + +## Track 2 — Swarm + +See `deploy/swarm/README.md` for the exact order of operations (WireGuard +tunnel first, then swarm init/join, then lock the Swarm ports down to the +tunnel interface, then label beta for Forgejo placement). This cluster's only +job is hosting Forgejo — it does not orchestrate the Terraform-managed app +deploys. + +## Track 3 — Forgejo, replacing GitHub + +### 3a. Stand up Forgejo +See `deploy/forgejo/README.md` — deploys the stack, walks through minting the +two Swarm secrets it needs (`forgejo_db_password`, `forgejo_runner_token`). + +### 3b. Migrate the repo (mirror first, verify, then cut over) +1. In Forgejo: **+ New Migration → GitHub**. Point it at + `https://github.com/griffemi/thermograph`, pull code + issues + PRs + + releases + labels. This is non-destructive to GitHub — do this as a mirror + and leave GitHub live and untouched until you're confident. +2. Spot-check: branch list matches, a few PRs/issues render correctly, LFS (if + any) came across. +3. **Don't retarget any secret or disable a GitHub workflow yet** — see 3d. + +### 3c. Workflows +`.forgejo/workflows/*.yml` already mirror `.github/workflows/*.yml` in this +PR — copied with the mechanical changes Forgejo needs: +- `runs-on: ubuntu-latest` → `runs-on: docker` (the label the Swarm-hosted + runner in `deploy/forgejo/docker-stack.yml` registers under). +- `appleboy/ssh-action` referenced by full GitHub URL (not mirrored in + Forgejo's default action registry, unlike `actions/checkout` / + `actions/setup-python`, which resolve unchanged). +- **The custom auto-merge workflow step is gone.** It existed only because + GitHub's native branch protection/auto-merge is paywalled on private + free-tier repos. Forgejo has no such tier — configure it directly: + **Settings → Branches → protect `dev` → Enable Status Check → list `build` + → merge style: squash.** Then a ready PR gets a real "Auto merge when checks + succeed" button. +- One knock-on simplification: a native auto-merge is an ordinary git push + (unlike GitHub's token-based merge commit, which doesn't re-trigger `push`), + so `deploy-dev.yml`'s push trigger fires naturally after auto-merge — no + more "direct push vs. PR-merge" double-trigger logic to maintain. +- `.forgejo/workflows/deploy.yml` still targets `main` faithfully mirroring + the current GitHub workflow, but `terraform.tfvars.example` describes prod's + branch as `release`. That mismatch is flagged in the file's own header + comment — reconcile with whoever's driving the Terraform/deploy side rather + than guessing here. + +### 3d. Cut over +Only after 3a–3c are verified: +1. Move the deploy secrets (`SSH_HOST`, `SSH_USER`, `SSH_KEY`, `SSH_PORT`) into + Forgejo's repo Secrets (Settings → Actions → Secrets). +2. Re-point the LAN dev runner: `bash deploy/forgejo/register-lan-runner.sh + ` — stops/disables the old GitHub runner service on + that same machine and registers `forgejo-runner` in its place, same + `systemd --user` pattern as before (see `DEPLOY-DEV.md` for the original; + this supersedes it for the runner specifically). +3. Open a test PR against `dev` in Forgejo, confirm the required check runs, + auto-merge fires, and the LAN deploy lands. +4. Only once that's solid: disable/archive the `.github/workflows/*.yml` + triggers (or the whole GitHub repo) — GitHub was the fallback during this + whole migration; retire it deliberately, not by accident. + +## Verification checklist +- [ ] `ssh agent@` and `ssh agent@` both work; `sudo whoami` → `root` +- [ ] Password SSH auth confirmed dead on both boxes +- [ ] `docker node ls` (from prod) shows both nodes `Ready` +- [ ] Swarm ports (2377/7946/4789) unreachable from outside the WireGuard tunnel +- [ ] `https://git.thermograph.org` (or your chosen domain) serves Forgejo over TLS +- [ ] A test PR: required check runs → auto-merges → LAN dev deploy lands +- [ ] GitHub retired only after all of the above hold diff --git a/deploy/forgejo/README.md b/deploy/forgejo/README.md new file mode 100644 index 0000000..635a7db --- /dev/null +++ b/deploy/forgejo/README.md @@ -0,0 +1,74 @@ +# Forgejo on the Swarm cluster + +Runs as `deploy/forgejo/docker-stack.yml` — the only workload this Swarm +cluster carries (the Thermograph app itself stays on the Terraform-managed +`docker compose` deploys; see `terraform/README.md`). Pinned to the **beta** +node (old VPS) via the `role=forge` label from +`deploy/swarm/label-forge-node.sh`. + +## Prerequisites + +1. Both boxes have joined the swarm (`deploy/swarm/`) and beta is labeled + `role=forge`. +2. `docker node ls` (from the manager) shows both `Ready`. + +## One-time setup: Swarm secrets + +Two secrets the stack expects to already exist (Swarm secrets, not files — +`external: true` in the stack file, so `docker stack deploy` never creates or +sees the values, only references them): + +```bash +# A strong random password for Forgejo's own Postgres (NOT related to +# Thermograph's app database — entirely separate instance/network). +openssl rand -base64 32 | docker secret create forgejo_db_password - + +# The runner registration token. Forgejo can't issue one before it's running, +# so this is a two-step dance the first time: +docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo # 1. bring Forgejo up (runner will crashloop briefly — expected) +# 2. once Forgejo answers at the domain, log in, go to +# Site Administration -> Actions -> Runners -> Create new Runner +# (or, for a repo-scoped runner: -> Settings -> Actions -> Runners), +# copy the token, then: +echo -n "PASTE_TOKEN_HERE" | docker secret create forgejo_runner_token - +docker service update --force forgejo_runner # picks up the new secret and registers +``` + +## Deploy / update + +```bash +docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo +``` + +Re-running is safe — Swarm only touches services whose spec actually changed. + +## DNS + +Point the Forgejo domain (default `git.thermograph.org`; override with +`FORGEJO_DOMAIN=...` before `docker stack deploy`, Swarm reads it from the +deploying shell's environment) at **either** node's public IP — the routing +mesh forwards published ports to wherever the task actually landed. + +## Why Postgres here and not the Thermograph app's TimescaleDB + +Separate instance, separate network (`forgejo_net`, not the app's compose +network), separate volume. Forgejo is a distinct product with its own schema +and its own backup/restore lifecycle — sharing a database with the app would +couple two things that should be able to fail, migrate, and restore +independently. + +## Verifying + +```bash +docker service ls # all forgejo_* services Running, 1/1 +curl -I https://git.thermograph.org/ # 200, valid cert +docker service logs forgejo_runner --tail 50 # "runner: successfully registered" then idle +``` + +## Rollback / removal + +```bash +docker stack rm forgejo +# volumes (forgejo_data, forgejo_db, ...) survive a stack rm — remove them +# explicitly only if you actually want to destroy the Forgejo instance's data. +``` diff --git a/deploy/forgejo/docker-stack.yml b/deploy/forgejo/docker-stack.yml new file mode 100644 index 0000000..78873cb --- /dev/null +++ b/deploy/forgejo/docker-stack.yml @@ -0,0 +1,174 @@ +# Forgejo (self-hosted Git + CI/CD) as a Docker Swarm stack — the only workload +# this Swarm cluster runs (see deploy/swarm/README.md). Deliberately separate +# from the Terraform-managed docker-compose.yml that runs the Thermograph app +# itself: this stack's only job is Forgejo, its runner, and the ingress in +# front of it. +# +# Deploy from the manager node (prod), after both boxes have joined the swarm +# and beta is labeled role=forge: +# +# docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo +# +# Requires these Swarm secrets to exist first (see deploy/forgejo/README.md): +# forgejo_db_password, forgejo_runner_token +# +# DNS: point git.thermograph.org (or your chosen domain — see plan) at EITHER +# node's public IP. Swarm's routing mesh accepts published ports on every node +# and forwards to wherever the task actually runs, so either IP works. + +services: + db: + image: postgres:16-alpine + environment: + POSTGRES_USER: forgejo + POSTGRES_DB: forgejo + POSTGRES_PASSWORD_FILE: /run/secrets/forgejo_db_password + secrets: [forgejo_db_password] + volumes: + - forgejo_db:/var/lib/postgresql/data + networks: [forgejo_net] + deploy: + placement: + constraints: [node.labels.role == forge] + restart_policy: + condition: on-failure + + forgejo: + image: codeberg.org/forgejo/forgejo:9-rootless + depends_on: [db] + environment: + FORGEJO__database__DB_TYPE: postgres + FORGEJO__database__HOST: db:5432 + FORGEJO__database__NAME: forgejo + FORGEJO__database__USER: forgejo + FORGEJO__server__DOMAIN: "${FORGEJO_DOMAIN:-git.thermograph.org}" + FORGEJO__server__ROOT_URL: "https://${FORGEJO_DOMAIN:-git.thermograph.org}/" + FORGEJO__server__SSH_PORT: "2222" + # Actions on: repo/org/user-level runners register against this instance. + FORGEJO__actions__ENABLED: "true" + # Gitea/Forgejo's app.ini env-mapping honors a __FILE suffix to read a + # value from a file instead of the literal env var — same convention as + # the official Postgres image's POSTGRES_PASSWORD_FILE above. + FORGEJO__database__PASSWD__FILE: /run/secrets/forgejo_db_password + FORGEJO__database__USER: forgejo + secrets: + - source: forgejo_db_password + target: forgejo_db_password + volumes: + - forgejo_data:/var/lib/gitea + networks: [forgejo_net] + ports: + # SSH for git@ clones — Swarm's routing mesh publishes this on both nodes. + - target: 2222 + published: 2222 + protocol: tcp + mode: host + deploy: + placement: + constraints: [node.labels.role == forge] + restart_policy: + condition: on-failure + labels: + - traefik.enable=true + - traefik.http.routers.forgejo.rule=Host(`${FORGEJO_DOMAIN:-git.thermograph.org}`) + - traefik.http.routers.forgejo.entrypoints=websecure + - traefik.http.routers.forgejo.tls.certresolver=le + - traefik.http.services.forgejo.loadbalancer.server.port=3000 + + # Docker-in-Docker sidecar for the runner. Chosen over mounting the host + # docker.sock (root-equivalent host access, zero isolation) — DinD is the + # documented mitigation for a trusted single-maintainer runner that only + # ever executes this repo's own workflows. + runner-dind: + image: docker:27-dind + privileged: true + environment: + DOCKER_TLS_CERTDIR: "" + volumes: + - runner_dind_certs:/certs + networks: [forgejo_net] + deploy: + placement: + constraints: [node.labels.role == forge] + restart_policy: + condition: on-failure + + runner: + image: code.forgejo.org/forgejo/runner:6 + depends_on: [runner-dind, forgejo] + environment: + DOCKER_HOST: tcp://runner-dind:2375 + FORGEJO_INSTANCE_URL: "https://${FORGEJO_DOMAIN:-git.thermograph.org}" + secrets: + - source: forgejo_runner_token + target: forgejo_runner_token + # Registers on first start using the token secret (see README for how to + # mint one), then just executes jobs — no docker.sock mount. + entrypoint: > + sh -c ' + if [ ! -f /data/.runner ]; then + forgejo-runner register --no-interactive + --instance "$$FORGEJO_INSTANCE_URL" + --token "$$(cat /run/secrets/forgejo_runner_token)" + --name swarm-runner --labels docker:docker://node:20-bookworm; + fi; + forgejo-runner daemon + ' + volumes: + - runner_data:/data + networks: [forgejo_net] + deploy: + placement: + constraints: [node.labels.role == forge] + restart_policy: + condition: on-failure + + traefik: + image: traefik:v3.1 + command: + - --providers.swarm.endpoint=unix:///var/run/docker.sock + - --providers.swarm.exposedbydefault=false + - --entrypoints.web.address=:80 + - --entrypoints.websecure.address=:443 + - --entrypoints.web.http.redirections.entrypoint.to=websecure + - --entrypoints.web.http.redirections.entrypoint.scheme=https + - --certificatesresolvers.le.acme.email=${ACME_EMAIL:-admin@thermograph.org} + - --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json + - --certificatesresolvers.le.acme.httpchallenge.entrypoint=web + ports: + - target: 80 + published: 80 + mode: host + - target: 443 + published: 443 + mode: host + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - traefik_certs:/letsencrypt + networks: [forgejo_net] + deploy: + placement: + # Pinned to the same node as the ACME cert store volume — a floating + # Traefik would re-request certs from Let's Encrypt on every + # reschedule and hit its rate limit. + constraints: [node.labels.role == forge] + restart_policy: + condition: on-failure + +networks: + forgejo_net: + driver: overlay + attachable: false + +volumes: + forgejo_db: + forgejo_data: + runner_data: + runner_dind_certs: + traefik_certs: + +secrets: + forgejo_db_password: + external: true + forgejo_runner_token: + external: true diff --git a/deploy/forgejo/register-lan-runner.sh b/deploy/forgejo/register-lan-runner.sh new file mode 100755 index 0000000..eccf731 --- /dev/null +++ b/deploy/forgejo/register-lan-runner.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# Re-points the existing LAN dev self-hosted runner from GitHub Actions to +# Forgejo Actions. Run on the SAME machine that already runs the GitHub +# runner (see DEPLOY-DEV.md) — this replaces that runner, it doesn't add a +# second one. Sudo-free, systemd --user, same pattern as the app service. +# +# bash deploy/forgejo/register-lan-runner.sh +# +# Get from the Forgejo web UI: +# repo -> Settings -> Actions -> Runners -> Create new Runner +# (or an org/instance-level runner page, if you want it to serve more than +# this one repo — same as the GitHub runner did). +set -euo pipefail + +FORGEJO_URL="${1:?usage: $0 }" +TOKEN="${2:?}" +RUNNER_DIR="${RUNNER_DIR:-$HOME/forgejo-runner}" +LABELS="${LABELS:-thermograph-lan}" + +echo "==> Stopping and disabling the old GitHub Actions runner service, if present" +systemctl --user stop github-actions-runner 2>/dev/null || true +systemctl --user disable github-actions-runner 2>/dev/null || true + +echo "==> Installing forgejo-runner into $RUNNER_DIR" +mkdir -p "$RUNNER_DIR" +cd "$RUNNER_DIR" +if [ ! -x ./forgejo-runner ]; then + ARCH="$(uname -m)" + case "$ARCH" in + x86_64) BIN_ARCH=amd64 ;; + aarch64) BIN_ARCH=arm64 ;; + *) echo "Unsupported arch: $ARCH — download the right binary by hand from" \ + "https://code.forgejo.org/forgejo/runner/releases" >&2; exit 1 ;; + esac + VER="${FORGEJO_RUNNER_VERSION:-6.3.1}" + curl -fsSL -o forgejo-runner \ + "https://code.forgejo.org/forgejo/runner/releases/download/v${VER}/forgejo-runner-${VER}-linux-${BIN_ARCH}" + chmod +x forgejo-runner +fi + +echo "==> Registering with $FORGEJO_URL (label: $LABELS)" +./forgejo-runner register --no-interactive \ + --instance "$FORGEJO_URL" \ + --token "$TOKEN" \ + --name "thermograph-lan-$(hostname -s)" \ + --labels "$LABELS" + +echo "==> systemd --user unit" +mkdir -p "$HOME/.config/systemd/user" +cat > "$HOME/.config/systemd/user/forgejo-runner.service" </dev/null || true + +cat <&2 + exit 1 +fi + +echo "==> User: $AGENT_USER" +if ! id "$AGENT_USER" >/dev/null 2>&1; then + adduser --disabled-password --gecos "" "$AGENT_USER" +fi + +echo "==> Passwordless sudo (full, by design — see header comment)" +echo "$AGENT_USER ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/"$AGENT_USER" +chmod 440 /etc/sudoers.d/"$AGENT_USER" +visudo -cf /etc/sudoers.d/"$AGENT_USER" # fail loudly on a syntax error rather than silently locking sudo + +echo "==> Authorized key" +install -d -m 700 -o "$AGENT_USER" -g "$AGENT_USER" /home/"$AGENT_USER"/.ssh +# Replace wholesale rather than append, so re-running after a key rotation +# doesn't accumulate stale keys. +echo "$AGENT_PUBKEY" > /home/"$AGENT_USER"/.ssh/authorized_keys +chown "$AGENT_USER":"$AGENT_USER" /home/"$AGENT_USER"/.ssh/authorized_keys +chmod 600 /home/"$AGENT_USER"/.ssh/authorized_keys + +echo "==> sshd hardening (fleet-wide, not just for this account)" +SSHD_DROPIN=/etc/ssh/sshd_config.d/99-agent-hardening.conf +cat > "$SSHD_DROPIN" <<'EOF' +# Managed by deploy/provision-agent-access.sh — key-only login, no passwords. +# Root itself still can't password in; PermitRootLogin prohibit-password +# leaves direct root key-login available for a human who already has a key +# there, without opening a password path for anyone. +PasswordAuthentication no +KbdInteractiveAuthentication no +PermitRootLogin prohibit-password +EOF +sshd -t # validate before reload; a bad config here can lock out the whole box +systemctl reload sshd 2>/dev/null || systemctl reload ssh 2>/dev/null || true + +echo "==> auditd: log root-effective commands (survives shell history -c)" +if ! command -v auditctl >/dev/null 2>&1; then + apt-get update -y -q + apt-get install -y -q auditd audispd-plugins +fi +AUDIT_RULES=/etc/audit/rules.d/agent-root-cmds.rules +cat > "$AUDIT_RULES" <<'EOF' +# Every root-effective exec, tagged for `ausearch -k agentcmd`. +-a always,exit -F arch=b64 -S execve -F euid=0 -k agentcmd +-a always,exit -F arch=b32 -S execve -F euid=0 -k agentcmd +EOF +augenrules --load 2>/dev/null || true +systemctl enable --now auditd 2>/dev/null || true + +echo +echo "Done. Verify from your own machine (not this box):" +echo " ssh -i $AGENT_USER@ sudo whoami # -> root" +echo " ssh -o PasswordAuthentication=no @ # password path confirmed dead" +echo +echo "Revoke any time: delete the key line in" +echo " /home/$AGENT_USER/.ssh/authorized_keys" +echo "or remove the account entirely:" +echo " deluser --remove-home $AGENT_USER && rm -f /etc/sudoers.d/$AGENT_USER" diff --git a/deploy/swarm/README.md b/deploy/swarm/README.md new file mode 100644 index 0000000..6cf8ff8 --- /dev/null +++ b/deploy/swarm/README.md @@ -0,0 +1,65 @@ +# 2-node Docker Swarm (prod + beta), for hosting Forgejo + +This Swarm cluster's only job is to run Forgejo (`deploy/forgejo/`) — it does +**not** orchestrate the Thermograph app itself, which stays on the +Terraform-managed `docker compose` deploys on each box independently (see +`terraform/README.md`). Keeping those separate means nothing here can strand +or interfere with the app's already-working, single-writer Postgres/TimescaleDB +deploys. + +**Nodes:** +- **manager** — prod, the new 48 GB / 12-core box (more headroom). +- **worker** — beta, the old VPS (`75.119.132.91`). + +1 manager + 1 worker, not 2 managers: Raft needs 3 nodes for real quorum-based +HA, so a second manager here would add complexity (split-brain risk) without +adding actual failover. If the manager goes down, the worker keeps running +whatever was already scheduled on it (Forgejo, since it's pinned there) but +the cluster can't reschedule anything until the manager's back — acceptable +for a 2-box hobby/small-team cluster whose only job is CI/CD. + +## Order of operations + +1. **Agent access first** (`deploy/provision-agent-access.sh`) on both boxes — + everything below is run through that access. +2. **WireGuard tunnel** (`setup-wireguard.sh`) — run on both boxes; see the + script's header for the two-pass key-exchange dance. Verify with + `ping ` before continuing. +3. **Swarm init** (`init-swarm.sh `) on the manager (prod) only. +4. **Swarm join** (`join-swarm.sh `) on the worker + (beta) only, using the token `init-swarm.sh` printed. +5. **Firewall lockdown** (`firewall-swarm.sh`) on **both** boxes — closes + 2377/7946/4789 to everything except the WireGuard interface. Do this + *after* joining is confirmed working, not before (locking the ports first + would make the join itself fail). +6. **Label beta** (`label-forge-node.sh `) on the manager — + `docker node ls` shows beta's node name/ID. +7. Deploy Forgejo: see `deploy/forgejo/README.md`. + +## Why WireGuard instead of relying on Swarm's built-in TLS alone + +Swarm's control plane (port 2377) is TLS-encrypted and mutually authenticated +by default. Its overlay data plane (VXLAN, port 4789) is **not** encrypted by +default, and Docker's own guidance is that port must never face the public +internet — these two boxes are on different providers' public IPs, not a +private LAN, so the tunnel is the network boundary the Swarm ports advertise +into, rather than trusting the public internet directly. + +## Verifying + +```bash +# On the manager: +docker node ls # both nodes Ready +docker node inspect --format '{{.Spec.Labels}}' # role:forge + +# From outside the WireGuard interface (e.g. your own machine), confirm the +# Swarm ports are NOT reachable on the public IP: +nc -zv -w2 2377 # should fail/timeout +nc -zvu -w2 4789 # should fail/timeout +``` + +## Adding a node label back out (undo) + +```bash +docker node update --label-rm role +``` diff --git a/deploy/swarm/firewall-swarm.sh b/deploy/swarm/firewall-swarm.sh new file mode 100755 index 0000000..ba00b6a --- /dev/null +++ b/deploy/swarm/firewall-swarm.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# Locks the Swarm ports (2377 control, 7946 gossip, 4789 overlay VXLAN) to the +# WireGuard interface only — they must never be reachable from the public +# internet. Run on BOTH boxes after joining the swarm. Existing app-facing +# rules (80/443, SSH, etc.) are untouched. +set -euo pipefail + +WG_IFACE="${WG_IFACE:-wg0}" + +if ! command -v ufw >/dev/null 2>&1; then + echo "ufw not found — apply the equivalent iptables/nftables rules by hand:" >&2 + echo " allow 2377/tcp, 7946/tcp, 7946/udp, 4789/udp only on interface ${WG_IFACE}" >&2 + echo " deny 2377/tcp, 7946/tcp, 7946/udp, 4789/udp on every other interface" >&2 + exit 1 +fi + +echo "==> Allowing Swarm ports on ${WG_IFACE} only" +ufw allow in on "$WG_IFACE" to any port 2377 proto tcp +ufw allow in on "$WG_IFACE" to any port 7946 proto tcp +ufw allow in on "$WG_IFACE" to any port 7946 proto udp +ufw allow in on "$WG_IFACE" to any port 4789 proto udp + +echo "==> Explicitly denying the same ports on every other interface" +ufw deny 2377/tcp +ufw deny 7946/tcp +ufw deny 7946/udp +ufw deny 4789/udp + +echo +echo "Rules added (not yet necessarily active — check ufw status):" +ufw status numbered | grep -E "2377|7946|4789|Status" || true +echo +echo "If ufw is inactive, enable it: ufw enable (make sure your SSH port is" +echo "allowed FIRST, or you will lock yourself out)." diff --git a/deploy/swarm/init-swarm.sh b/deploy/swarm/init-swarm.sh new file mode 100755 index 0000000..1992783 --- /dev/null +++ b/deploy/swarm/init-swarm.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# Run ONCE, on the manager node only (prod — the new 48 GB box). Initializes +# the Swarm advertising the WireGuard address, so cluster traffic never +# touches the public interface. Run setup-wireguard.sh on both boxes first. +set -euo pipefail + +MY_WG_IP="${1:?usage: $0 }" + +if docker info 2>/dev/null | grep -q "Swarm: active"; then + echo "Swarm is already active on this node. Current state:" + docker node ls + echo + echo "Worker join token:" + docker swarm join-token -q worker + exit 0 +fi + +echo "==> docker swarm init, advertising ${MY_WG_IP} (the WireGuard address, not the public IP)" +docker swarm init --advertise-addr "$MY_WG_IP" --listen-addr "${MY_WG_IP}:2377" + +echo +echo "==> Worker join command (run this on the worker/beta node):" +TOKEN="$(docker swarm join-token -q worker)" +echo " docker swarm join --token ${TOKEN} ${MY_WG_IP}:2377" +echo +echo "(Or run deploy/swarm/join-swarm.sh on the worker.)" diff --git a/deploy/swarm/join-swarm.sh b/deploy/swarm/join-swarm.sh new file mode 100755 index 0000000..bd71939 --- /dev/null +++ b/deploy/swarm/join-swarm.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Run ONCE, on the worker node only (beta — the old VPS). Joins the Swarm +# initialized by init-swarm.sh on the manager, over the WireGuard tunnel. +set -euo pipefail + +MANAGER_WG_IP="${1:?usage: $0 }" +JOIN_TOKEN="${2:?}" + +if docker info 2>/dev/null | grep -q "Swarm: active"; then + echo "This node is already part of a swarm:" + docker info 2>/dev/null | grep -A2 "Swarm:" + exit 0 +fi + +docker swarm join --token "$JOIN_TOKEN" "${MANAGER_WG_IP}:2377" + +echo +echo "Joined. Verify from the manager: docker node ls" diff --git a/deploy/swarm/label-forge-node.sh b/deploy/swarm/label-forge-node.sh new file mode 100755 index 0000000..cf07fcf --- /dev/null +++ b/deploy/swarm/label-forge-node.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# Run ONCE, on the manager node, after both boxes have joined the swarm. +# Labels the worker (beta) so the Forgejo stack's placement constraint +# (node.labels.role == forge) schedules there and nowhere else. +set -euo pipefail + +WORKER_HOSTNAME="${1:?usage: $0 (see: docker node ls)}" + +docker node update --label-add role=forge "$WORKER_HOSTNAME" + +echo +echo "Labels on ${WORKER_HOSTNAME}:" +docker node inspect "$WORKER_HOSTNAME" --format '{{.Spec.Labels}}' diff --git a/deploy/swarm/setup-wireguard.sh b/deploy/swarm/setup-wireguard.sh new file mode 100755 index 0000000..4631854 --- /dev/null +++ b/deploy/swarm/setup-wireguard.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# Sets up a point-to-point WireGuard tunnel between the two Swarm hosts. Docker +# Swarm's control plane (2377/tcp) is TLS-encrypted by default, but the overlay +# data plane (VXLAN, 4789/udp) is NOT — and it should never face the public +# internet. Swarm is joined over this tunnel's private IPs instead. +# +# Run on EACH box, in either order. Each run generates that box's own WireGuard +# keypair (if missing) and prints its public key — paste it into the OTHER +# box's invocation. Two runs total, one per box: +# +# box A (prod): bash setup-wireguard.sh 10.10.0.1 10.10.0.2 +# box B (beta): bash setup-wireguard.sh 10.10.0.2 10.10.0.1 +# +# First run on either box: leave the last argument empty, note the printed +# pubkey, then run the second box with that value, then re-run the first box +# once more with the second box's now-known pubkey. (A local key never +# changes across re-runs — only the peer section updates.) +set -euo pipefail + +MY_WG_IP="${1:?usage: $0 [peer_wg_pubkey]}" +PEER_WG_IP="${2:?}" +MY_PUBLIC_IP="${3:?}" +PEER_PUBLIC_IP="${4:?}" +PEER_PUBKEY="${5:-}" + +WG_PORT="${WG_PORT:-51820}" +WG_DIR=/etc/wireguard + +if [ "$(id -u)" -ne 0 ]; then + echo "Run as root (or via the agent user's sudo)." >&2 + exit 1 +fi + +echo "==> Installing WireGuard if needed" +command -v wg >/dev/null 2>&1 || { apt-get update -y -q && apt-get install -y -q wireguard; } + +install -d -m 700 "$WG_DIR" +if [ ! -f "$WG_DIR/privatekey" ]; then + echo "==> Generating this box's WireGuard keypair" + umask 077 + wg genkey | tee "$WG_DIR/privatekey" | wg pubkey > "$WG_DIR/publickey" +fi +MY_PRIVKEY="$(cat "$WG_DIR/privatekey")" +MY_PUBKEY="$(cat "$WG_DIR/publickey")" + +echo +echo "==> This box's WireGuard public key (give this to the OTHER box's run):" +echo " $MY_PUBKEY" +echo + +if [ -z "$PEER_PUBKEY" ]; then + echo "No peer public key supplied yet — writing a config with no [Peer] section." + echo "Run this same command again once you have it (from the other box's output above)." + cat > "$WG_DIR/wg0.conf" < "$WG_DIR/wg0.conf" < Bringing up wg0" +systemctl enable --now wg-quick@wg0 2>/dev/null || (wg-quick down wg0 2>/dev/null; wg-quick up wg0) +# Re-apply if the config changed on an already-up interface. +wg syncconf wg0 <(wg-quick strip wg0) 2>/dev/null || true + +echo "==> Firewall: only let the OTHER box's public IP reach the WireGuard port" +if command -v ufw >/dev/null 2>&1; then + ufw allow from "$PEER_PUBLIC_IP" to any port "$WG_PORT" proto udp comment "wireguard peer" +fi + +echo +echo "wg0 status:" +wg show wg0 || true +echo +if [ -n "$PEER_PUBKEY" ]; then + echo "==> Verify from here once BOTH boxes have run with each other's pubkey:" + echo " ping -c2 ${PEER_WG_IP}" +fi