7ff3873661
Bring the Swarm+Forgejo layer in line with the canonical topology in docs/runbooks/implementation-handoff.md: three nodes (prod, beta, and the desktop LAN dev machine) instead of two, with the Forgejo Actions runner registered on the desktop as a plain systemd service rather than a Swarm-scheduled Docker-in-Docker sidecar. - setup-wireguard.sh: full N-peer mesh instead of point-to-point - docker-stack.yml: drop the runner/runner-dind services, volumes, and runner-token secret; only forgejo_db_password remains - register-lan-runner.sh: register under both docker and thermograph-lan labels, since one runner now covers both job types - init-swarm.sh, join-swarm.sh, swarm/README.md, forgejo/README.md, INFRA.md: updated node lists, order of operations, and access-state table for three nodes - deploy.yml: renamed to "Deploy to beta VPS" and reconciled the main-vs-release branch question against terraform.tfvars.example (this workflow already targets beta; a release-triggered prod deploy doesn't exist yet and isn't invented here)
83 lines
4.1 KiB
Markdown
83 lines
4.1 KiB
Markdown
# 3-node Docker Swarm (prod + beta + desktop), for hosting Forgejo
|
|
|
|
This Swarm cluster's only job is to run Forgejo (`deploy/forgejo/`) — it does
|
|
**not** orchestrate the Thermograph app itself, which stays on the
|
|
Terraform-managed `docker compose` deploys on prod/beta independently (see
|
|
`terraform/README.md`). Keeping those separate means nothing here can strand
|
|
or interfere with the app's already-working, single-writer Postgres/TimescaleDB
|
|
deploys.
|
|
|
|
This is the canonical topology from
|
|
`docs/runbooks/implementation-handoff.md` (Track B steps 2-3) — three nodes,
|
|
not two. An earlier revision of this doc/scripts covered just prod+beta;
|
|
the desktop (this LAN dev machine) joins too.
|
|
|
|
**Nodes:**
|
|
- **manager** — prod, the new 48 GB / 12-core box (more headroom).
|
|
- **worker** — beta, the old VPS (`75.119.132.91`).
|
|
- **worker** — desktop, this LAN dev machine (also runs the Forgejo Actions
|
|
runner as a plain systemd service — see `deploy/forgejo/README.md` — not as
|
|
a Swarm-scheduled container).
|
|
|
|
One manager, not more: Raft needs 3 nodes for real quorum-based HA, and this
|
|
cluster only has 3 nodes total, so making even one more of them a manager
|
|
would still fall short of real HA while adding split-brain risk. If the
|
|
manager (prod) goes down, the workers keep running whatever was already
|
|
scheduled on them (Forgejo, pinned to beta) but the cluster can't reschedule
|
|
anything until prod's back — acceptable for a small cluster whose only job is
|
|
CI/CD.
|
|
|
|
## Order of operations
|
|
|
|
1. **Agent access first** (`deploy/provision-agent-access.sh`) on prod and
|
|
beta — everything below on those two boxes is run through that access. The
|
|
desktop is wherever you're already working from; no separate access step
|
|
needed there.
|
|
2. **WireGuard mesh** (`setup-wireguard.sh <my_wg_ip> <peers_file>`) — run on
|
|
**all three** nodes. See the script's header for the peer-list format and
|
|
the two-pass key-exchange dance (pubkeys aren't known until every node has
|
|
run it once). Verify with `ping <peer_wg_ip>` to each of the other two
|
|
before continuing.
|
|
3. **Swarm init** (`init-swarm.sh <manager_wg_ip>`) on the manager (prod) only.
|
|
4. **Swarm join** (`join-swarm.sh <manager_wg_ip> <token>`) on **each** of the
|
|
two workers (beta, desktop) — same token for both.
|
|
5. **Firewall lockdown** (`firewall-swarm.sh`) on **all three** nodes — closes
|
|
2377/7946/4789 to everything except the WireGuard interface. Do this
|
|
*after* joining is confirmed working on all three, not before (locking the
|
|
ports first would make the join itself fail).
|
|
6. **Label beta** (`label-forge-node.sh <beta-node-name>`) on the manager —
|
|
`docker node ls` shows each node's name/ID. Only beta gets `role=forge`;
|
|
the desktop and prod don't need a Swarm label for anything in this setup.
|
|
7. Deploy Forgejo: see `deploy/forgejo/README.md`.
|
|
8. Register the Actions runner **on the desktop** (not through Swarm):
|
|
`deploy/forgejo/register-lan-runner.sh`.
|
|
|
|
## Why WireGuard instead of relying on Swarm's built-in TLS alone
|
|
|
|
Swarm's control plane (port 2377) is TLS-encrypted and mutually authenticated
|
|
by default. Its overlay data plane (VXLAN, port 4789) is **not** encrypted by
|
|
default, and Docker's own guidance is that port must never face the public
|
|
internet — these nodes are on different networks (two separate providers'
|
|
public IPs, plus a home/LAN connection for the desktop), not one private LAN,
|
|
so the tunnel is the network boundary the Swarm ports advertise into, rather
|
|
than trusting the public internet (or the desktop's home network) directly.
|
|
|
|
## Verifying
|
|
|
|
```bash
|
|
# On the manager:
|
|
docker node ls # all three nodes Ready
|
|
docker node inspect <beta-node> --format '{{.Spec.Labels}}' # role:forge
|
|
|
|
# From a FOURTH machine outside the mesh entirely, confirm the Swarm ports
|
|
# are NOT reachable on either VPS's public IP (the desktop has no public IP
|
|
# to check this way):
|
|
nc -zv -w2 <prod_or_beta_public_ip> 2377 # should fail/timeout
|
|
nc -zvu -w2 <prod_or_beta_public_ip> 4789 # should fail/timeout
|
|
```
|
|
|
|
## Adding a node label back out (undo)
|
|
|
|
```bash
|
|
docker node update --label-rm role <beta-node>
|
|
```
|