d138f00a20
Some checks failed
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Failing after 6s
secrets-guard / encrypted (push) Successful in 6s
shell-lint / shellcheck (push) Successful in 13s
Validate observability stack / validate (push) Successful in 17s
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Successful in 18s
PR build (required check) / gate (pull_request) Successful in 2s
90 lines
4.6 KiB
Markdown
90 lines
4.6 KiB
Markdown
# 3-node Docker Swarm (vps2 + vps1 + desktop)
|
|
|
|
This Swarm mesh's only **Swarm-scheduled** workload is Forgejo
|
|
(`deploy/forgejo/`), pinned to vps1. It does **not** orchestrate the
|
|
Thermograph app the same way — prod and beta each run as their own `docker
|
|
stack deploy` (`deploy/stack/thermograph-stack.yml` /
|
|
`thermograph-beta-stack.yml`) that happens to land on this same manager node
|
|
(vps2) because their volumes are local to it today. Keeping Forgejo's stack
|
|
and the app stacks conceptually separate means nothing here can strand or
|
|
interfere with the app's single-writer Postgres/TimescaleDB.
|
|
|
|
This is the canonical topology from
|
|
`thermograph-docs/runbooks/implementation-handoff.md` (Track B steps 2-3) — three nodes,
|
|
not two. The desktop (formerly the LAN dev machine) joins as a worker for flex
|
|
capacity and AI-model hosting; it hosts no Thermograph environment.
|
|
|
|
**Nodes:**
|
|
- **manager** — vps2 (`169.58.46.181`), the box with headroom for prod's and
|
|
beta's Swarm stacks and their local volumes.
|
|
- **worker** — vps1 (`75.119.132.91`), pinned to run Forgejo
|
|
(`node.labels.role == forge`). Also runs Grafana/Loki/Alloy and the dev
|
|
environment, both outside this Swarm cluster (plain Docker/compose on the
|
|
same host's daemon, not Swarm-scheduled).
|
|
- **worker** — desktop (this machine), flex capacity plus AI-model hosting. No
|
|
Thermograph environment runs here.
|
|
|
|
One manager, not more: Raft needs 3 nodes for real quorum-based HA, and this
|
|
cluster only has 3 nodes total, so making even one more of them a manager
|
|
would still fall short of real HA while adding split-brain risk. If the
|
|
manager (vps2) goes down, the workers keep running whatever was already
|
|
scheduled on them (Forgejo, pinned to vps1) but the cluster can't reschedule
|
|
anything until vps2's back — acceptable for a small cluster whose only
|
|
Swarm-scheduled job is CI/CD.
|
|
|
|
## Order of operations
|
|
|
|
1. **Agent access first** (`deploy/provision-agent-access.sh`) on vps1 and
|
|
vps2 — everything below on those two boxes is run through that access. The
|
|
desktop is wherever you're already working from; no separate access step
|
|
needed there.
|
|
2. **WireGuard mesh** (`setup-wireguard.sh <my_wg_ip> <peers_file>`) — run on
|
|
**all three** nodes. See the script's header for the peer-list format and
|
|
the two-pass key-exchange dance (pubkeys aren't known until every node has
|
|
run it once). Verify with `ping <peer_wg_ip>` to each of the other two
|
|
before continuing.
|
|
3. **Swarm init** (`init-swarm.sh <manager_wg_ip>`) on the manager (vps2) only.
|
|
4. **Swarm join** (`join-swarm.sh <manager_wg_ip> <token>`) on **each** of the
|
|
two workers (vps1, desktop) — same token for both.
|
|
5. **Firewall lockdown** (`firewall-swarm.sh`) on **all three** nodes — closes
|
|
2377/7946/4789 to everything except the WireGuard interface. Do this
|
|
*after* joining is confirmed working on all three, not before (locking the
|
|
ports first would make the join itself fail).
|
|
6. **Label vps1** (`label-forge-node.sh <vps1-node-name>`) on the manager —
|
|
`docker node ls` shows each node's name/ID. Only vps1 gets `role=forge`;
|
|
the desktop and vps2 don't need a Swarm label for anything in this setup.
|
|
7. Deploy Forgejo: see `deploy/forgejo/README.md`.
|
|
8. Register the Actions runner: `deploy/forgejo/register-lan-runner.sh`. See
|
|
that document (and `DEPLOY-DEV.md`) for where it actually runs today — its
|
|
own header comment predates the vps1/vps2 rename and still describes "the
|
|
desktop" as the canonical placement.
|
|
|
|
## Why WireGuard instead of relying on Swarm's built-in TLS alone
|
|
|
|
Swarm's control plane (port 2377) is TLS-encrypted and mutually authenticated
|
|
by default. Its overlay data plane (VXLAN, port 4789) is **not** encrypted by
|
|
default, and Docker's own guidance is that port must never face the public
|
|
internet — these nodes are on different networks (two separate providers'
|
|
public IPs, plus a home/LAN connection for the desktop), not one private LAN,
|
|
so the tunnel is the network boundary the Swarm ports advertise into, rather
|
|
than trusting the public internet (or the desktop's home network) directly.
|
|
|
|
## Verifying
|
|
|
|
```bash
|
|
# On the manager (vps2):
|
|
docker node ls # all three nodes Ready
|
|
docker node inspect <vps1-node> --format '{{.Spec.Labels}}' # role:forge
|
|
|
|
# From a FOURTH machine outside the mesh entirely, confirm the Swarm ports
|
|
# are NOT reachable on either VPS's public IP (the desktop has no public IP
|
|
# to check this way):
|
|
nc -zv -w2 <vps1_or_vps2_public_ip> 2377 # should fail/timeout
|
|
nc -zvu -w2 <vps1_or_vps2_public_ip> 4789 # should fail/timeout
|
|
```
|
|
|
|
## Adding a node label back out (undo)
|
|
|
|
```bash
|
|
docker node update --label-rm role <vps1-node>
|
|
```
|