162 lines
9 KiB
Markdown
162 lines
9 KiB
Markdown
# Agent access, Docker Swarm, and Forgejo CI/CD
|
|
|
|
This covers three additive infrastructure layers on top of the host
|
|
provisioning / Postgres / Terraform work described in `terraform/README.md`
|
|
and (for the historical Track A/Track B plan this whole effort grew from,
|
|
including the decision to split this repo out of the app monorepo)
|
|
`thermograph-docs/runbooks/implementation-handoff.md` and
|
|
`thermograph-docs/architecture/repo-topology-and-infrastructure.md`. Terraform
|
|
provisions **prod** (the new 48 GB / 12-core box, `thermograph.org`) and
|
|
**beta** (the old VPS, `75.119.132.91`) — see `terraform/README.md` /
|
|
`terraform.tfvars.example`. **None of this touches the app repo** (its
|
|
backend/frontend source, `Dockerfile`, or CI) — this repo owns only how and
|
|
where the already-built app image runs; the app repo owns building it.
|
|
|
|
```
|
|
Track 1: deploy/provision-agent-access.sh — a dedicated full-root login for me
|
|
Track 2: deploy/swarm/ — Swarm cluster spanning THREE nodes
|
|
Track 3: deploy/forgejo/ + .forgejo/ — Forgejo + Forgejo Actions, replacing GitHub
|
|
```
|
|
|
|
**Three nodes, not two**: prod, beta, and **the desktop** (the LAN dev
|
|
machine — same box that already runs the pre-Forgejo GitHub self-hosted
|
|
runner). This matches the canonical Track B design in the handoff doc; an
|
|
earlier revision of this whole effort covered just prod+beta and has been
|
|
realigned.
|
|
|
|
## Track 1 — Agent access
|
|
|
|
Run `sudo bash deploy/provision-agent-access.sh` on **prod and beta** (not
|
|
the desktop — that's wherever you're already working from, no separate
|
|
access-provisioning step needed there). Creates a dedicated `agent` user (not
|
|
raw root login — a distinct name gives a clean audit trail) with passwordless
|
|
sudo, installs the agent's public key, disables SSH password auth
|
|
fleet-wide, and turns on `auditd` logging of every root-effective command.
|
|
See the script's header comment for the full rationale and revocation steps
|
|
(one line to delete, or delete the whole account — your own access is never
|
|
affected).
|
|
|
|
### Current access state (live)
|
|
|
|
Both VPS boxes are provisioned and reachable as of this writing:
|
|
|
|
| Host | Role | Public IP | Login |
|
|
|------|------|-----------|-------|
|
|
| prod | Swarm manager; Thermograph's live prod home (`release`, thermograph.org) | `169.58.46.181` | `agent` |
|
|
| beta | Swarm worker; `main`/beta.thermograph.org + hosts Forgejo | `75.119.132.91` | `agent` |
|
|
| desktop | Swarm worker, LAN dev machine + Forgejo Actions runner | (local) | (already have access) |
|
|
|
|
```
|
|
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181 # prod
|
|
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91 # beta
|
|
```
|
|
|
|
The private half of the agent's dedicated keypair lives at
|
|
`~/.ssh/thermograph_agent_ed25519` on the operator's own machine (never in
|
|
this repo, never in a CI secret) — same convention `DEPLOY.md` already uses
|
|
for the separate CI deploy key (`~/.ssh/thermograph_ci`). If you're setting
|
|
this up fresh elsewhere, generate a new pair the same way
|
|
(`ssh-keygen -t ed25519 -a 100 -C "claude-agent@thermograph-infra" -f
|
|
~/.ssh/thermograph_agent_ed25519 -N ""`), embed the new public half in
|
|
`AGENT_PUBKEY` at the top of `provision-agent-access.sh`, and re-run it on
|
|
both boxes — that replaces the authorized key wholesale rather than
|
|
appending, so an old key stops working the moment the new one lands.
|
|
|
|
**Keep the private key somewhere that survives** — not a scratch/temp
|
|
directory that gets cleaned up, not anywhere-in-repo. Losing it just means
|
|
regenerating and re-running the provisioning script; it does not lock either
|
|
box, since your own login is untouched.
|
|
|
|
**Live as of 2026-07-21** (was: "no Docker / Terraform not applied / Swarm
|
|
inactive" — all now done): Docker is installed on all three nodes; the
|
|
WireGuard mesh and Docker Swarm are up with all three nodes `Ready` (prod
|
|
manager, beta + desktop workers); Terraform has been applied to both VPS
|
|
boxes (prod stood up fresh on `release`, beta rebuilt on `main`); Forgejo is
|
|
serving at `git.thermograph.org`; and the desktop runs the Forgejo Actions
|
|
runner. See the verification checklist at the end.
|
|
|
|
## Track 2 — Swarm
|
|
|
|
See `deploy/swarm/README.md` for the exact order of operations across all
|
|
**three** nodes (WireGuard mesh first, then swarm init/join, then lock the
|
|
Swarm ports down to the tunnel interface, then label beta for Forgejo
|
|
placement). This cluster's only job is hosting Forgejo — it does not
|
|
orchestrate the Terraform-managed app deploys.
|
|
|
|
## Track 3 — Forgejo, replacing GitHub
|
|
|
|
### 3a. Stand up Forgejo
|
|
See `deploy/forgejo/README.md` — deploys the stack (pinned to beta), then
|
|
walks through registering the Actions runner **on the desktop** as a plain
|
|
systemd service (`register-lan-runner.sh`), not as a Swarm-scheduled
|
|
container. Only one Swarm secret to mint now (`forgejo_db_password`) — the
|
|
runner token is no longer a Swarm secret, since the runner isn't a Swarm
|
|
service anymore.
|
|
|
|
### 3b. Migrate the repo (mirror first, verify, then cut over)
|
|
1. In Forgejo: **+ New Migration → GitHub**. Point it at
|
|
`https://github.com/griffemi/thermograph`, pull code + issues + PRs +
|
|
releases + labels. This is non-destructive to GitHub — do this as a mirror
|
|
and leave GitHub live and untouched until you're confident.
|
|
2. Spot-check: branch list matches, a few PRs/issues render correctly, LFS (if
|
|
any) came across.
|
|
3. **Don't retarget any secret or disable a GitHub workflow yet** — see 3d.
|
|
|
|
### 3c. Workflows
|
|
`.forgejo/workflows/{build,pr-build,deploy-dev,deploy}.yml` mirror
|
|
`.github/workflows/*.yml`, copied with the mechanical changes Forgejo needs:
|
|
- `runs-on: ubuntu-latest` → `runs-on: docker` (one of the two labels the
|
|
desktop's runner registers under — see 3a; general CI/deploy jobs get a
|
|
fresh container, the LAN-deploy job in `deploy-dev.yml` runs bare/host-native
|
|
under the `thermograph-lan` label instead, since it needs real filesystem
|
|
access).
|
|
- `appleboy/ssh-action` referenced by full GitHub URL (not mirrored in
|
|
Forgejo's default action registry, unlike `actions/checkout` /
|
|
`actions/setup-python`, which resolve unchanged).
|
|
- **The custom auto-merge workflow step is gone.** It existed only because
|
|
GitHub's native branch protection/auto-merge is paywalled on private
|
|
free-tier repos. Forgejo has no such tier: `dev` is protected with the
|
|
`build` status check required (Settings → Branches). Forgejo does **not**
|
|
auto-merge on green by itself — a ready, green, mergeable PR is merged
|
|
**explicitly** (`POST .../pulls/{n}/merge`, `{"Do": "squash"}`), per
|
|
`CLAUDE.md`. That merge is an ordinary push, so `deploy-dev.yml`'s push
|
|
trigger fires naturally afterward.
|
|
- **Branch/deploy mapping (settled):** `.forgejo/workflows/deploy.yml`
|
|
("Deploy to beta VPS") triggers on `main` and SSHes to **beta**
|
|
(`75.119.132.91`) running `deploy/deploy.sh`. **Prod is not deployed by a
|
|
push-triggered workflow** — it's deployed with `terraform apply`
|
|
(`terraform/README.md`) on the `release` branch. So the promotion chain is
|
|
`dev` → `main` (this workflow deploys beta) → `release` (Terraform deploys
|
|
prod). There is deliberately no `release`-triggered workflow.
|
|
|
|
A second, independent set of Forgejo workflows
|
|
(`.forgejo/workflows/{ci,build-push,ops-cron}.yml`) was added by a parallel
|
|
effort implementing `thermograph-docs/runbooks/implementation-handoff.md` Track A chunk
|
|
7 — see that doc and its own reconciliation PR (which already fixed its
|
|
runner-label and ssh-action guesses to match the real infrastructure here).
|
|
`ci.yml` duplicated this PR's `build.yml` exactly and was dropped in that
|
|
reconciliation; `build-push.yml` (image → Forgejo's built-in registry) and
|
|
`ops-cron.yml` (backup + IndexNow) are novel and still present.
|
|
|
|
### 3d. Cut over — DONE (2026-07-21)
|
|
The GitHub → Forgejo cutover is complete; this records what was done:
|
|
1. Deploy secrets (`SSH_HOST`, `SSH_USER`, `SSH_KEY`, `SSH_PORT`) live in
|
|
Forgejo's repo Secrets, pointing at beta with a dedicated CI deploy key.
|
|
2. The LAN dev runner was re-pointed to Forgejo via
|
|
`deploy/forgejo/register-lan-runner.sh` (systemd --user, on the desktop).
|
|
3. The repo was migrated into Forgejo; `dev`/`main`/`release` all promoted
|
|
through Forgejo and deploys verified live.
|
|
4. GitHub is retired as git host and CI (PR "Retire GitHub as the git host and
|
|
CI platform") — `origin` remains only as a read-only mirror of history.
|
|
|
|
## Verification checklist — all met (2026-07-21)
|
|
- [x] `ssh agent@<prod>` and `ssh agent@<beta>` both work; `sudo whoami` → `root`
|
|
- [x] Password SSH auth confirmed dead on both boxes (tested with an actual
|
|
failed login attempt, not just config inspection)
|
|
- [x] `docker node ls` (from prod) shows all three nodes `Ready`
|
|
- [x] Swarm ports (2377/7946/4789) unreachable from outside the WireGuard tunnel
|
|
- [x] `https://git.thermograph.org` serves Forgejo over TLS; `/v2/` registry
|
|
API is 403 from off-mesh (firewalled to the WireGuard CIDR)
|
|
- [x] A test PR flow verified: required `build` check runs → explicit squash
|
|
merge → LAN dev deploy lands
|
|
- [x] GitHub retired as git host + CI; kept only as a read-only history mirror
|