Compare commits

...

6 commits

Author SHA1 Message Date
Emi Griffith
6745f8db27 Merge remote-tracking branch 'origin/dev' into fix/registry-namespace-admin-emi
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 4s
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / build-frontend (pull_request) Successful in 21s
PR build (required check) / validate-observability (pull_request) Successful in 22s
PR build (required check) / build-backend (pull_request) Successful in 44s
PR build (required check) / gate (pull_request) Successful in 1s
2026-08-01 09:31:18 -07:00
admin_emi
7fca3556e8 Merge pull request 'forgejo: serve dev.jinemi.com alongside git.thermograph.org' (#148) from feat/forgejo-jinemi-alias into dev
All checks were successful
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 5s
Sync infra to hosts / sync-centralis (push) Has been skipped
secrets-guard / encrypted (push) Successful in 4s
shell-lint / shellcheck (push) Successful in 7s
2026-08-01 16:28:49 +00:00
Emi Griffith
000a8a1048 Merge dev into feat/forgejo-jinemi-alias (dev moved while #148 was open)
All checks were successful
PR build (required check) / changes (pull_request) Successful in 7s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
2026-08-01 09:27:04 -07:00
Emi Griffith
3993b7552f lb: trust the host Caddy's X-Forwarded-Proto
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 4s
PR build (required check) / changes (pull_request) Successful in 6s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 6s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
Sync infra to hosts / sync-beta (push) Has been skipped
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-centralis (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 6s
secrets-guard / encrypted (push) Successful in 4s
shell-lint / shellcheck (push) Successful in 6s
Every OAuth redirect URI the backend builds is currently http://, whatever
scheme the user actually arrived with.

There are two Caddy hops in front of the app: the host Caddy terminates TLS
and sets X-Forwarded-Proto: https, then proxies to 127.0.0.1:8137 over plain
HTTP; the stack LB is the second hop. Caddy preserves an incoming
X-Forwarded-* header only when the immediate peer is a trusted proxy, and
trusted_proxies was never set here -- so the LB overwrote the header with the
scheme of the connection it had just received, which is http.

accounts/oauth.py:_redirect_uri builds the callback from x-forwarded-proto,
and that URI must match what is registered in the provider's console.

Measured on vps2, same request to each hop:

  direct to web, XFP=https      -> https://thermograph.org/api/v2/...
  through the LB, XFP=https     -> http://thermograph.org/api/v2/...
  through the PATCHED LB        -> https://thermograph.org/api/v2/...
  through the PATCHED LB, no XFP-> http://thermograph.org/api/v2/...

The last line is the point of using trusted_proxies rather than hardcoding
`header_up X-Forwarded-Proto https`: the LB still reports the truth when
nothing in front of it claims otherwise.

private_ranges covers 127.0.0.1/8 and the Docker bridge ranges, which is where
the host Caddy reaches this container from. The LB binds loopback only,
precisely so it cannot be reached un-fronted, so the only party that can set
these headers is the host Caddy -- trusting the local peer widens nothing.

Applied to beta's LB too. Beta is where a provider change gets tested before
it reaches thermograph.org, so it needs the same behaviour or the test is not
a test.
2026-08-01 16:24:34 +00:00
Emi Griffith
5b5f28560d gitignore: ignore .claude/settings.local.json
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 7s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 2s
It was untracked but not ignored, so a single `git add -A` would have
committed one operator's local permission allowlist — including which vault
secrets they may decrypt — as shared repo config. `.claude/settings.json`
stays tracked as the reviewed team config.
2026-08-01 09:16:06 -07:00
Emi Griffith
396b80cf91 forgejo: serve dev.jinemi.com alongside git.thermograph.org
Both hostnames go on a single Caddy site block. That is load-bearing rather
than cosmetic: the /v2/* mesh-only matcher that keeps the OCI registry API
off the public internet (hazard #15) is scoped per block, so a separate
block for the new name would serve the same Forgejo with the registry open.

git.thermograph.org stays canonical — Forgejo has one ROOT_URL and builds
every absolute URL from it, so clone URLs, the OAuth callback and post-login
redirects keep naming the .org host. Flipping FORGEJO_DOMAIN additionally
requires the Google OAuth redirect URI, the registry host baked into image
names and runner labels, and the mesh /etc/hosts pins; README documents
those prerequisites.

Config verified with `caddy validate` and by asserting against the adapted
JSON that the /v2 403 guard matches both hostnames.
2026-08-01 09:01:59 -07:00
8 changed files with 122 additions and 14 deletions

6
.gitignore vendored
View file

@ -2,3 +2,9 @@
# Claude Code worktrees. These are other sessions' checkouts living inside this
# repo; `git add -A` will otherwise commit them as embedded git repositories.
.claude/worktrees/
# Per-operator Claude Code settings — local permission grants, not team config.
# Untracked but not ignored meant one `git add -A` would publish one operator's
# allowlist (including which secrets they may decrypt) to everyone.
# `.claude/settings.json` IS tracked and stays the shared, reviewed config.
.claude/settings.local.json

View file

@ -26,7 +26,7 @@ rather than which environment happens to live there:
| Host | Public IP | Mesh IP | Role |
|------|-----------|---------|------|
| **vps1** | `75.119.132.91` | `10.10.0.2` | "Operational programs": Forgejo (git + CI + registry, `git.thermograph.org`), Grafana/Loki/Alloy (`dashboard.thermograph.org`), the `emigriffith.dev` portfolio, and the **dev** environment (mesh-only, `10.10.0.2:8137`, no public DNS/Caddy/TLS) |
| **vps1** | `75.119.132.91` | `10.10.0.2` | "Operational programs": Forgejo (git + CI + registry, `git.thermograph.org`, also served at `dev.jinemi.com` — migration in progress, `.org` still canonical), Grafana/Loki/Alloy (`dashboard.thermograph.org`), the `emigriffith.dev` portfolio, and the **dev** environment (mesh-only, `10.10.0.2:8137`, no public DNS/Caddy/TLS) |
| **vps2** | `169.58.46.181` | `10.10.0.1` | "The deployed environment": **prod** and **beta**, as two separate Docker Swarm stacks, plus Centralis, Postfix, the backups, and the one shared TimescaleDB instance both stacks use |
| **desktop** | — | `10.10.0.3` | AI-model hosting (voice-to-text, an upcoming-feature LLM) plus flex Swarm-worker capacity. Hosts **no** Thermograph environment — `make dev-up` still works there, but only as a laptop-local rehearsal |

View file

@ -12,7 +12,9 @@ Two VPS boxes plus the operator's desktop, on one WireGuard mesh, named by
ROLE rather than by environment:
- **vps1**`75.119.132.91`, mesh `10.10.0.2`. "Operational programs": Forgejo
(git + CI + registry, `git.thermograph.org`), Grafana/Loki/Alloy
(git + CI + registry, `git.thermograph.org`, also served at `dev.jinemi.com`
— one Caddy site block, the `.org` name still canonical; migration in
progress, see `deploy/forgejo/README.md`), Grafana/Loki/Alloy
(`dashboard.thermograph.org`), the `emigriffith.dev` portfolio site, and the
**dev** environment (`/opt/thermograph-dev`, tracks `dev`), with its own
Postgres container. Dev is mesh-only here — published on `10.10.0.2:8137`,

View file

@ -3,6 +3,9 @@
# vps1 serves:
# emigriffith.dev -> static portfolio site (from disk)
# git.thermograph.org -> Forgejo (see deploy/forgejo/caddy-git.conf)
# dev.jinemi.com -> Forgejo, same site block — migration target for
# git.thermograph.org, both served, the .org one
# still canonical (that file's header explains)
# dashboard.thermograph.org -> Grafana (see observability/caddy-grafana.conf)
#
# and hosts the **dev** environment at dev.thermograph.org (below).

View file

@ -83,6 +83,53 @@ pattern as its other site blocks, same automatic-HTTPS.
reach the registry over the mesh, not the public internet — see "Registry
access from mesh clients" below.
## Migrating the domain (git.thermograph.org -> dev.jinemi.com)
In progress. **Both hostnames are served**; `git.thermograph.org` is still the
canonical one.
`caddy-git.conf` lists both names on a *single* site block. That is deliberate,
not cosmetic: the `/v2/*` mesh-only matcher is per-block, so a second block for
`dev.jinemi.com` would re-expose the registry API publicly and undo hazard #15.
Whatever else changes, keep the two names in one block.
Serving a second hostname is the cheap half. Forgejo has exactly one
`ROOT_URL`, and it builds every absolute URL from it, so on the non-canonical
host today:
- clone URLs on repo pages read `git.thermograph.org`,
- the OAuth callback and post-login redirect land on `git.thermograph.org`
(login is Google-SSO-only — see `docker-stack.yml`),
- webhook payload URLs and notification mail links use `git.thermograph.org`.
Browsing, git-over-HTTP and the API all work on `dev.jinemi.com` regardless.
Flipping canonical (`FORGEJO_DOMAIN=dev.jinemi.com`, then `docker stack deploy`)
needs these **first**, or it breaks login and CI rather than just renaming
things:
1. **Google OAuth** — add `https://dev.jinemi.com/user/oauth2/<name>/callback`
as an authorized redirect URI in the Google Cloud console *before* the flip.
Not in this repo; nothing here can verify it. Miss it and no one can log in,
including to undo the flip.
2. **The registry host in image names** — images are named by registry host, and
the `git.thermograph.org/` prefix is baked into the runner labels
(`register-lan-runner.sh`, `runner-vps2/README.md`), the CI-runner image
(`ci-runner/Dockerfile`), `REGISTRY_HOST` in `infra/.env.example`, and every
already-pushed tag. Renaming the registry host is a separate migration from
renaming the web UI; the two need not happen together, and doing the web UI
alone is much the smaller change.
3. **Mesh `/etc/hosts` pins** — every node in "Registry access from mesh
clients" below pins `git.thermograph.org` to `10.10.0.2`. A node pulling
under a new hostname needs its own pin, or Caddy's `/v2/*` matcher sees a
public source IP and returns 403.
4. **Runner instance URL** — registered runners hold the instance URL they were
registered with. Existing runners keep working against the old name as long
as it resolves; a *re*-registration must use whichever name is canonical then.
Until all four are handled, leaving `ROOT_URL` on `git.thermograph.org` is the
correct state, not an unfinished one.
## Registry access from mesh clients
Any node that needs `docker login`/push/pull against the registry (the CI

View file

@ -1,6 +1,6 @@
# Only the block below (from "git.thermograph.org {") goes into the live
# Caddyfile — append just that part, not this instructional header, or it
# ends up as a confusing orphaned comment in production config with no
# Only the block below (from "git.thermograph.org, dev.jinemi.com {") goes into
# the live Caddyfile — append just that part, not this instructional header, or
# it ends up as a confusing orphaned comment in production config with no
# surrounding context (docker-stack.yml isn't visible from there).
#
# Target: /etc/caddy/Caddyfile on vps1 (the box `forgejo` is pinned to — see
@ -9,19 +9,30 @@
# Forgejo publishes its web UI to 127.0.0.1:3080 only (host-local), and this
# block is the only thing that ever talks to it.
#
# Registry exposure (hazard #15, see docker-stack.yml's header comment):
# git.thermograph.org/v2/* is the built-in OCI registry API. It's blocked
# from anywhere except the WireGuard mesh (10.10.0.0/24) — reachable to CI
# runners and Swarm nodes over wg0, not from the public internet. Everything
# else (web UI, git-over-HTTP, PR pages) stays public like the rest of the
# site.
# TWO hostnames, ONE block, and that is load-bearing — see "Registry exposure"
# below. Both names must share a site block so the /v2/* restriction covers
# both. Splitting dev.jinemi.com into its own block serves the same Forgejo
# with the registry API open to the public internet.
#
# Update the domain below to match whatever you actually pointed at vps1's
# IP, if not git.thermograph.org.
# dev.jinemi.com is the migration target for git.thermograph.org; both are
# served during the transition. Each gets its own Let's Encrypt cert
# automatically (HTTP-01) — both A records already point at vps1.
#
# git.thermograph.org stays CANONICAL until Forgejo's ROOT_URL is flipped
# (docker-stack.yml, FORGEJO_DOMAIN). Until then dev.jinemi.com serves every
# page, but Forgejo builds absolute URLs from ROOT_URL, so clone URLs, the
# OAuth callback and post-login redirects still name git.thermograph.org.
# Flipping it is NOT just this file — see README.md, "Migrating the domain".
#
# Registry exposure (hazard #15, see docker-stack.yml's header comment):
# /v2/* is the built-in OCI registry API. It's blocked from anywhere except
# the WireGuard mesh (10.10.0.0/24) — reachable to CI runners and Swarm nodes
# over wg0, not from the public internet. Everything else (web UI,
# git-over-HTTP, PR pages) stays public like the rest of the site.
# --- copy from here down into /etc/caddy/Caddyfile ---
git.thermograph.org {
git.thermograph.org, dev.jinemi.com {
encode zstd gzip
@registry_external {

View file

@ -13,6 +13,34 @@
{
auto_https off
admin off
# WITHOUT THIS, EVERY OAUTH REDIRECT URI THE BACKEND BUILDS IS http://.
#
# There are two Caddy hops in front of the app: the HOST Caddy terminates TLS
# and sets X-Forwarded-Proto: https, then proxies to 127.0.0.1:8137 over plain
# HTTP; this LB is the second hop. Caddy only preserves an incoming
# X-Forwarded-* header when the immediate peer is a TRUSTED proxy — otherwise
# it overwrites the header with the scheme of the connection it just received,
# which here is http. So the app saw http:// no matter how the user arrived.
#
# accounts/oauth.py:_redirect_uri builds the callback from x-forwarded-proto,
# and that URI must match what is registered in the provider's console.
# Discord tolerates the http:// form; GOOGLE REJECTS a non-HTTPS redirect URI
# for a web client outright, so Google sign-in could not work at all until
# this was fixed — independently of whether the credentials were configured.
#
# Measured on vps2 before the fix, same request to each hop:
# direct to web, XFP=https -> https://thermograph.org/api/v2/...
# through this LB, XFP=https -> http://thermograph.org/api/v2/...
#
# private_ranges covers 127.0.0.1/8 and the Docker bridge ranges, which is
# where the host Caddy reaches this container from. Nothing outside the box
# can connect here — the LB is bound to loopback precisely so it cannot be
# reached un-fronted (hazard #6, above) — so trusting the local peer does not
# widen anything: the only party that can set these headers is the host Caddy.
servers {
trusted_proxies static private_ranges
}
}
:8137 {

View file

@ -17,6 +17,17 @@
{
auto_https off
admin off
# Same reason as ./Caddyfile — see the long note there. Without it this second
# Caddy hop overwrites the host Caddy's X-Forwarded-Proto: https with http,
# and every OAuth redirect URI the backend builds comes out non-HTTPS, which
# Google rejects outright for a web client.
#
# Beta needs it as much as prod does, and arguably first: beta is where an
# OAuth provider change gets tested before it reaches thermograph.org.
servers {
trusted_proxies static private_ranges
}
}
:8137 {