lb: trust the host Caddy's X-Forwarded-Proto #145

Merged

admin_emi merged 1 commit from fix/lb-forwarded-proto into dev

2026-08-01 16:25:16 +00:00

admin_emi commented

2026-08-01 16:06:38 +00:00

Owner

⚠️ Do not merge-and-deploy before adding the https:// redirect URI to the Discord app

This change alters what the backend sends providers. Today it sends http://thermograph.org/api/v2/discord/link/callback; after this it sends the https:// form. If the Discord developer console only has the http:// URI registered, Discord sign-in breaks on the next deploy. Both can coexist — add the https:// one first, deploy, then remove the http:// one later. Same for beta.

The bug

Every OAuth redirect URI the backend builds is http://, whatever scheme the user actually arrived with.

Prod is two Caddy hops: the host Caddy terminates TLS and sets X-Forwarded-Proto: https, then proxies to 127.0.0.1:8137 over plain HTTP; the stack LB is the second hop. Caddy preserves an incoming X-Forwarded-* header only when the immediate peer is a trusted proxy, and trusted_proxies was never set here — so the LB overwrote the header with the scheme of the connection it had just received.

accounts/oauth.py:_redirect_uri builds the callback from x-forwarded-proto, and that URI must match what is registered in the provider's console.

Google rejects a non-HTTPS redirect URI for a web client outright. So THERMOGRAPH_GOOGLE_CLIENT_ID/_SECRET could not have worked even once configured — Google sign-in shipped to prod in #122 and /api/v2/oauth/config reports google: enabled=false today, but this would have been the next wall. Discord tolerates the http:// form, which is why nobody noticed.

Measured on vps2

Same request to each hop:

path	redirect_uri produced
direct to `web`, `XFP: https`	`https://thermograph.org/api/v2/...`
through the LB, `XFP: https`	`http://thermograph.org/api/v2/...`
through the patched LB, `XFP: https`	`https://thermograph.org/api/v2/...`
through the patched LB, no XFP	`http://thermograph.org/api/v2/...`

Verified by running a throwaway caddy:2-alpine on thermograph_internal with this config, on a spare loopback port, against the live web service; caddy validate passes; container removed afterwards.

That last row is why this uses trusted_proxies rather than hardcoding header_up X-Forwarded-Proto https — the LB still reports the truth when nothing in front of it claims otherwise, which keeps STACK_TEST=1 rehearsals honest.

private_ranges covers 127.0.0.1/8 and the Docker bridge ranges, which is where the host Caddy reaches this container from. The LB binds loopback only, precisely so it cannot be reached un-fronted (hazard #6), so the only party that can set these headers is the host Caddy — trusting the local peer widens nothing.

Applied to beta's LB too: beta is where a provider change gets tested before it reaches thermograph.org, so it needs the same behaviour or the test is not a test.

Takes effect when deploy-stack.sh recreates the LB container, not on infra-sync.

## ⚠️ Do not merge-and-deploy before adding the https:// redirect URI to the Discord app This change alters what the backend sends providers. Today it sends `http://thermograph.org/api/v2/discord/link/callback`; after this it sends the `https://` form. **If the Discord developer console only has the `http://` URI registered, Discord sign-in breaks on the next deploy.** Both can coexist — add the `https://` one first, deploy, then remove the `http://` one later. Same for beta. ## The bug Every OAuth redirect URI the backend builds is `http://`, whatever scheme the user actually arrived with. Prod is two Caddy hops: the host Caddy terminates TLS and sets `X-Forwarded-Proto: https`, then proxies to `127.0.0.1:8137` over plain HTTP; the stack LB is the second hop. Caddy preserves an incoming `X-Forwarded-*` header **only when the immediate peer is a trusted proxy**, and `trusted_proxies` was never set here — so the LB overwrote the header with the scheme of the connection it had just received. `accounts/oauth.py:_redirect_uri` builds the callback from `x-forwarded-proto`, and that URI must match what is registered in the provider's console. **Google rejects a non-HTTPS redirect URI for a web client outright.** So `THERMOGRAPH_GOOGLE_CLIENT_ID`/`_SECRET` could not have worked even once configured — Google sign-in shipped to prod in #122 and `/api/v2/oauth/config` reports `google: enabled=false` today, but this would have been the *next* wall. Discord tolerates the `http://` form, which is why nobody noticed. ## Measured on vps2 Same request to each hop: | path | redirect_uri produced | |---|---| | direct to `web`, `XFP: https` | `https://thermograph.org/api/v2/...` | | through the LB, `XFP: https` | `http://thermograph.org/api/v2/...` | | through the **patched** LB, `XFP: https` | `https://thermograph.org/api/v2/...` | | through the **patched** LB, no XFP | `http://thermograph.org/api/v2/...` | Verified by running a throwaway `caddy:2-alpine` on `thermograph_internal` with this config, on a spare loopback port, against the live `web` service; `caddy validate` passes; container removed afterwards. That last row is why this uses `trusted_proxies` rather than hardcoding `header_up X-Forwarded-Proto https` — the LB still reports the truth when nothing in front of it claims otherwise, which keeps `STACK_TEST=1` rehearsals honest. `private_ranges` covers `127.0.0.1/8` and the Docker bridge ranges, which is where the host Caddy reaches this container from. The LB binds loopback only, precisely so it cannot be reached un-fronted (hazard #6), so the only party that can set these headers is the host Caddy — trusting the local peer widens nothing. Applied to beta's LB too: beta is where a provider change gets tested before it reaches thermograph.org, so it needs the same behaviour or the test is not a test. Takes effect when `deploy-stack.sh` recreates the LB container, not on `infra-sync`.

admin_emi added 1 commit 2026-08-01 16:06:39 +00:00

lb: trust the host Caddy's X-Forwarded-Proto

PR build (required check) / changes (pull_request) Successful in 7s

Details

secrets-guard / encrypted (pull_request) Successful in 5s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

shell-lint / shellcheck (pull_request) Successful in 7s

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

a4bcae5bf9

Every OAuth redirect URI the backend builds is currently http://, whatever
scheme the user actually arrived with.

There are two Caddy hops in front of the app: the host Caddy terminates TLS
and sets X-Forwarded-Proto: https, then proxies to 127.0.0.1:8137 over plain
HTTP; the stack LB is the second hop. Caddy preserves an incoming
X-Forwarded-* header only when the immediate peer is a trusted proxy, and
trusted_proxies was never set here -- so the LB overwrote the header with the
scheme of the connection it had just received, which is http.

accounts/oauth.py:_redirect_uri builds the callback from x-forwarded-proto,
and that URI must match what is registered in the provider's console.

Measured on vps2, same request to each hop:

  direct to web, XFP=https      -> https://thermograph.org/api/v2/...
  through the LB, XFP=https     -> http://thermograph.org/api/v2/...
  through the PATCHED LB        -> https://thermograph.org/api/v2/...
  through the PATCHED LB, no XFP-> http://thermograph.org/api/v2/...

The last line is the point of using trusted_proxies rather than hardcoding
`header_up X-Forwarded-Proto https`: the LB still reports the truth when
nothing in front of it claims otherwise.

private_ranges covers 127.0.0.1/8 and the Docker bridge ranges, which is where
the host Caddy reaches this container from. The LB binds loopback only,
precisely so it cannot be reached un-fronted, so the only party that can set
these headers is the host Caddy -- trusting the local peer widens nothing.

Applied to beta's LB too. Beta is where a provider change gets tested before
it reaches thermograph.org, so it needs the same behaviour or the test is not
a test.

admin_emi force-pushed fix/lb-forwarded-proto from a4bcae5bf9 to 3993b7552f

2026-08-01 16:24:36 +00:00

Compare