Completes the Forgejo domain migration. ROOT_URL moved to dev.jinemi.com
earlier; the registry half was deliberately deferred. Every image name,
REGISTRY_HOST default, runner label and --add-host pin now names
dev.jinemi.com, so the registry host and the bearer-token realm agree again.
Both names address the same Forgejo, so no image needs re-pushing and a
rollback to a tag pushed under the old prefix still resolves.
git.thermograph.org therefore stays served off the same Caddy site block --
one block, so the /v2/* mesh-only matcher keeps covering both names -- for
pre-migration tags and for runners holding it as their registered instance
URL.
Mesh clients now pin both names in /etc/hosts: the new one as registry host
and token realm, the old one for pre-migration tags. runner-vps2/config.yaml
carries both --add-host entries for the same reason.
Also renames Centralis' endpoint to mcp.jinemi.com in the two places this
repo names it; Centralis itself is provisioned outside this repo.
Host-side steps this cannot do (documented in deploy/forgejo/README.md,
"Host-side steps"): the Forgejo Actions variable REGISTRY_HOST, docker login
against the new host, and the /etc/hosts pins.
Repos moved to the Jinemi org; the container packages did not follow, since
Forgejo does not transfer packages with a repo. The deploy path still resolved
`emi/thermograph/*` — a user_redirect to admin_emi — while build-push.yml
derives its push path from ${github.repository}, now jinemi/thermograph. The
next backend or frontend build would have published somewhere no deploy looks.
Point the image paths at jinemi/thermograph/* (lowercase: OCI references admit
no uppercase, which is why build-push.yml already pipes through tr), the clone
URLs at Jinemi/thermograph, and the registry logins at admin_emi — the account
that actually owns the tokens, rather than the redirect.
The live tags and both ci-runner tags were copied into the Jinemi namespace
first, so the switch has something to pull. thermograph-infra,
thermograph-observability and the retired */app packages stay under admin_emi;
they did not move.
Every OAuth redirect URI the backend builds is currently http://, whatever
scheme the user actually arrived with.
There are two Caddy hops in front of the app: the host Caddy terminates TLS
and sets X-Forwarded-Proto: https, then proxies to 127.0.0.1:8137 over plain
HTTP; the stack LB is the second hop. Caddy preserves an incoming
X-Forwarded-* header only when the immediate peer is a trusted proxy, and
trusted_proxies was never set here -- so the LB overwrote the header with the
scheme of the connection it had just received, which is http.
accounts/oauth.py:_redirect_uri builds the callback from x-forwarded-proto,
and that URI must match what is registered in the provider's console.
Measured on vps2, same request to each hop:
direct to web, XFP=https -> https://thermograph.org/api/v2/...
through the LB, XFP=https -> http://thermograph.org/api/v2/...
through the PATCHED LB -> https://thermograph.org/api/v2/...
through the PATCHED LB, no XFP-> http://thermograph.org/api/v2/...
The last line is the point of using trusted_proxies rather than hardcoding
`header_up X-Forwarded-Proto https`: the LB still reports the truth when
nothing in front of it claims otherwise.
private_ranges covers 127.0.0.1/8 and the Docker bridge ranges, which is where
the host Caddy reaches this container from. The LB binds loopback only,
precisely so it cannot be reached un-fronted, so the only party that can set
these headers is the host Caddy -- trusting the local peer widens nothing.
Applied to beta's LB too. Beta is where a provider change gets tested before
it reaches thermograph.org, so it needs the same behaviour or the test is not
a test.
main took two direct PRs (#19 shell-lint, #21 the thermograph-daemon Go
service) while dev accumulated the ERA5 lake stack; both sides added compose/
stack services and touched the same seams. Resolutions keep both worlds:
- compose + Swarm stack: lake AND daemon are sibling services; backend env
carries THERMOGRAPH_LAKE_URL and THERMOGRAPH_INTERNAL_TOKEN.
- deploy.sh: backend deploys roll (backend lake daemon); all adds frontend.
- requirements.txt: websockets/apscheduler stay removed (moved to the Go
daemon), duckdb stays (the lake role's engine).
Merged tree: backend suite green, both compose configs validate.
The gateway bot and APScheduler were long-lived stateful I/O loops running
inside the async web app under a leader election. They move into a single Go
binary that owns ONLY that I/O -- websocket, RESUME, heartbeat, backoff, timers.
It owns no grading logic. Anything needing data calls back over a new
internal-only surface (/internal/discord/grade, /internal/jobs/*). Grading
depends on polars and the parquet cache; reimplementing it in Go would let the
bot's grades drift from the API's. The grade route returns gateway-ready JSON
and Go relays the bytes verbatim.
The binary ships in the backend image and runs as a second compose service off
the same tag, so the two ends of the /internal/* contract can never skew.
deploy.sh rolls daemon alongside backend -- without that the service would never
be created, since a single-service deploy uses --no-deps. It also probes the
image first and skips the daemon when rolling a tag that predates the binary:
infra tracks main while image tags are env-staged, so a host can legitimately be
asked to roll an older backend image, and creating the service anyway would
leave a container crash-looping on a missing binary.
replicas: 1 with order: stop-first replaces the leader election -- Discord
permits one gateway connection per bot token.
THERMOGRAPH_INTERNAL_TOKEN is optional: both ends derive it from
THERMOGRAPH_AUTH_SECRET via HMAC under a domain-separation label, so this needs
no new vault entry. The derivation is pinned to a shared cross-language test
vector asserted on both sides, so drift fails CI instead of 401ing every call.
Fail closed when neither secret is set.
Improvements over the Python: a close intended for RESUME uses 4000 rather than
1000 (Discord invalidates a session closed 1000, so the old default defeated its
own resume); MESSAGE_CREATE runs on a bounded worker pool; and a malformed HELLO
returns an error rather than a clean reconnect, which would otherwise reset
backoff and hot-loop against the gateway.
365 Python tests pass; Go build/vet/test -race clean; shellcheck 0 findings.
Adds .forgejo/workflows/shell-lint.yml (pinned shellcheck v0.11.0 + sha256, -x,
default severity, fail on any finding, not path-filtered) and drives all 26
scripts to zero findings.
Two defects shellcheck cannot see:
render-secrets.sh left DECRYPTED vault contents in /tmp whenever a sops decrypt
failed -- the caller's set -e aborted the function before either cleanup ran.
Now removed on every exit path, with `|| return 1` on both sops calls so a
decrypt failure can never write a partial /etc/thermograph.env regardless of the
caller's shell options. Explicitly not a `trap ... RETURN`: such a trap set in a
sourced function persists into the caller's shell and re-fires when the caller's
next `.`/source completes, where the function-local tmp is unset -- fatal and
silent under deploy.sh's set -u. The file now records that reasoning.
autoscale.sh ran `set -eu` without pipefail while piping docker stats into awk,
so a failed left side was swallowed and the loop autoscaled on empty input.
Promoted to pipefail with a missed sample treated as a skip; verified busybox ash
in docker:27-cli supports it.
Also: capture-fixtures.sh's `jq . || cat` ran cat after jq had consumed stdin,
silently writing truncated fixtures; deploy.sh/deploy-stack.sh `# shellcheck
source=` paths corrected for the monorepo layout.
The cutover moved the host checkout's deploy tree to /opt/thermograph/infra/deploy/,
but the stack file's absolute bind sources (db init, env-entrypoint.sh, autoscale.sh)
still pointed at the old /opt/thermograph/deploy/ -- a service update kept the stale
mounts and new tasks failed with 'bind source path does not exist'.
/opt/thermograph becomes a monorepo checkout: deploy.sh gains INFRA_DIR (git
ops at the root, compose work cd'd into infra/), lock/tags/render paths move
under infra/deploy/, image-path defaults become emi/thermograph/backend|
frontend across compose, stack, and the tag-prune. docker-compose.yml pins
name: thermograph -- without it compose run from .../infra derives project
"infra" and recreates the whole stack beside the running one;
deploy-dev.sh pins COMPOSE_PROJECT_NAME=thermograph-dev (env wins over the
file key) to keep LAN dev's separate project.
