daemon: move the Discord gateway and scheduler out of the web process into Go #21

Merged

admin_emi merged 1 commit from go-daemon into main

2026-07-23 22:49:56 +00:00

admin_emi commented

2026-07-23 22:43:28 +00:00

Owner

web/app.py started two long-lived background jobs under a leader election: the
Discord gateway bot and an APScheduler. Both are stateful I/O loops — reconnect,
RESUME, heartbeat, backoff, interval timers — living inside an async web app that
also has to serve requests. This moves them into a single Go binary.

The split

Go owns only the stateful I/O. It owns no climate or grading logic. Anything
needing data calls back into Python over a new internal-only surface
(/internal/discord/grade, /internal/jobs/warm-cities,
/internal/jobs/indexnow).

That boundary is the whole design. Grading depends on polars and the parquet
cache; reimplementing it in Go would let the bot's grades drift from the API's,
and the slash-command path deliberately shares one grade builder so the two can
never disagree. The grade route returns gateway-ready JSON — including the
ephemeral-flag drop discord_bot.py used to do — and Go relays those bytes
verbatim without ever parsing the embed.

Packaging

The binary is built by a golang:1.26 stage in the backend Dockerfile and
shipped in the same image, run as a second compose service off the same
tag. The daemon and backend share the /internal/* contract, so they must
never skew versions; one image makes that structural rather than a convention.
Its entrypoint bypasses entrypoint.sh — the backend owns alembic, and two
racing migrators is a real hazard.

deploy.sh now rolls daemon alongside backend. Without that the service
would never be created at all: a single-service deploy runs
up -d --no-deps <targets>, so a daemon present only in compose would have
been silently skipped and the bot would never have started.

replicas: 1 in the Swarm stack is load-bearing — Discord permits exactly one
gateway connection per bot token, so the pin replaces
core/singleton.claim_leader for this workload. order: stop-first, since
start-first would briefly run two gateways. autoscale.sh targets
${STACK_NAME}_web only, so it cannot scale this.

No new secret to provision

THERMOGRAPH_INTERNAL_TOKEN is optional. Unset, both ends derive the same
token from THERMOGRAPH_AUTH_SECRET via HMAC-SHA256 under a fixed
domain-separation label. A key-gap audit confirms that secret is already set in
every environment, so this stands up with no vault edit and no operator step.
Set the var explicitly only to rotate this surface independently.

HMAC under a distinct label rather than reusing the auth secret directly, so a
leak of this token can't be replayed as the session-signing key it came from.
The derivation is pinned to a shared cross-language test vector asserted on
both sides (config/derive_test.go and test_internal_token_derivation.py) — if
either drifts, CI fails, rather than every callback 401ing in a way that reads
like an auth bug.

With neither secret set, both ends fail closed: the router 404s and the daemon
refuses to start.

Security

hmac.compare_digest for the token; the whole router 404s when no token can be
established — fail closed, never default open. Caddy only routes /api/*,
/digest and /discord/interactions to the backend, so /internal/* was never
publicly reachable; the token is defence in depth. The router mounts before the
catch-all frontend proxy so /internal/* can't fall through to it.

Behaviour preserved, with the reasoning carried into the Go comments

Non-privileged intents (no MESSAGE_CONTENT, so no portal review); fatal close
codes 4004/4010–4014 stop rather than loop; the bot-author and self-author
mention-loop guard; allowed_mentions locked to {"parse":[],"replied_user":true}
so a crafted query can't turn a reply into an @everyone ping; the first cron
tick deferred a full interval rather than firing at boot, since warm-cities
already runs at deploy time; and no overlapping warm-cities run, which would
double-spend the archive-fetch quota.

Three deliberate improvements over the Python

A close intended for RESUME now uses 4000, not 1000 — Discord invalidates a
session closed 1000/1001, so the Python's default close silently defeated its
own resume.
MESSAGE_CREATE is handled on a bounded worker pool rather than an
unbounded thread hand-off, so a flood of mentions can't spawn unbounded work
against the backend.
A malformed HELLO now returns an error rather than a clean reconnect. A
clean return is the planned-reconnect path, which resets backoff and redials
with no sleep — a gateway stuck sending a bad HELLO would have become a tight
reconnect loop against Discord.

A .dockerignore is added because a disposable backend/.venv was being
swallowed by COPY . /app/ and duplicated by the chown layer, inflating the
image to 1.8 GB; it now builds at 578 MB.

Verification

Go: build, vet, test -race clean across all 4 packages; gofmt clean.
29 gateway tests cover every behaviour the deleted test_discord_bot.py
asserted.
Python: 365 passed, 7 skipped, including 10 internal-route tests
(fail-closed, auth, flag drop, per-job 409 guard) and 6 derivation tests.
docker build succeeds; binary runs as uid 10001 and exits 1 with a clear
message when it has nothing to derive from — verified in-image.
shellcheck -x: 0 findings (the guard from #19 covers the deploy.sh change
here).
Rebased onto current main, so it includes the Open-Meteo migration and the
domain-sha tag keying. That keying composes correctly: backend/daemon/ lives
under backend/, so a daemon-only change re-keys the backend image tag.

Deploy note

On beta the gateway is inert: THERMOGRAPH_DISCORD_BOT/_BOT_TOKEN are set
on prod only, so the daemon runs cron-only there. The gateway path first
activates when this reaches prod.

`web/app.py` started two long-lived background jobs under a leader election: the Discord gateway bot and an APScheduler. Both are stateful I/O loops — reconnect, RESUME, heartbeat, backoff, interval timers — living inside an async web app that also has to serve requests. This moves them into a single Go binary. ## The split **Go owns only the stateful I/O. It owns no climate or grading logic.** Anything needing data calls back into Python over a new internal-only surface (`/internal/discord/grade`, `/internal/jobs/warm-cities`, `/internal/jobs/indexnow`). That boundary is the whole design. Grading depends on polars and the parquet cache; reimplementing it in Go would let the bot's grades drift from the API's, and the slash-command path deliberately shares one grade builder so the two can never disagree. The grade route returns *gateway-ready* JSON — including the ephemeral-flag drop `discord_bot.py` used to do — and Go relays those bytes verbatim without ever parsing the embed. ## Packaging The binary is built by a `golang:1.26` stage in the backend Dockerfile and shipped **in the same image**, run as a second compose service off the **same tag**. The daemon and backend share the `/internal/*` contract, so they must never skew versions; one image makes that structural rather than a convention. Its entrypoint bypasses `entrypoint.sh` — the backend owns alembic, and two racing migrators is a real hazard. `deploy.sh` now rolls `daemon` alongside `backend`. Without that the service would never be created at all: a single-service deploy runs `up -d --no-deps <targets>`, so a `daemon` present only in compose would have been silently skipped and the bot would never have started. `replicas: 1` in the Swarm stack is load-bearing — Discord permits exactly one gateway connection per bot token, so the pin replaces `core/singleton.claim_leader` for this workload. `order: stop-first`, since start-first would briefly run two gateways. `autoscale.sh` targets `${STACK_NAME}_web` only, so it cannot scale this. ## No new secret to provision `THERMOGRAPH_INTERNAL_TOKEN` is **optional**. Unset, both ends derive the same token from `THERMOGRAPH_AUTH_SECRET` via HMAC-SHA256 under a fixed domain-separation label. A key-gap audit confirms that secret is already set in **every** environment, so this stands up with no vault edit and no operator step. Set the var explicitly only to rotate this surface independently. HMAC under a distinct label rather than reusing the auth secret directly, so a leak of this token can't be replayed as the session-signing key it came from. The derivation is pinned to a **shared cross-language test vector** asserted on both sides (`config/derive_test.go` and `test_internal_token_derivation.py`) — if either drifts, CI fails, rather than every callback 401ing in a way that reads like an auth bug. With neither secret set, both ends fail closed: the router 404s and the daemon refuses to start. ## Security `hmac.compare_digest` for the token; the whole router 404s when no token can be established — fail closed, never default open. Caddy only routes `/api/*`, `/digest` and `/discord/interactions` to the backend, so `/internal/*` was never publicly reachable; the token is defence in depth. The router mounts *before* the catch-all frontend proxy so `/internal/*` can't fall through to it. ## Behaviour preserved, with the reasoning carried into the Go comments Non-privileged intents (no `MESSAGE_CONTENT`, so no portal review); fatal close codes 4004/4010–4014 stop rather than loop; the bot-author and self-author mention-loop guard; `allowed_mentions` locked to `{"parse":[],"replied_user":true}` so a crafted query can't turn a reply into an `@everyone` ping; the first cron tick deferred a full interval rather than firing at boot, since warm-cities already runs at deploy time; and no overlapping warm-cities run, which would double-spend the archive-fetch quota. ## Three deliberate improvements over the Python - A close intended for RESUME now uses **4000, not 1000** — Discord invalidates a session closed 1000/1001, so the Python's default close silently defeated its own resume. - `MESSAGE_CREATE` is handled on a **bounded worker pool** rather than an unbounded thread hand-off, so a flood of mentions can't spawn unbounded work against the backend. - A malformed HELLO now returns an **error rather than a clean reconnect**. A clean return is the *planned*-reconnect path, which resets backoff and redials with no sleep — a gateway stuck sending a bad HELLO would have become a tight reconnect loop against Discord. A `.dockerignore` is added because a disposable `backend/.venv` was being swallowed by `COPY . /app/` and duplicated by the `chown` layer, inflating the image to **1.8 GB**; it now builds at **578 MB**. ## Verification - Go: `build`, `vet`, `test -race` clean across all 4 packages; `gofmt` clean. 29 gateway tests cover every behaviour the deleted `test_discord_bot.py` asserted. - Python: **365 passed, 7 skipped**, including 10 internal-route tests (fail-closed, auth, flag drop, per-job 409 guard) and 6 derivation tests. - `docker build` succeeds; binary runs as uid 10001 and exits 1 with a clear message when it has nothing to derive from — verified in-image. - `shellcheck -x`: 0 findings (the guard from #19 covers the `deploy.sh` change here). - Rebased onto current `main`, so it includes the Open-Meteo migration and the domain-sha tag keying. That keying composes correctly: `backend/daemon/` lives under `backend/`, so a daemon-only change re-keys the backend image tag. ## Deploy note On **beta** the gateway is inert: `THERMOGRAPH_DISCORD_BOT`/`_BOT_TOKEN` are set on prod only, so the daemon runs **cron-only** there. The gateway path first activates when this reaches prod.

admin_emi added 1 commit 2026-07-23 22:43:31 +00:00

daemon: move the Discord gateway and scheduler out of the web process into Go

shell-lint / shellcheck (pull_request) Successful in 8s

Details

PR build (required check) / build-backend (pull_request) Successful in 1m18s

Details

PR build (required check) / changes (pull_request) Successful in 6s

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

PR build (required check) / gate (pull_request) Successful in 3s

Details

secrets-guard / encrypted (pull_request) Successful in 5s

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

c3b906a9ed

web/app.py started two long-lived background jobs under a leader election: the
Discord gateway bot and an APScheduler. Both are stateful I/O loops -- reconnect,
RESUME, heartbeat, backoff, interval timers -- living inside an async web app
that also has to serve requests. This moves them into a single Go binary.

Go owns ONLY the stateful I/O. It owns no climate or grading logic: anything
needing data calls back into Python over a new internal-only HTTP surface
(/internal/discord/grade, /internal/jobs/warm-cities, /internal/jobs/indexnow).
Grading depends on polars and the parquet cache; reimplementing it in Go would
make the bot's grades drift from the API's, and the slash-command path
deliberately shares one grade builder so the two can never disagree. The grade
route returns gateway-ready JSON -- including the ephemeral-flag drop that
discord_bot.py used to do -- and Go relays those bytes verbatim without parsing
the embed.

Packaging: the binary is built by a golang:1.26 stage in the backend Dockerfile
and shipped in the SAME image, run as a second compose service off the SAME tag.
The daemon and backend share the /internal/* contract, so they must never skew
versions; one image makes that structural rather than a convention. Its
entrypoint bypasses entrypoint.sh -- the backend owns alembic, and two racing
migrators is a real hazard.

replicas: 1 in the Swarm stack is load-bearing. Discord permits exactly one
gateway connection per bot token; the pin replaces core/singleton.claim_leader
for this workload. update_config uses order: stop-first, since start-first would
briefly run two gateways. autoscale.sh targets ${STACK_NAME}_web only, so it
cannot scale this.

Security: the internal routes compare the token with hmac.compare_digest and the
whole router 404s when THERMOGRAPH_INTERNAL_TOKEN is unset -- fail closed, never
default open. Caddy only routes /api/*, /digest and /discord/interactions to the
backend, so /internal/* was never publicly reachable; the token is defence in
depth. The router mounts before the catch-all frontend proxy so /internal/*
cannot fall through to it. The daemon refuses to start without the token.

Behaviour preserved from the Python, with the reasoning carried into the Go
comments: non-privileged intents (no MESSAGE_CONTENT, so no portal review);
fatal close codes 4004/4010-4014 stop rather than loop; the bot-author and
self-author mention-loop guard; allowed_mentions locked to {"parse":[],
"replied_user":true} so a crafted query cannot turn a reply into an @everyone
ping; the first cron tick deferred one full interval rather than firing at boot,
since warm-cities already runs at deploy time; and no overlapping warm-cities
run, which would double-spend the archive-fetch quota.

Two deliberate improvements over the Python. A close intended for RESUME now
uses 4000 rather than 1000 -- Discord invalidates a session closed 1000/1001, so
the Python's default close silently defeated its own resume. And MESSAGE_CREATE
is handled on a bounded worker pool rather than an unbounded thread hand-off, so
a flood of mentions cannot spawn unbounded work against the backend.

A .dockerignore is added because a disposable backend/.venv was being swallowed
by COPY . /app/ and duplicated again by the chown layer, inflating the image to
1.8 GB; it builds at 578 MB.

Tests: 29 Go gateway tests covering every behaviour the deleted
test_discord_bot.py asserted, plus cron/config/apiclient suites; 10 new Python
tests for the internal routes (fail-closed, auth, flag drop, per-job 409 guard).
Full suite 359 passed / 7 skipped; go build, vet and test -race clean.

admin_emi force-pushed go-daemon from c3b906a9ed to adf824b33f

2026-07-23 22:47:13 +00:00

Compare