thermograph/backend/api/internal_routes.py
Emi Griffith c3b906a9ed
All checks were successful
shell-lint / shellcheck (pull_request) Successful in 8s
PR build (required check) / build-backend (pull_request) Successful in 1m18s
PR build (required check) / changes (pull_request) Successful in 6s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 3s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / validate-observability (pull_request) Has been skipped
daemon: move the Discord gateway and scheduler out of the web process into Go
web/app.py started two long-lived background jobs under a leader election: the
Discord gateway bot and an APScheduler. Both are stateful I/O loops -- reconnect,
RESUME, heartbeat, backoff, interval timers -- living inside an async web app
that also has to serve requests. This moves them into a single Go binary.

Go owns ONLY the stateful I/O. It owns no climate or grading logic: anything
needing data calls back into Python over a new internal-only HTTP surface
(/internal/discord/grade, /internal/jobs/warm-cities, /internal/jobs/indexnow).
Grading depends on polars and the parquet cache; reimplementing it in Go would
make the bot's grades drift from the API's, and the slash-command path
deliberately shares one grade builder so the two can never disagree. The grade
route returns gateway-ready JSON -- including the ephemeral-flag drop that
discord_bot.py used to do -- and Go relays those bytes verbatim without parsing
the embed.

Packaging: the binary is built by a golang:1.26 stage in the backend Dockerfile
and shipped in the SAME image, run as a second compose service off the SAME tag.
The daemon and backend share the /internal/* contract, so they must never skew
versions; one image makes that structural rather than a convention. Its
entrypoint bypasses entrypoint.sh -- the backend owns alembic, and two racing
migrators is a real hazard.

replicas: 1 in the Swarm stack is load-bearing. Discord permits exactly one
gateway connection per bot token; the pin replaces core/singleton.claim_leader
for this workload. update_config uses order: stop-first, since start-first would
briefly run two gateways. autoscale.sh targets ${STACK_NAME}_web only, so it
cannot scale this.

Security: the internal routes compare the token with hmac.compare_digest and the
whole router 404s when THERMOGRAPH_INTERNAL_TOKEN is unset -- fail closed, never
default open. Caddy only routes /api/*, /digest and /discord/interactions to the
backend, so /internal/* was never publicly reachable; the token is defence in
depth. The router mounts before the catch-all frontend proxy so /internal/*
cannot fall through to it. The daemon refuses to start without the token.

Behaviour preserved from the Python, with the reasoning carried into the Go
comments: non-privileged intents (no MESSAGE_CONTENT, so no portal review);
fatal close codes 4004/4010-4014 stop rather than loop; the bot-author and
self-author mention-loop guard; allowed_mentions locked to {"parse":[],
"replied_user":true} so a crafted query cannot turn a reply into an @everyone
ping; the first cron tick deferred one full interval rather than firing at boot,
since warm-cities already runs at deploy time; and no overlapping warm-cities
run, which would double-spend the archive-fetch quota.

Two deliberate improvements over the Python. A close intended for RESUME now
uses 4000 rather than 1000 -- Discord invalidates a session closed 1000/1001, so
the Python's default close silently defeated its own resume. And MESSAGE_CREATE
is handled on a bounded worker pool rather than an unbounded thread hand-off, so
a flood of mentions cannot spawn unbounded work against the backend.

A .dockerignore is added because a disposable backend/.venv was being swallowed
by COPY . /app/ and duplicated again by the chown layer, inflating the image to
1.8 GB; it builds at 578 MB.

Tests: 29 Go gateway tests covering every behaviour the deleted
test_discord_bot.py asserted, plus cron/config/apiclient suites; 10 new Python
tests for the internal routes (fail-closed, auth, flag drop, per-job 409 guard).
Full suite 359 passed / 7 skipped; go build, vet and test -race clean.
2026-07-23 15:42:44 -07:00

158 lines
6.8 KiB
Python

"""Internal control surface for the Go daemon (backend/daemon/).
The gateway bot and the recurring worker jobs moved out of this process into
`thermograph-daemon` — Go owns only the long-lived stateful I/O (the Discord
websocket, RESUME/heartbeat/backoff, interval timers). Anything that needs
*data* calls back in here, because grading depends on polars + the parquet
cache: reimplementing it out-of-process would let the bot's grades drift from
the API's, and the slash-command path deliberately shares one grade builder so
the two never drift. These routes hand the daemon finished answers, keeping it
entirely out of the grading contract.
Not publicly reachable: the deploy Caddyfile routes only /api/*, /digest and
/discord/interactions to this backend, so /internal/* never resolves through
the public domain. The shared-secret header is defence in depth on top of
that, not the only control — and it fails closed: with no token configured
the whole surface answers 404, never "open".
Deliberately NOT under BASE (same posture as /healthz): the daemon targets
THERMOGRAPH_API_BASE_INTERNAL directly and shouldn't need to know
THERMOGRAPH_BASE to construct a path.
"""
import hashlib
import hmac
import os
import threading
from fastapi import APIRouter, Depends, Header, HTTPException
from pydantic import BaseModel
_TOKEN_ENV = "THERMOGRAPH_INTERNAL_TOKEN"
_HEADER = "X-Thermograph-Internal-Token"
# Domain-separation label for the derived token. MUST match the Go daemon's
# internal/config constant byte-for-byte or every callback 401s.
_DERIVE_LABEL = b"thermograph/internal-api/v1"
def resolve_internal_token() -> str:
"""The shared secret guarding /internal/*, or "" if it can't be established.
An explicit THERMOGRAPH_INTERNAL_TOKEN always wins. Otherwise it is DERIVED
from THERMOGRAPH_AUTH_SECRET, which every environment already provisions, so
standing up the daemon needs no new vault entry and no operator step -- one
less secret to rotate, leak, or forget on a new host.
HMAC with a distinct label rather than reusing AUTH_SECRET directly: that is
ordinary domain separation, so a leak of this token can't be replayed as the
session-signing key it was derived from, and the derivation is one-way.
Deliberately reads THERMOGRAPH_AUTH_SECRET from the environment rather than
users.py's effective value: users.py falls back to a random per-process
secret when the env var is unset, and two processes would then derive two
different tokens and fail every call in a way that looks like a bug rather
than missing config. Unset env var => "" => the surface stays closed."""
explicit = os.environ.get(_TOKEN_ENV, "").strip()
if explicit:
return explicit
auth_secret = os.environ.get("THERMOGRAPH_AUTH_SECRET", "").strip()
if not auth_secret:
return ""
return hmac.new(auth_secret.encode(), _DERIVE_LABEL, hashlib.sha256).hexdigest()
def _require_internal_token(
x_thermograph_internal_token: str = Header(default=""),
) -> None:
"""Gate every internal route on the shared secret.
Read from the environment per request (not at import) so the fail-closed
behaviour is a property of the running config, not of import order — and so
tests can exercise both states without re-importing the app. An unset/empty
token means the surface was never provisioned: 404 (the route doesn't
exist), matching the metrics endpoint's don't-reveal posture. A present but
wrong header is a real auth failure: 401. compare_digest, not ==, so the
comparison doesn't leak the token's prefix through response timing."""
token = resolve_internal_token()
if not token:
raise HTTPException(status_code=404)
if not hmac.compare_digest(x_thermograph_internal_token, token):
raise HTTPException(status_code=401, detail="bad internal token")
# include_in_schema=False: an internal surface has no business in the public
# OpenAPI document, same as /api/v2/metrics.
router = APIRouter(prefix="/internal", include_in_schema=False,
dependencies=[Depends(_require_internal_token)])
# Every handler below is a plain `def`, not `async def`, on purpose: they all do
# blocking work (sync parquet/DB reads, grading math, upstream HTTP fetches).
# FastAPI runs sync handlers on its threadpool, so a slow grade or a long warm
# run never stalls the event loop the rest of the API is serving on — the same
# reason web/app.py wraps discord_interactions.handle in run_in_threadpool.
class _GradeBody(BaseModel):
query: str
@router.post("/discord/grade")
def discord_grade(body: _GradeBody) -> dict:
"""The gateway-ready reply for one bot mention/DM: the same embed the /grade
slash command builds, via the shared discord_interactions._grade_message —
reusing one builder is what keeps the two paths from ever drifting. Local
import for the same reason the in-process gateway bot used one: don't drag
the interactions module's transitive deps into contexts that only mount the
router."""
from notifications import discord_interactions
data = discord_interactions._grade_message(body.query)
# Gateway messages can't be ephemeral — the flag is interactions-only, and
# Discord rejects it outside an interaction response.
data.pop("flags", None)
return data
# One in-flight run per job, enforced here rather than trusted to the daemon's
# timers: warm_cities spends the Open-Meteo quota, so a stacked second run
# double-spends it, and a slow run must never let invocations pile up holding
# requests open. Non-blocking acquire => the loser answers 409 immediately.
_JOB_LOCKS: dict[str, threading.Lock] = {
"warm-cities": threading.Lock(),
"indexnow": threading.Lock(),
}
def _run_exclusive(job: str, fn) -> dict:
lock = _JOB_LOCKS[job]
if not lock.acquire(blocking=False):
raise HTTPException(status_code=409, detail=f"{job} is already running")
try:
fn()
finally:
lock.release()
return {"ok": True}
@router.post("/jobs/warm-cities")
def jobs_warm_cities() -> dict:
"""(Re-)warm the curated city set. The daemon owns the cadence (default
daily — every already-cached city is a cheap skip); this endpoint just runs
one pass. Local import: warm_cities is a script-style top-level module, so
only the process that actually runs the job pays its import cost."""
def _run():
import warm_cities
warm_cities.main()
return _run_exclusive("warm-cities", _run)
@router.post("/jobs/indexnow")
def jobs_indexnow() -> dict:
"""Check whether the indexable URL set changed and, if so, ping IndexNow.
submit_if_changed() is a cheap no-op when nothing changed, which is why the
daemon can tick this more often than warm-cities without spending anything
on most runs."""
def _run():
import indexnow
indexnow.submit_if_changed()
return _run_exclusive("indexnow", _run)