Promote main to release (deploy prod): ERA5 lake stack + Go daemon #29

Merged

admin_emi merged 14 commits from main into release 2026-07-24 03:28:07 +00:00

Conversation 0 Commits 14 Files changed 102 +11285 -708

admin_emi commented 2026-07-24 03:27:54 +00:00

Owner

Copy link

Lake service (bucket-hosted ERA5 history primary, DuckDB SQL indexer, 1-2 autoscaled replicas), Go daemon (Discord gateway + scheduler), reconciled stack. Beta QA green incl. source=era5-lake on a fresh cell. Extraction ~45% and filling the bucket independently; uncovered cells fall through to NASA per-cell.

Lake service (bucket-hosted ERA5 history primary, DuckDB SQL indexer, 1-2 autoscaled replicas), Go daemon (Discord gateway + scheduler), reconciled stack. Beta QA green incl. source=era5-lake on a fresh cell. Extraction ~45% and filling the bucket independently; uncovered cells fall through to NASA per-cell.

admin_emi added 14 commits 2026-07-24 03:27:55 +00:00

Merge pull request 'Reconcile: merge main back into dev' (#13) from main into dev ebf5efadfb

ERA5 lake: bucket-hosted history primary + SQL indexer service (#15) 370a4ffdc1

Vault the ERA5 lake bucket credentials (prod) (#16) d9537798c6

Parallelize lake tile uploads (#17) 21eea3a4c2

shell: add shellcheck CI guard and drive the tree to zero findings (#19) ... de8e847f9f

Adds .forgejo/workflows/shell-lint.yml (pinned shellcheck v0.11.0 + sha256, -x,
default severity, fail on any finding, not path-filtered) and drives all 26
scripts to zero findings.

Two defects shellcheck cannot see:

render-secrets.sh left DECRYPTED vault contents in /tmp whenever a sops decrypt
failed -- the caller's set -e aborted the function before either cleanup ran.
Now removed on every exit path, with `|| return 1` on both sops calls so a
decrypt failure can never write a partial /etc/thermograph.env regardless of the
caller's shell options. Explicitly not a `trap ... RETURN`: such a trap set in a
sourced function persists into the caller's shell and re-fires when the caller's
next `.`/source completes, where the function-local tmp is unset -- fatal and
silent under deploy.sh's set -u. The file now records that reasoning.

autoscale.sh ran `set -eu` without pipefail while piping docker stats into awk,
so a failed left side was swallowed and the loop autoscaled on empty input.
Promoted to pipefail with a missed sample treated as a skip; verified busybox ash
in docker:27-cli supports it.

Also: capture-fixtures.sh's `jq . || cat` ran cat after jq had consumed stdin,
silently writing truncated fixtures; deploy.sh/deploy-stack.sh `# shellcheck
source=` paths corrected for the monorepo layout.

Run the ERA5 lake service in every environment (#20) 8e09155aaf

Survive Contabo PUT throttling in the lake extractor (#22) a787f79f92

daemon: move the Discord gateway and scheduler out of the web process into Go (#21) ... 2d3f37c474

The gateway bot and APScheduler were long-lived stateful I/O loops running
inside the async web app under a leader election. They move into a single Go
binary that owns ONLY that I/O -- websocket, RESUME, heartbeat, backoff, timers.

It owns no grading logic. Anything needing data calls back over a new
internal-only surface (/internal/discord/grade, /internal/jobs/*). Grading
depends on polars and the parquet cache; reimplementing it in Go would let the
bot's grades drift from the API's. The grade route returns gateway-ready JSON
and Go relays the bytes verbatim.

The binary ships in the backend image and runs as a second compose service off
the same tag, so the two ends of the /internal/* contract can never skew.
deploy.sh rolls daemon alongside backend -- without that the service would never
be created, since a single-service deploy uses --no-deps. It also probes the
image first and skips the daemon when rolling a tag that predates the binary:
infra tracks main while image tags are env-staged, so a host can legitimately be
asked to roll an older backend image, and creating the service anyway would
leave a container crash-looping on a missing binary.

replicas: 1 with order: stop-first replaces the leader election -- Discord
permits one gateway connection per bot token.

THERMOGRAPH_INTERNAL_TOKEN is optional: both ends derive it from
THERMOGRAPH_AUTH_SECRET via HMAC under a domain-separation label, so this needs
no new vault entry. The derivation is pinned to a shared cross-language test
vector asserted on both sides, so drift fails CI instead of 401ing every call.
Fail closed when neither secret is set.

Improvements over the Python: a close intended for RESUME uses 4000 rather than
1000 (Discord invalidates a session closed 1000, so the old default defeated its
own resume); MESSAGE_CREATE runs on a bounded worker pool; and a malformed HELLO
returns an error rather than a clean reconnect, which would otherwise reset
backoff and hot-loop against the gateway.

365 Python tests pass; Go build/vet/test -race clean; shellcheck 0 findings.

Own /state in the image; lake cache degrades instead of failing (#25) 6396d553d2

Iceberg conversion container for the ERA5 lake (infra/lake-iceberg) (#24) 16bddb6625

Reconcile: merge main (shellcheck guard, Go daemon) into dev ... 30b0c70bbb

main took two direct PRs (#19 shell-lint, #21 the thermograph-daemon Go
service) while dev accumulated the ERA5 lake stack; both sides added compose/
stack services and touched the same seams. Resolutions keep both worlds:

- compose + Swarm stack: lake AND daemon are sibling services; backend env
  carries THERMOGRAPH_LAKE_URL and THERMOGRAPH_INTERNAL_TOKEN.
- deploy.sh: backend deploys roll (backend lake daemon); all adds frontend.
- requirements.txt: websockets/apscheduler stay removed (moved to the Go
  daemon), duckdb stays (the lake role's engine).

Merged tree: backend suite green, both compose configs validate.

Merge pull request 'Reconcile: merge main (shellcheck, Go daemon) into dev' (#27) from reconcile/main-into-dev into dev 4e3c7e82c6

Merge pull request 'Promote dev to main (deploy beta): ERA5 lake stack' (#26) from dev into main 755886e244

frontend: rewrite the SSR content service in Go (#28) ... 92e74c585a

Ports frontend/ (Jinja2/FastAPI, ~1180 LOC) to Go with html/template. No
climate math, no DB, no auth -- every route fetches from the backend's
/content/* API.

Verified with a golden-HTML diff, not just unit tests: both the Python
original and the Go rewrite were run against the same committed fixtures
and every route compared byte-for-byte, confirmed programmatically. That
process caught defects unit tests alone missed, since map[string]any has
no compile-time field check:

- Render-context keys were snake_case throughout while the templates read
  PascalCase fields. A missing map key doesn't error, it silently renders
  empty -- title, meta description, canonical URL, OpenGraph tags, and the
  homepage's entire ranked list were blank on every page despite every
  route returning 200. Fixed by renaming every key to match each
  template's own documented field contract, and passing API structs
  straight through wherever their fields already matched (removes a whole
  layer of future drift risk).
- Three pages 500'd: ToolHref needed a composed href, not a bare
  "lat,lon" fragment; the records table needed the raw API struct.
- JSON-LD was double-encoded: <script type="application/ld+json"> is
  JAVASCRIPT context to html/template's escaper regardless of the
  script's type attribute, so template.HTML gets re-escaped as a quoted
  JS string. Needed template.JS. The glossary term page's JSON-LD was
  never built at all -- added.
- html/template silently strips literal HTML and JS comments from parsed
  output (verified in isolation) -- both need a FuncMap function
  returning template.HTML/template.JS to survive.

Packaging: 187MB -> 22.6MB. Two defects caught before reaching a host: the
Swarm stack's entrypoint override with no explicit command drops the
image's CMD entirely (every deploy would have exited 127), and
COPY --chown by name fails under the classic Docker builder on Alpine.
Both fixed.

go build/vet/test -race clean; docker build passes its embedded test step
under both BuildKit and the classic builder; shellcheck 0 findings.

admin_emi merged commit f8481ecf70 into release 2026-07-24 03:28:07 +00:00

admin_emi referenced this pull request from a commit 2026-07-24 03:28:08 +00:00

Merge pull request 'Promote main to release (deploy prod): ERA5 lake stack + Go daemon' (#29) from main into release

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

  

Compat/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

No labels

Compat/Breaking

Kind/Bug

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Jinemi/thermograph#29

No description provided.