Four holes, all of which let untested or unchecked code reach an environment.
- PRs into `release` ran no build and no test. release is the branch that deploys
to prod, so the intended promotion path was the least-checked path in the repo:
the last seven release PRs ran shellcheck and the secrets check, nothing else.
pr-build now triggers on it.
- shell-lint and secrets-guard reported as standalone statuses outside `gate`,
and pr-build's own setup notes tell the operator to require only `gate`. Taken
literally that means a commit adding a plaintext SOPS file was mergeable. Both
are now called from pr-build and reduced into gate. They are deliberately not
domain-gated and not `needs: changes`: they are repo-wide invariants that must
hold on every PR, including one touching no app domain -- exactly the case
where every domain job skips and gate used to pass vacuously. For the same
reason `skipped` counts as a failure for these two, while it stays a pass for
the domain jobs it legitimately describes.
They keep their standalone pull_request trigger, so they will run twice on a
PR until branch protection is confirmed to require only `gate`. Cheap, and it
avoids breaking a rule that may still require them by name.
- build-push.yml built and pushed with no test step, and deploy.yml consumes a
TAG rather than a commit status -- so an image reaching the registry by any
route other than a dev/main PR (a direct push, a dispatch, a v*.*.* tag) was
never tested by CI, and beta/prod then rolled it. The test now sits between
build and push, so the artifact that deploys is the artifact that was tested.
Gating the artifact is what makes this work; gating the branch would not.
- backend/Dockerfile ran `go build` on daemon/ but never its tests. All 48 of
them (gateway, cron, apiclient, config) ran nowhere: `grep -rn "go test"
.forgejo/workflows/` found nothing, while the daemon owns the prod Discord
gateway websocket and every recurring-job timer. It now runs gofmt + vet +
test before the build, which is the pattern frontend/Dockerfile already uses --
and being inside `docker build` it gates build-push too. Verified clean: gofmt
reports nothing, vet passes, all four packages pass.
A test run could message real subscribers, and would have reported green while
doing it.
Every send gate in notifications/ reads its config from ambient env at import
time, and conftest neutralised none of it -- it set four env vars, none touching
Discord, SMTP or VAPID. test_notify.py calls run_pass() seven times without
patching notifications.discord, and notify.py:358 reaches
discord.post_subscription_alert() for every notification it creates. The
subscription channel notifies real people.
The failure mode is silent: notify.py's send path and discord.py's _bot_post
swallow every exception, so a real send raises nothing and no assertion goes red.
An operator who had sourced /etc/thermograph.env to debug -- routine -- turned
`pytest tests/notifications` into a live broadcast. CI was safe only by omission:
it runs in a container with no credentials baked in, and one added -e removed
that.
Two layers:
1. Config blanked, so every enabled()/dm_enabled()/subscription_enabled() gate
reports False. This is the state CI already runs in, now guaranteed locally
instead of depending on the shell.
2. The transports themselves replaced with sentinels that raise. A test that
forgets to patch one fails loudly rather than sending.
Tests that need to exercise a send path patch these themselves, which is the
point: the escape hatch becomes explicit in the test that takes it. The idiom is
lifted from tests/test_warm_content.py, which already did this for one call.
test_send_safety.py asserts the guarantee, because a block like this fails in
only one direction -- weaken it and every other test still passes, the sole
symptom being a message delivered to somebody.
Full suite: 457 passed, 8 skipped. Nothing relied on a live transport.
The SELECT guard was a denylist of DDL/DML verbs. DuckDB's file readers and
setting introspection are plain SELECTs, so it passed all of these:
SELECT * FROM read_csv_auto('/etc/passwd') -> 200, file contents
SELECT * FROM glob('/etc/*') -> 200, directory listing
SELECT current_setting('s3_secret_access_key') -> 200, the bucket credentials
The last one is the worst: bucket mode installed the S3 credentials with
SET s3_secret_access_key, and settings are readable back out. So one
unauthenticated POST returned the ERA5 bucket key in plaintext -- precisely what
the module docstring says this service exists to avoid ("without shipping S3
credentials to every web task"). /query had no authentication of any kind, while
the comparable /internal/* router has gated its whole surface since it landed.
Three changes, outermost first:
- /query now requires the shared internal token, reusing internal_routes'
dependency. Nothing calls it: the only in-repo caller of this service is
era5lake.py hitting /history, and Centralis reaches the lake through its own
DuckDB rather than this endpoint. /history is deliberately left open for now --
gating it means threading the token through the web->lake hot path.
- Credentials are installed as a DuckDB SECRET instead of SET s3_*. Secret values
are opaque: current_setting() returns NULL and duckdb_secrets() lists the name
and scope but no key material.
- Bucket mode disables LocalFileSystem after the views are created (they resolve
lazily, so ordering matters) and locks the configuration. Local mode cannot do
this -- its views read local parquet -- so the function denylist stays as
defence in depth for dev and tests.
The new guard tests assert the *guard* refused, not merely that the request
failed: several of these shapes (read_parquet on a text file, a bare
'/etc/passwd') error inside DuckDB anyway, so a status-only assertion would keep
passing with the guard deleted. That is the trap the original four-case test fell
into -- it tested negatives shaped like the regex that had been written.
After a cell's archive advances, the first content request recomputes a 45yr payload cold. Add warm_cities.warm_content — idempotent, cache-only, bounded — that pre-populates the content derived-store under the same (kind,key,content_token) the routes read, wired into the leader-gated notifier loop (50 cities/30-min tick) so the ~1000-city set stays warm within hours of each daily advance.
The three content handlers loaded the full ~45yr archive before checking the cache, so even hits paid the load. Restructure to resolve cell + compute content_token cheaply first, check the store, and load history only inside the miss-only build() closure. Cache hits and 304s now return without touching the archive. Keys/kinds/ETag unchanged.
Content pages keyed cache validity on the full-history hist_end, invalidating ~1-2x/day and paying a 45yr load even to compute the token. Add content_token(cell_id) = PAYLOAD_VER:CONTENT_VER:max_date, backed by an indexed MAX(date) (climate_store.history_max_date / climate.history_max_date), so it survives intra-day top-ups and needs no full-history load.
main took two direct PRs (#19 shell-lint, #21 the thermograph-daemon Go
service) while dev accumulated the ERA5 lake stack; both sides added compose/
stack services and touched the same seams. Resolutions keep both worlds:
- compose + Swarm stack: lake AND daemon are sibling services; backend env
carries THERMOGRAPH_LAKE_URL and THERMOGRAPH_INTERNAL_TOKEN.
- deploy.sh: backend deploys roll (backend lake daemon); all adds frontend.
- requirements.txt: websockets/apscheduler stay removed (moved to the Go
daemon), duckdb stays (the lake role's engine).
Merged tree: backend suite green, both compose configs validate.
The gateway bot and APScheduler were long-lived stateful I/O loops running
inside the async web app under a leader election. They move into a single Go
binary that owns ONLY that I/O -- websocket, RESUME, heartbeat, backoff, timers.
It owns no grading logic. Anything needing data calls back over a new
internal-only surface (/internal/discord/grade, /internal/jobs/*). Grading
depends on polars and the parquet cache; reimplementing it in Go would let the
bot's grades drift from the API's. The grade route returns gateway-ready JSON
and Go relays the bytes verbatim.
The binary ships in the backend image and runs as a second compose service off
the same tag, so the two ends of the /internal/* contract can never skew.
deploy.sh rolls daemon alongside backend -- without that the service would never
be created, since a single-service deploy uses --no-deps. It also probes the
image first and skips the daemon when rolling a tag that predates the binary:
infra tracks main while image tags are env-staged, so a host can legitimately be
asked to roll an older backend image, and creating the service anyway would
leave a container crash-looping on a missing binary.
replicas: 1 with order: stop-first replaces the leader election -- Discord
permits one gateway connection per bot token.
THERMOGRAPH_INTERNAL_TOKEN is optional: both ends derive it from
THERMOGRAPH_AUTH_SECRET via HMAC under a domain-separation label, so this needs
no new vault entry. The derivation is pinned to a shared cross-language test
vector asserted on both sides, so drift fails CI instead of 401ing every call.
Fail closed when neither secret is set.
Improvements over the Python: a close intended for RESUME uses 4000 rather than
1000 (Discord invalidates a session closed 1000, so the old default defeated its
own resume); MESSAGE_CREATE runs on a bounded worker pool; and a malformed HELLO
returns an error rather than a clean reconnect, which would otherwise reset
backoff and hot-loop against the gateway.
365 Python tests pass; Go build/vet/test -race clean; shellcheck 0 findings.
Adds .forgejo/workflows/shell-lint.yml (pinned shellcheck v0.11.0 + sha256, -x,
default severity, fail on any finding, not path-filtered) and drives all 26
scripts to zero findings.
Two defects shellcheck cannot see:
render-secrets.sh left DECRYPTED vault contents in /tmp whenever a sops decrypt
failed -- the caller's set -e aborted the function before either cleanup ran.
Now removed on every exit path, with `|| return 1` on both sops calls so a
decrypt failure can never write a partial /etc/thermograph.env regardless of the
caller's shell options. Explicitly not a `trap ... RETURN`: such a trap set in a
sourced function persists into the caller's shell and re-fires when the caller's
next `.`/source completes, where the function-local tmp is unset -- fatal and
silent under deploy.sh's set -u. The file now records that reasoning.
autoscale.sh ran `set -eu` without pipefail while piping docker stats into awk,
so a failed left side was swallowed and the loop autoscaled on empty input.
Promoted to pipefail with a missed sample treated as a skip; verified busybox ash
in docker:27-cli supports it.
Also: capture-fixtures.sh's `jq . || cat` ran cat after jq had consumed stdin,
silently writing truncated fixtures; deploy.sh/deploy-stack.sh `# shellcheck
source=` paths corrected for the monorepo layout.
Steady-state usage of the Open-Meteo API is removed; it remains only as a dormant fallback. Geocoding -> local GeoNames + Nominatim; wind gusts -> Meteostat; history -> NASA POWER (curated cells seeded with keyless ERA5); recent+forecast -> NASA range + MET Norway. Plus a drift-check tool comparing NASA vs Open-Meteo. All keyless, no new infra. Backend suite green in-image (gate).
One root workflow set replaces the four repos' copies (deleted -- root-only
is where Forgejo reads them, and dead copies are a trap): per-domain
build-push with explicit image paths (emi/thermograph/backend|frontend; the
old github.repository-derived path collides in a monorepo), path-filtered
per-domain beta/prod/dev deploys, a domain-input reusable build check, a
single always-reporting PR gate (path-filtered required checks deadlock
auto-merge), a new infra-sync pipeline (host checkout + secrets render on
infra/** pushes), and ports of secrets-guard / ops-cron /
observability-validate to monorepo paths.
