main took two direct PRs (#19 shell-lint, #21 the thermograph-daemon Go
service) while dev accumulated the ERA5 lake stack; both sides added compose/
stack services and touched the same seams. Resolutions keep both worlds:
- compose + Swarm stack: lake AND daemon are sibling services; backend env
carries THERMOGRAPH_LAKE_URL and THERMOGRAPH_INTERNAL_TOKEN.
- deploy.sh: backend deploys roll (backend lake daemon); all adds frontend.
- requirements.txt: websockets/apscheduler stay removed (moved to the Go
daemon), duckdb stays (the lake role's engine).
Merged tree: backend suite green, both compose configs validate.
Adds .forgejo/workflows/shell-lint.yml (pinned shellcheck v0.11.0 + sha256, -x,
default severity, fail on any finding, not path-filtered) and drives all 26
scripts to zero findings.
Two defects shellcheck cannot see:
render-secrets.sh left DECRYPTED vault contents in /tmp whenever a sops decrypt
failed -- the caller's set -e aborted the function before either cleanup ran.
Now removed on every exit path, with `|| return 1` on both sops calls so a
decrypt failure can never write a partial /etc/thermograph.env regardless of the
caller's shell options. Explicitly not a `trap ... RETURN`: such a trap set in a
sourced function persists into the caller's shell and re-fires when the caller's
next `.`/source completes, where the function-local tmp is unset -- fatal and
silent under deploy.sh's set -u. The file now records that reasoning.
autoscale.sh ran `set -eu` without pipefail while piping docker stats into awk,
so a failed left side was swallowed and the loop autoscaled on empty input.
Promoted to pipefail with a missed sample treated as a skip; verified busybox ash
in docker:27-cli supports it.
Also: capture-fixtures.sh's `jq . || cat` ran cat after jq had consumed stdin,
silently writing truncated fixtures; deploy.sh/deploy-stack.sh `# shellcheck
source=` paths corrected for the monorepo layout.
Prod DB dumps and Forgejo (db + data volume) are now copied off-box to the
era5-thermograph S3 bucket, age-encrypted to the vault recipient so the host
age key at /etc/thermograph/age.key decrypts them for restore. Nothing plaintext
leaves the box; no local intermediate (streamed via age | rclone rcat).
ops-cron: the prod pg_dump job gains an off-box push; a new forgejo-backup job
dumps Forgejo's Postgres + tars its data volume on beta (the git host had NO
backup at all). 30-day off-box retention on both. S3 creds are Forgejo Actions
secrets (S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY), mirrored into the SOPS vault
(prod.yaml/beta.yaml) for host-side use. Contabo needs path-style addressing.
Adds deploy/backup/README.md with the DR restore runbook.
