The lake is one shared warehouse (s3://era5-thermograph/iceberg), so the
environments differ only in where the engine runs: a throwaway docker run
of a small DuckDB image (httpfs + iceberg extensions baked in) on the
target box, locally for LAN dev and over SSH for beta/prod. No daemon, no
listening port, no tunnel, and no container names to resolve.
No catalog service: each table's newest metadata JSON is resolved at call
time and exposed as a view named after its table directory, so reads track
a table that is still being loaded.
Sessions lock themselves down before user SQL runs (allowed_directories
pinned to the lake prefixes, external access disabled, configuration
locked), so writes to backups/ or local files are refused by the engine
itself. Credentials resolve at call time from the target host's env file
or a caller-side sops decrypt, handed over via stdin, never argv.
Handoff for the Iceberg worker: the contract iceberg.sh must satisfy (same
interface shape as dbq.sh, read-only enforced by the system, no new network
endpoints, call-time container resolution), the environment constraints that
will otherwise bite -- prod's Swarm overlay is unroutable from its own host so
tunnels are impossible, Contabo object storage requires path-style addressing,
the backups/ prefix is live, and beta's /etc/thermograph.env is unreadable by
the deploy user -- plus the decisions to report back, acceptance criteria with
a demonstrated refused write, and anti-goals.
Uniform read-only query access to the LAN dev, beta and prod databases, plus a
thermograph_ro role in each to enforce it.
None of the databases are exposed over TCP: each listens only on its private
docker network, and prod's is a Swarm overlay the host cannot route to -- so
ssh -L works for beta but is impossible for prod, and publishing 5432 would mean
new ufw rules and a Swarm endpoint change on production. Running psql inside the
db container works identically everywhere with no ports, tunnels or infra
changes, so dbq.sh does that: local docker exec for dev, ssh for beta/prod. The
prod container is a Swarm task whose name changes each redeploy, so it is
resolved at call time rather than hardcoded.
Queries connect as thermograph_ro (NOSUPERUSER, granted only pg_read_all_data),
so a stray write fails with "permission denied" even against prod; the app's own
superuser role is deliberately unused here. Extra args pass through to psql and
stdin is forwarded.
ops/README.md documents usage and the conventions Iceberg will follow when it
lands as a sibling (env-keyed dispatch, run the engine where the data is
reachable, dedicated read-only identity).
Steady-state usage of the Open-Meteo API is removed; it remains only as a dormant fallback. Geocoding -> local GeoNames + Nominatim; wind gusts -> Meteostat; history -> NASA POWER (curated cells seeded with keyless ERA5); recent+forecast -> NASA range + MET Norway. Plus a drift-check tool comparing NASA vs Open-Meteo. All keyless, no new infra. Backend suite green in-image (gate).
The cutover moved the host checkout's deploy tree to /opt/thermograph/infra/deploy/,
but the stack file's absolute bind sources (db init, env-entrypoint.sh, autoscale.sh)
still pointed at the old /opt/thermograph/deploy/ -- a service update kept the stale
mounts and new tasks failed with 'bind source path does not exist'.
Backend runs its full hermetic suite, frontend its unit tier, inside the
just-built image -- the dev-only feature both app repos carried in their own
build.yml (deleted here in favor of the root workflow).
/opt/thermograph becomes a monorepo checkout: deploy.sh gains INFRA_DIR (git
ops at the root, compose work cd'd into infra/), lock/tags/render paths move
under infra/deploy/, image-path defaults become emi/thermograph/backend|
frontend across compose, stack, and the tag-prune. docker-compose.yml pins
name: thermograph -- without it compose run from .../infra derives project
"infra" and recreates the whole stack beside the running one;
deploy-dev.sh pins COMPOSE_PROJECT_NAME=thermograph-dev (env wins over the
file key) to keep LAN dev's separate project.
One root workflow set replaces the four repos' copies (deleted -- root-only
is where Forgejo reads them, and dead copies are a trap): per-domain
build-push with explicit image paths (emi/thermograph/backend|frontend; the
old github.repository-derived path collides in a monorepo), path-filtered
per-domain beta/prod/dev deploys, a domain-input reusable build check, a
single always-reporting PR gate (path-filtered required checks deadlock
auto-merge), a new infra-sync pipeline (host checkout + secrets render on
infra/** pushes), and ports of secrets-guard / ops-cron /
observability-validate to monorepo paths.
main received the perf hardening line (#5, #8) and its own landing of the
Discord gateway bot (#7) while dev carried a parallel port of the same bot
(#3). The two implementations are near-identical ports of the same archived
source; main's includes one extra hardening (grading calls moved off the
gateway event loop via asyncio.to_thread) and broader tests, so bot files
resolve wholesale to main's side. dev keeps its test-runner/smoke tooling and
the run-tests-in-image CI, which main lacks.
# Conflicts:
# notifications/discord_bot.py
# tests/notifications/test_discord_bot.py
# web/app.py
