thermograph/infra/ops/README.md

ops/ — operator + agent query tooling

Read-only query access to Thermograph data, uniform across environments.

dbq.sh — Postgres, any environment

infra/ops/dbq.sh dev  "select count(*) from climate_history"
infra/ops/dbq.sh prod -tA -c "select max(date) from climate_history"
infra/ops/dbq.sh beta -c '\dt'
echo "select 1" | infra/ops/dbq.sh prod -f -

Environments: dev (own compose stack + own Postgres container, on vps1, 75.119.132.91), beta and prod (both on vps2, 169.58.46.181 — separate roles/databases on the ONE shared TimescaleDB instance there). Host, SSH target and container filter all come from deploy/env-topology.sh now, never a hardcoded table — dbq.sh SSHes to vps1 for dev and to vps2 for beta/prod, and beta (which has no db container of its own) resolves prod's shared thermograph_db task. Extra args pass straight through to psql (-tA, -x, --csv, -f, …); stdin is forwarded.

Known gap: beta has no read-only role of its own yet — deploy/db/provision-env-db.sh only provisions the environment's owning role (thermograph_beta), so a dbq.sh beta query connects as thermograph_beta (which can write) rather than a read-only role, unlike dev/prod. See dbq.sh's own header for the tracked follow-up (a thermograph_beta_ro role).

Why it execs into the container instead of using a connection string

None of the databases are exposed over TCP — each listens only on its private docker network. Prod's and beta's is a Swarm overlay (thermograph_internal) that the vps2 host itself cannot route to, so ssh -L is impossible for either of them — treat beta exactly like prod here, now that it's a Swarm service too, not the compose-network exception it used to be. Publishing 5432 would mean new ufw rules plus a Swarm endpoint change on a live environment. Running psql inside the db container works identically in all three environments with no ports, no tunnels, and no infra changes.

The prod container name is a Swarm task name that changes on every redeploy, so it is resolved at call time via docker ps --filter name=…, never hardcoded.

Safety

dev and prod connect as thermograph_ro — a NOSUPERUSER role granted only pg_read_all_data. Read-only is enforced by Postgres, not by convention:

$ infra/ops/dbq.sh prod -c "create table t(i int)"
ERROR:  permission denied for schema public

beta is the exception, today: deploy/db/provision-env-db.sh only ever provisions the environment's owning role (thermograph_beta — NOSUPERUSER/NOCREATEDB/NOCREATEROLE, but it owns its own database and schema) and revokes CONNECT from PUBLIC on thermograph_beta, so thermograph_ro was never granted CONNECT there. A dbq.sh beta query therefore connects as thermograph_beta until a beta-scoped read-only role exists (tracked in dbq.sh's own header as a TODO) — it can write to beta's database, though never to prod's.

The app's own thermograph role is a superuser and is deliberately not used by this tool. If thermograph_ro is ever missing (fresh instance), recreate it with:

CREATE ROLE thermograph_ro LOGIN;
GRANT CONNECT ON DATABASE thermograph TO thermograph_ro;
GRANT pg_read_all_data TO thermograph_ro;

iceberg.sh — the Iceberg lake, any environment

infra/ops/iceberg.sh dev  -c "show tables"
infra/ops/iceberg.sh prod -c "select count(*) from era5_daily"
infra/ops/iceberg.sh beta -json -c "select * from era5_daily limit 3"
echo "select 1" | infra/ops/iceberg.sh prod -f -

Same environments and argument shape as dbq.sh (spec: ICEBERG-HANDOFF.md). Extra args pass straight through to the duckdb CLI (-json, -csv, -line, -c, -f, …); with -f -, stdin is forwarded and read as piped SQL; a single bare argument is treated as -c SQL.

Why an ephemeral DuckDB container instead of a query service

The lake is one shared warehouse in Contabo object storage (s3://era5-thermograph/iceberg), reachable from every environment with the same credentials — so unlike Postgres there is no per-env database to reach, and the environments differ only in where the engine runs: a throwaway docker run of a small DuckDB image (duckdb CLI + httpfs/iceberg extensions pre-installed) on the target box — vps1 for dev, vps2 for beta/prod, resolved from deploy/env-topology.sh rather than a hardcoded table (beta and prod are two Swarm stacks on the same vps2 box now). The image is built on the host the first time it's needed from a Dockerfile embedded in the script; its tag is a hash of that Dockerfile, so editing the script rolls every host forward on the next call.

A container per query means no daemon, no listening port, no tunnel (beta's and prod's Swarm overlay cannot be tunnelled), no firewall or Swarm changes — sudo ss -ltnp is identical before and after a query. It also never execs into an app container and resolves no container names, so a redeploy can't break it.

There is no catalog service to run or expose: an Iceberg table's current state is fully described by the newest metadata JSON under <warehouse>/<table>/metadata/. iceberg.sh resolves that at call time (numeric metadata version, newest wins) and exposes each table directory as a view of the same name — which is also how it keeps reading fresh snapshots of a table that is still being loaded.

Credentials are resolved at call time and never stored in the repo: the target host's /etc/thermograph.env (THERMOGRAPH_LAKE_S3_*) wins if present; otherwise the calling machine decrypts the SOPS vault (infra/deploy/secrets/prod.yaml) and hands the two values to the remote shell over stdin — never argv, so never visible in ps, and never written to a file on the target.

Safety

Every session locks itself down before user SQL runs:

SET allowed_directories=['s3://era5-thermograph/iceberg/', 's3://era5-thermograph/era5/'];
SET enable_external_access=false;
SET lock_configuration=true;

Both prefixes are required for reads: era5_daily was built with pyiceberg add_files over the existing hive parquet, so its metadata lives under iceberg/ while its data files stay in place under era5/daily/. DuckDB itself then refuses everything else — the live backups/ prefix, all local files — and the lockdown cannot be SET away by query text:

$ infra/ops/iceberg.sh prod -c "COPY (SELECT 1) TO 's3://era5-thermograph/backups/x.csv'"
Permission Error: Cannot access file "s3://era5-thermograph/backups/x.csv" - file system operations are disabled by configuration
$ infra/ops/iceberg.sh dev -c "SET enable_external_access=true"
Invalid Input Error: Cannot change configuration option "enable_external_access" - the configuration has been locked

Known gap: the only provisioned keypair for the bucket is read-write, so a raw COPY TO targeting a path inside the two allowed prefixes is not refused — the DuckDB iceberg extension cannot write Iceberg tables, but it could still clobber raw objects under iceberg/ or era5/. Path-based restriction cannot both allow reading a prefix and forbid writing it; closing the gap needs a scoped read-only keypair (the object-storage equivalent of thermograph_ro). When the operator mints one, render it as THERMOGRAPH_LAKE_S3_* on the hosts or swap it into the vault — iceberg.sh needs no code change.