# ops/ — operator + agent query tooling Read-only query access to Thermograph data, uniform across environments. ## `dbq.sh` — Postgres, any environment ```sh infra/ops/dbq.sh dev "select count(*) from climate_history" infra/ops/dbq.sh prod -tA -c "select max(date) from climate_history" infra/ops/dbq.sh beta -c '\dt' echo "select 1" | infra/ops/dbq.sh prod -f - ``` Environments: `dev` (own compose stack + own Postgres container, on vps1, `75.119.132.91`), `beta` and `prod` (both on vps2, `169.58.46.181` — separate roles/databases on the ONE shared TimescaleDB instance there). Host, SSH target and container filter all come from `deploy/env-topology.sh` now, never a hardcoded table — `dbq.sh` SSHes to vps1 for `dev` and to vps2 for `beta`/`prod`, and beta (which has no db container of its own) resolves prod's shared `thermograph_db` task. Extra args pass straight through to `psql` (`-tA`, `-x`, `--csv`, `-f`, …); stdin is forwarded. **Known gap:** beta has no read-only role of its own yet — `deploy/db/provision-env-db.sh` only provisions the environment's *owning* role (`thermograph_beta`), so a `dbq.sh beta` query connects as `thermograph_beta` (which can write) rather than a read-only role, unlike `dev`/`prod`. See `dbq.sh`'s own header for the tracked follow-up (a `thermograph_beta_ro` role). ### Why it execs into the container instead of using a connection string None of the databases are exposed over TCP — each listens only on its private docker network. Prod's and beta's is a Swarm **overlay** (`thermograph_internal`) that the vps2 host itself cannot route to, so `ssh -L` is impossible for either of them — treat beta exactly like prod here, now that it's a Swarm service too, not the compose-network exception it used to be. Publishing 5432 would mean new ufw rules plus a Swarm endpoint change on a live environment. Running `psql` inside the db container works identically in all three environments with no ports, no tunnels, and no infra changes. The prod container name is a Swarm task name that changes on every redeploy, so it is resolved at call time via `docker ps --filter name=…`, never hardcoded. ### Safety `dev` and `prod` connect as **`thermograph_ro`** — a `NOSUPERUSER` role granted only `pg_read_all_data`. Read-only is enforced by Postgres, not by convention: ``` $ infra/ops/dbq.sh prod -c "create table t(i int)" ERROR: permission denied for schema public ``` **`beta` is the exception, today:** `deploy/db/provision-env-db.sh` only ever provisions the environment's *owning* role (`thermograph_beta` — `NOSUPERUSER`/`NOCREATEDB`/`NOCREATEROLE`, but it owns its own database and schema) and revokes `CONNECT` from `PUBLIC` on `thermograph_beta`, so `thermograph_ro` was never granted `CONNECT` there. A `dbq.sh beta` query therefore connects as `thermograph_beta` until a beta-scoped read-only role exists (tracked in `dbq.sh`'s own header as a TODO) — it can write to beta's database, though never to prod's. The app's own `thermograph` role is a superuser and is deliberately **not** used by this tool. If `thermograph_ro` is ever missing (fresh instance), recreate it with: ```sql CREATE ROLE thermograph_ro LOGIN; GRANT CONNECT ON DATABASE thermograph TO thermograph_ro; GRANT pg_read_all_data TO thermograph_ro; ``` ## `iceberg.sh` — the Iceberg lake, any environment ```sh infra/ops/iceberg.sh dev -c "show tables" infra/ops/iceberg.sh prod -c "select count(*) from era5_daily" infra/ops/iceberg.sh beta -json -c "select * from era5_daily limit 3" echo "select 1" | infra/ops/iceberg.sh prod -f - ``` Same environments and argument shape as `dbq.sh` (spec: [`ICEBERG-HANDOFF.md`](./ICEBERG-HANDOFF.md)). Extra args pass straight through to the duckdb CLI (`-json`, `-csv`, `-line`, `-c`, `-f`, …); with `-f -`, stdin is forwarded and read as piped SQL; a single bare argument is treated as `-c` SQL. ### Why an ephemeral DuckDB container instead of a query service The lake is **one shared warehouse** in Contabo object storage (`s3://era5-thermograph/iceberg`), reachable from every environment with the same credentials — so unlike Postgres there is no per-env database to reach, and the environments differ only in *where the engine runs*: a throwaway `docker run` of a small DuckDB image (duckdb CLI + `httpfs`/`iceberg` extensions pre-installed) on the target box — vps1 for `dev`, vps2 for `beta`/`prod`, resolved from `deploy/env-topology.sh` rather than a hardcoded table (beta and prod are two Swarm stacks on the same vps2 box now). The image is built on the host the first time it's needed from a Dockerfile embedded in the script; its tag is a hash of that Dockerfile, so editing the script rolls every host forward on the next call. A container per query means no daemon, no listening port, no tunnel (beta's and prod's Swarm overlay cannot be tunnelled), no firewall or Swarm changes — `sudo ss -ltnp` is identical before and after a query. It also never execs into an app container and resolves no container names, so a redeploy can't break it. There is no catalog service to run or expose: an Iceberg table's current state is fully described by the newest metadata JSON under `//metadata/`. `iceberg.sh` resolves that at call time (numeric metadata version, newest wins) and exposes each table directory as a view of the same name — which is also how it keeps reading fresh snapshots of a table that is still being loaded. Credentials are resolved at call time and never stored in the repo: the target host's `/etc/thermograph.env` (`THERMOGRAPH_LAKE_S3_*`) wins if present; otherwise the calling machine decrypts the SOPS vault (`infra/deploy/secrets/prod.yaml`) and hands the two values to the remote shell over stdin — never argv, so never visible in `ps`, and never written to a file on the target. ### Safety Every session locks itself down *before* user SQL runs: ```sql SET allowed_directories=['s3://era5-thermograph/iceberg/', 's3://era5-thermograph/era5/']; SET enable_external_access=false; SET lock_configuration=true; ``` Both prefixes are required for reads: `era5_daily` was built with pyiceberg `add_files` over the existing hive parquet, so its metadata lives under `iceberg/` while its data files stay in place under `era5/daily/`. DuckDB itself then refuses everything else — the live `backups/` prefix, all local files — and the lockdown cannot be SET away by query text: ``` $ infra/ops/iceberg.sh prod -c "COPY (SELECT 1) TO 's3://era5-thermograph/backups/x.csv'" Permission Error: Cannot access file "s3://era5-thermograph/backups/x.csv" - file system operations are disabled by configuration $ infra/ops/iceberg.sh dev -c "SET enable_external_access=true" Invalid Input Error: Cannot change configuration option "enable_external_access" - the configuration has been locked ``` **Known gap:** the only provisioned keypair for the bucket is read-write, so a raw `COPY TO` targeting a path *inside* the two allowed prefixes is not refused — the DuckDB iceberg extension cannot write Iceberg tables, but it could still clobber raw objects under `iceberg/` or `era5/`. Path-based restriction cannot both allow reading a prefix and forbid writing it; closing the gap needs a scoped read-only keypair (the object-storage equivalent of `thermograph_ro`). When the operator mints one, render it as `THERMOGRAPH_LAKE_S3_*` on the hosts or swap it into the vault — `iceberg.sh` needs no code change.