thermograph/infra/ops/README.md

# ops/ — operator + agent query tooling

Read-only query access to Thermograph data, uniform across environments.

## `dbq.sh` — Postgres, any environment

```sh
infra/ops/dbq.sh dev  "select count(*) from climate_history"
infra/ops/dbq.sh prod -tA -c "select max(date) from climate_history"
infra/ops/dbq.sh beta -c '\dt'
echo "select 1" | infra/ops/dbq.sh prod -f -
```

Environments: `dev` (own compose stack + own Postgres container, on vps1,
`75.119.132.91`), `beta` and `prod` (both on vps2, `169.58.46.181` — separate
roles/databases on the ONE shared TimescaleDB instance there). Host, SSH
target and container filter all come from `deploy/env-topology.sh` now, never
a hardcoded table — `dbq.sh` SSHes to vps1 for `dev` and to vps2 for
`beta`/`prod`, and beta (which has no db container of its own) resolves
prod's shared `thermograph_db` task. Extra args pass straight through to
`psql` (`-tA`, `-x`, `--csv`, `-f`, …); stdin is forwarded.

**Known gap:** beta has no read-only role of its own yet — `deploy/db/provision-env-db.sh`
only provisions the environment's *owning* role (`thermograph_beta`), so a
`dbq.sh beta` query connects as `thermograph_beta` (which can write) rather
than a read-only role, unlike `dev`/`prod`. See `dbq.sh`'s own header for the
tracked follow-up (a `thermograph_beta_ro` role).

### Why it execs into the container instead of using a connection string

None of the databases are exposed over TCP — each listens only on its private
docker network. Prod's and beta's is a Swarm **overlay** (`thermograph_internal`)
that the vps2 host itself cannot route to, so `ssh -L` is impossible for
either of them — treat beta exactly like prod here, now that it's a Swarm
service too, not the compose-network exception it used to be. Publishing 5432
would mean new ufw rules plus a Swarm endpoint change on a live environment.
Running `psql` inside the db container works identically in all three
environments with no ports, no tunnels, and no infra changes.

The prod container name is a Swarm task name that changes on every redeploy, so
it is resolved at call time via `docker ps --filter name=…`, never hardcoded.

### Safety

`dev` and `prod` connect as **`thermograph_ro`** — a `NOSUPERUSER` role granted
only `pg_read_all_data`. Read-only is enforced by Postgres, not by convention:

```
$ infra/ops/dbq.sh prod -c "create table t(i int)"
ERROR:  permission denied for schema public
```

**`beta` is the exception, today:** `deploy/db/provision-env-db.sh` only ever
provisions the environment's *owning* role (`thermograph_beta` —
`NOSUPERUSER`/`NOCREATEDB`/`NOCREATEROLE`, but it owns its own database and
schema) and revokes `CONNECT` from `PUBLIC` on `thermograph_beta`, so
`thermograph_ro` was never granted `CONNECT` there. A `dbq.sh beta` query
therefore connects as `thermograph_beta` until a beta-scoped read-only role
exists (tracked in `dbq.sh`'s own header as a TODO) — it can write to beta's
database, though never to prod's.

The app's own `thermograph` role is a superuser and is deliberately **not** used
by this tool. If `thermograph_ro` is ever missing (fresh instance), recreate it with:

```sql
CREATE ROLE thermograph_ro LOGIN;
GRANT CONNECT ON DATABASE thermograph TO thermograph_ro;
GRANT pg_read_all_data TO thermograph_ro;
```

## `iceberg.sh` — the Iceberg lake, any environment

```sh
infra/ops/iceberg.sh dev  -c "show tables"
infra/ops/iceberg.sh prod -c "select count(*) from era5_daily"
infra/ops/iceberg.sh beta -json -c "select * from era5_daily limit 3"
echo "select 1" | infra/ops/iceberg.sh prod -f -
```

Same environments and argument shape as `dbq.sh` (spec:
[`ICEBERG-HANDOFF.md`](./ICEBERG-HANDOFF.md)). Extra args pass straight
through to the duckdb CLI (`-json`, `-csv`, `-line`, `-c`, `-f`, …); with
`-f -`, stdin is forwarded and read as piped SQL; a single bare argument is
treated as `-c` SQL.

### Why an ephemeral DuckDB container instead of a query service

The lake is **one shared warehouse** in Contabo object storage
(`s3://era5-thermograph/iceberg`), reachable from every environment with the
same credentials — so unlike Postgres there is no per-env database to reach,
and the environments differ only in *where the engine runs*: a throwaway
`docker run` of a small DuckDB image (duckdb CLI + `httpfs`/`iceberg`
extensions pre-installed) on the target box — vps1 for `dev`, vps2 for
`beta`/`prod`, resolved from `deploy/env-topology.sh` rather than a hardcoded
table (beta and prod are two Swarm stacks on the same vps2 box now). The
image is built on the host the first time it's needed from a Dockerfile
embedded in the script; its tag is a hash of that Dockerfile, so editing the
script rolls every host forward on the next call.

A container per query means no daemon, no listening port, no tunnel (beta's
and prod's Swarm overlay cannot be tunnelled), no firewall or Swarm changes —
`sudo ss -ltnp` is identical before and after a query. It also never execs
into an app container and resolves no container names, so a redeploy can't
break it.

There is no catalog service to run or expose: an Iceberg table's current
state is fully described by the newest metadata JSON under
`<warehouse>/<table>/metadata/`. `iceberg.sh` resolves that at call time
(numeric metadata version, newest wins) and exposes each table directory as a
view of the same name — which is also how it keeps reading fresh snapshots of
a table that is still being loaded.

Credentials are resolved at call time and never stored in the repo: the
target host's `/etc/thermograph.env` (`THERMOGRAPH_LAKE_S3_*`) wins if
present; otherwise the calling machine decrypts the SOPS vault
(`infra/deploy/secrets/prod.yaml`) and hands the two values to the remote
shell over stdin — never argv, so never visible in `ps`, and never written to
a file on the target.

### Safety

Every session locks itself down *before* user SQL runs:

```sql
SET allowed_directories=['s3://era5-thermograph/iceberg/', 's3://era5-thermograph/era5/'];
SET enable_external_access=false;
SET lock_configuration=true;
```

Both prefixes are required for reads: `era5_daily` was built with pyiceberg
`add_files` over the existing hive parquet, so its metadata lives under
`iceberg/` while its data files stay in place under `era5/daily/`. DuckDB
itself then refuses everything else — the live `backups/` prefix, all local
files — and the lockdown cannot be SET away by query text:

```
$ infra/ops/iceberg.sh prod -c "COPY (SELECT 1) TO 's3://era5-thermograph/backups/x.csv'"
Permission Error: Cannot access file "s3://era5-thermograph/backups/x.csv" - file system operations are disabled by configuration
$ infra/ops/iceberg.sh dev -c "SET enable_external_access=true"
Invalid Input Error: Cannot change configuration option "enable_external_access" - the configuration has been locked
```

**Known gap:** the only provisioned keypair for the bucket is read-write, so
a raw `COPY TO` targeting a path *inside* the two allowed prefixes is not
refused — the DuckDB iceberg extension cannot write Iceberg tables, but it
could still clobber raw objects under `iceberg/` or `era5/`. Path-based
restriction cannot both allow reading a prefix and forbid writing it; closing
the gap needs a scoped read-only keypair (the object-storage equivalent of
`thermograph_ro`). When the operator mints one, render it as
`THERMOGRAPH_LAKE_S3_*` on the hosts or swap it into the vault —
`iceberg.sh` needs no code change.