ops/dbq.sh: read-only Postgres queries across all environments (#18)

infra/ops/ICEBERG-HANDOFF.md

@@ -0,0 +1,133 @@
+# Iceberg: implementing the agent-queryable service
+
+Handoff spec for whoever builds the Iceberg layer. The Postgres equivalent is
+already shipped (`infra/ops/dbq.sh`) — **mirror it**. This document is the
+contract, the environment facts you'll trip over, and how the result gets
+verified.
+
+## Definition of done
+
+`infra/ops/iceberg.sh` exists and behaves like `dbq.sh`:
+
+```sh
+infra/ops/iceberg.sh prod -c "select count(*) from <namespace>.<table>"
+infra/ops/iceberg.sh dev  -c "show tables"
+echo "select 1" | infra/ops/iceberg.sh beta -f -
+```
+
+An agent (or operator) can query Iceberg in any environment, read-only, with no
+new network exposure and no interactive steps.
+
+## The contract (non-negotiable)
+
+1. **Same interface shape as `dbq.sh`** — `iceberg.sh <dev|beta|prod> [flags]
+   "<sql>"`. Extra flags pass through to the engine; **stdin is forwarded** so
+   `-f -` works. Non-interactive, deterministic, safe to call from CI.
+2. **Read-only enforced by the system, not by convention.** A dedicated
+   read-only identity — not an admin/superuser credential. You must be able to
+   *demonstrate* a refused write (see Acceptance).
+3. **No new network endpoints.** Run the engine where the data is already
+   reachable (containerised), rather than publishing ports or opening firewall
+   rules. See the prod constraint below — this is not negotiable, it's physics.
+4. **Env-keyed dispatch, names resolved at call time.** Prod is Docker Swarm;
+   task container names change on **every redeploy**. Resolve via
+   `docker ps --filter name=…`; never hardcode a container name.
+5. **No secrets in the repo** or in workflow files.
+
+## Environment facts you need
+
+- Monorepo `emi/thermograph` (domain dirs: `backend/ frontend/ infra/
+  observability/`). Ops tooling lives in `infra/ops/`.
+- Environments: **dev** = LAN compose stack on the dev machine; **beta** =
+  `75.119.132.91`; **prod** = `169.58.46.181`. SSH as `agent` with
+  `~/.ssh/thermograph_agent_ed25519` (passwordless sudo on both boxes).
+- **Prod runs Docker Swarm.** Its app database sits on the overlay network
+  `thermograph_internal` (`10.0.2.0/24`), which **the prod host itself cannot
+  route to**. Consequence, learned the hard way: `ssh -L` tunnelling works for
+  beta but is *impossible* for prod. Any design that depends on a tunnel or a
+  published port will fail on prod — that's why `dbq.sh` execs into the
+  container instead. Assume the same for anything you deploy there.
+- The overlay is `attachable=true`, and the **dev machine is a Swarm worker** in
+  the prod cluster (NodeAddr `10.10.0.3`) — so `docker run --network
+  thermograph_internal …` from the dev box is a plausible route to prod-side
+  services. Untested; verify before relying on it.
+- WireGuard mesh: prod `10.10.0.1`, beta `10.10.0.2`, dev `10.10.0.3`.
+- `rclone` and `age` are already installed on prod and beta.
+- Reference implementation to copy: **`infra/ops/dbq.sh`** + `infra/ops/README.md`.
+
+## Storage — almost certainly your warehouse
+
+Contabo Object Storage (S3-compatible), already provisioned and in use:
+
+- Endpoint `https://eu2.contabostorage.com` · bucket `era5-thermograph` ·
+  region `default` · ~2 TB · **private** (keep it that way).
+- ⚠️ **Contabo requires PATH-STYLE addressing.** Set
+  `force_path_style=true` / `s3.path-style-access=true` (whatever your engine's
+  FileIO calls it) and `region=default`. Virtual-host-style addressing **fails**.
+  This is validated, not theoretical.
+- ⚠️ **The `backups/` prefix is in use** by the nightly backup jobs (prod DB +
+  Forgejo). Put Iceberg data under its own prefix (e.g. `iceberg/` or
+  `warehouse/`). Do not write outside your prefix.
+- Credentials already exist — **reuse them, don't mint new ones silently**:
+  - SOPS vault: `infra/deploy/secrets/{prod,beta}.yaml` as `THERMOGRAPH_S3_*`
+    (rendered to `/etc/thermograph.env` at deploy by
+    `infra/deploy/render-secrets.sh`).
+  - Forgejo Actions secrets: `S3_ENDPOINT`, `S3_BUCKET`, `S3_ACCESS_KEY`,
+    `S3_SECRET_KEY`.
+  - If read-only S3 keys are needed for the query identity, ask the operator to
+    mint a scoped keypair rather than reusing the read-write one.
+
+## Secrets handling
+
+- Host-side: SOPS + age. Recipient `age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2`;
+  the private key is on each host at `/etc/thermograph/age.key` and on the
+  operator's machine at `~/.config/sops/age/keys.txt`.
+- CI-side: Forgejo Actions secrets.
+- ⚠️ **beta gotcha:** `/etc/thermograph.env` on beta is `agent:agent 0640` — the
+  `deploy` user **cannot read it**. If your service or job runs as `deploy` on
+  beta, pull credentials from Actions secrets instead, or change the ownership
+  deliberately and say so.
+
+## Decisions you must make — and report back
+
+1. **Catalog**: REST (Lakekeeper / Nessie / Polaris), JDBC-on-Postgres, Hive, or
+   filesystem/hadoop — and where it runs per environment.
+2. **Engine**: DuckDB + iceberg extension, Trino, Spark, or pyiceberg. Prefer the
+   lightest that satisfies the contract; it must run containerised and headless.
+3. **Warehouse location**: bucket + prefix, and whether environments are isolated
+   by separate prefixes, namespaces, or buckets.
+4. **Read-only mechanism**: scoped S3 keys? catalog RBAC? engine restricted mode?
+   State which, and how a write is refused.
+5. **Where the engine runs** for each env (local container on dev, on the box for
+   beta/prod, or attached to the overlay) — consistent with constraint #3.
+
+## Acceptance criteria
+
+The work is accepted when all of these are demonstrated with pasted output:
+
+- [ ] `infra/ops/iceberg.sh <env> -c "<select>"` returns rows for **each**
+      environment that has data (explicitly state any env that doesn't yet).
+- [ ] **A write/DDL attempt is refused by the system**, with the error pasted.
+      The Postgres bar to match:
+      `create table` on prod → `ERROR:  permission denied for schema public`.
+- [ ] **No new exposure**: `sudo ss -ltnp` on prod and beta shows no new
+      listening port; no new ufw rules; no Swarm `EndpointSpec` publish added.
+      State this explicitly.
+- [ ] Non-interactive: extra flags pass through, stdin (`-f -`) works.
+- [ ] Prod container/task resolution is dynamic (survives a redeploy).
+- [ ] `infra/ops/README.md` updated with an Iceberg section: usage examples and
+      the *why* (same style as the Postgres rationale).
+
+## Anti-goals
+
+- ❌ Publishing catalog/engine/DB ports publicly, or adding firewall holes.
+- ❌ Any design requiring a persistent SSH tunnel (**prod cannot be tunnelled**).
+- ❌ Querying with an admin/superuser/read-write credential.
+- ❌ Secrets committed to the repo or pasted into workflow files.
+- ❌ Writing outside your own S3 prefix (`backups/` is live and load-bearing).
+
+## Questions to route back to the operator
+
+Anything that would require: minting new cloud credentials, opening a port,
+changing firewall/Swarm configuration, or writing to a shared prefix. Don't
+improvise on those — ask.

infra/ops/README.md

@@ -0,0 +1,65 @@
+# ops/ — operator + agent query tooling
+
+Read-only query access to Thermograph data, uniform across environments.
+
+## `dbq.sh` — Postgres, any environment
+
+```sh
+infra/ops/dbq.sh dev  "select count(*) from climate_history"
+infra/ops/dbq.sh prod -tA -c "select max(date) from climate_history"
+infra/ops/dbq.sh beta -c '\dt'
+echo "select 1" | infra/ops/dbq.sh prod -f -
+```
+
+Environments: `dev` (LAN compose stack on the dev machine), `beta`
+(75.119.132.91), `prod` (169.58.46.181). Extra args pass straight through to
+`psql` (`-tA`, `-x`, `--csv`, `-f`, …); stdin is forwarded.
+
+### Why it execs into the container instead of using a connection string
+
+None of the databases are exposed over TCP — each listens only on its private
+docker network. Prod's is a Swarm **overlay** (`10.0.2.0/24`) that the host
+cannot route to, so `ssh -L` works for beta but is *impossible* for prod;
+publishing 5432 would mean new ufw rules plus a Swarm endpoint change on
+production. Running `psql` inside the db container works identically in all
+three environments with no ports, no tunnels, and no infra changes.
+
+The prod container name is a Swarm task name that changes on every redeploy, so
+it is resolved at call time via `docker ps --filter name=…`, never hardcoded.
+
+### Safety
+
+Queries connect as **`thermograph_ro`** — a `NOSUPERUSER` role granted only
+`pg_read_all_data`. Read-only is enforced by Postgres, not by convention:
+
+```
+$ infra/ops/dbq.sh prod -c "create table t(i int)"
+ERROR:  permission denied for schema public
+```
+
+The app's own `thermograph` role is a superuser and is deliberately **not** used
+by this tool. If the role is ever missing (fresh database), recreate it with:
+
+```sql
+CREATE ROLE thermograph_ro LOGIN;
+GRANT CONNECT ON DATABASE thermograph TO thermograph_ro;
+GRANT pg_read_all_data TO thermograph_ro;
+```
+
+## Iceberg (planned)
+
+**Implementation spec: [`ICEBERG-HANDOFF.md`](./ICEBERG-HANDOFF.md)** — the
+contract, environment constraints, acceptance criteria and anti-goals for the
+Iceberg worker.
+
+Iceberg lands as a sibling in this directory following the same three
+conventions, so the query surface stays uniform:
+
+1. **Env-keyed dispatch** — `iceberg.sh <dev|beta|prod> "<sql>"`, same argument
+   shape as `dbq.sh`.
+2. **Run the engine where the data is reachable** — the catalog/warehouse
+   dictates it (a Trino/DuckDB/pyiceberg invocation, containerised the same way),
+   rather than exposing new network endpoints.
+3. **A dedicated read-only identity** — the Iceberg equivalent of
+   `thermograph_ro`, so the same "read-only enforced by the system, not by
+   convention" property holds.

infra/ops/dbq.sh

@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+# dbq -- run READ-ONLY SQL against any Thermograph Postgres (LAN dev / beta / prod).
+#
+#   infra/ops/dbq.sh dev  "select count(*) from climate_history"
+#   infra/ops/dbq.sh prod -tA "select max(date) from climate_history"
+#   infra/ops/dbq.sh beta -c '\dt'
+#   echo "select 1" | infra/ops/dbq.sh prod -f -
+#
+# Why exec-into-the-container instead of a connection string:
+# none of the databases are exposed over TCP. Each listens only on its private
+# docker network -- and prod's is a Swarm *overlay* (10.0.2.0/24) that the host
+# itself cannot route to, so `ssh -L` works for beta but is impossible for prod.
+# Publishing 5432 would mean new firewall + Swarm endpoint changes on production.
+# Running psql *inside* the db container works identically in all three
+# environments with no ports, no tunnels and no infra changes -- locally for LAN
+# dev, over SSH for beta/prod.
+#
+# Safety: always connects as `thermograph_ro`, a NOSUPERUSER role granted only
+# pg_read_all_data. Read-only is enforced by Postgres itself, not by convention,
+# so a stray INSERT/DDL fails with "permission denied" even against prod. (The
+# app's own `thermograph` role is a superuser -- deliberately not used here.)
+#
+# Any extra arguments are passed straight through to psql, so -tA, -c, -f, -x,
+# --csv etc. all work. Stdin is forwarded, so `-f -` reads piped SQL.
+set -euo pipefail
+
+KEYFILE="${THERMOGRAPH_AGENT_KEY:-$HOME/.ssh/thermograph_agent_ed25519}"
+DB_USER="${THERMOGRAPH_DB_QUERY_USER:-thermograph_ro}"
+DB_NAME="${THERMOGRAPH_DB_NAME:-thermograph}"
+
+usage() {
+  sed -n '2,9p' "$0" | sed 's/^# \{0,1\}//' >&2
+  echo "environments: dev | beta | prod" >&2
+  exit 2
+}
+
+env_name="${1:-}"
+[ -n "$env_name" ] || usage
+shift
+
+case "$env_name" in
+  dev)  filter=thermograph-dev-db ; ssh_target= ;;
+  beta) filter=thermograph-db-1   ; ssh_target=agent@75.119.132.91 ;;
+  prod) filter=thermograph_db     ; ssh_target=agent@169.58.46.181 ;;
+  -h|--help) usage ;;
+  *) echo "dbq: unknown environment '$env_name' (want: dev|beta|prod)" >&2; exit 2 ;;
+esac
+
+[ $# -gt 0 ] || usage
+
+# Quote the psql arguments so they survive the remote shell intact.
+psql_args=$(printf '%q ' "$@")
+
+# The container name is resolved on the target host at call time: prod's Swarm
+# task name changes on every redeploy, so it can never be hardcoded.
+remote=$(cat <<REMOTE
+cid=\$(docker ps -qf name=${filter} | head -1)
+[ -n "\$cid" ] || { echo "dbq: no running db container matching '${filter}'" >&2; exit 1; }
+exec docker exec -i "\$cid" psql -U ${DB_USER} -d ${DB_NAME} -v ON_ERROR_STOP=1 ${psql_args}
+REMOTE
+)
+
+if [ -z "$ssh_target" ]; then
+  exec bash -c "$remote"
+else
+  exec ssh -i "$KEYFILE" -o StrictHostKeyChecking=no -o ConnectTimeout=15 \
+    "$ssh_target" "$remote"
+fi