Add ICEBERG-HANDOFF.md: implementation spec for the Iceberg query service

Handoff for the Iceberg worker: the contract iceberg.sh must satisfy (same
interface shape as dbq.sh, read-only enforced by the system, no new network
endpoints, call-time container resolution), the environment constraints that
will otherwise bite -- prod's Swarm overlay is unroutable from its own host so
tunnels are impossible, Contabo object storage requires path-style addressing,
the backups/ prefix is live, and beta's /etc/thermograph.env is unreadable by
the deploy user -- plus the decisions to report back, acceptance criteria with
a demonstrated refused write, and anti-goals.

infra/ops/ICEBERG-HANDOFF.md

@@ -0,0 +1,133 @@
+# Iceberg: implementing the agent-queryable service
+
+Handoff spec for whoever builds the Iceberg layer. The Postgres equivalent is
+already shipped (`infra/ops/dbq.sh`) — **mirror it**. This document is the
+contract, the environment facts you'll trip over, and how the result gets
+verified.
+
+## Definition of done
+
+`infra/ops/iceberg.sh` exists and behaves like `dbq.sh`:
+
+```sh
+infra/ops/iceberg.sh prod -c "select count(*) from <namespace>.<table>"
+infra/ops/iceberg.sh dev  -c "show tables"
+echo "select 1" | infra/ops/iceberg.sh beta -f -
+```
+
+An agent (or operator) can query Iceberg in any environment, read-only, with no
+new network exposure and no interactive steps.
+
+## The contract (non-negotiable)
+
+1. **Same interface shape as `dbq.sh`** — `iceberg.sh <dev|beta|prod> [flags]
+   "<sql>"`. Extra flags pass through to the engine; **stdin is forwarded** so
+   `-f -` works. Non-interactive, deterministic, safe to call from CI.
+2. **Read-only enforced by the system, not by convention.** A dedicated
+   read-only identity — not an admin/superuser credential. You must be able to
+   *demonstrate* a refused write (see Acceptance).
+3. **No new network endpoints.** Run the engine where the data is already
+   reachable (containerised), rather than publishing ports or opening firewall
+   rules. See the prod constraint below — this is not negotiable, it's physics.
+4. **Env-keyed dispatch, names resolved at call time.** Prod is Docker Swarm;
+   task container names change on **every redeploy**. Resolve via
+   `docker ps --filter name=…`; never hardcode a container name.
+5. **No secrets in the repo** or in workflow files.
+
+## Environment facts you need
+
+- Monorepo `emi/thermograph` (domain dirs: `backend/ frontend/ infra/
+  observability/`). Ops tooling lives in `infra/ops/`.
+- Environments: **dev** = LAN compose stack on the dev machine; **beta** =
+  `75.119.132.91`; **prod** = `169.58.46.181`. SSH as `agent` with
+  `~/.ssh/thermograph_agent_ed25519` (passwordless sudo on both boxes).
+- **Prod runs Docker Swarm.** Its app database sits on the overlay network
+  `thermograph_internal` (`10.0.2.0/24`), which **the prod host itself cannot
+  route to**. Consequence, learned the hard way: `ssh -L` tunnelling works for
+  beta but is *impossible* for prod. Any design that depends on a tunnel or a
+  published port will fail on prod — that's why `dbq.sh` execs into the
+  container instead. Assume the same for anything you deploy there.
+- The overlay is `attachable=true`, and the **dev machine is a Swarm worker** in
+  the prod cluster (NodeAddr `10.10.0.3`) — so `docker run --network
+  thermograph_internal …` from the dev box is a plausible route to prod-side
+  services. Untested; verify before relying on it.
+- WireGuard mesh: prod `10.10.0.1`, beta `10.10.0.2`, dev `10.10.0.3`.
+- `rclone` and `age` are already installed on prod and beta.
+- Reference implementation to copy: **`infra/ops/dbq.sh`** + `infra/ops/README.md`.
+
+## Storage — almost certainly your warehouse
+
+Contabo Object Storage (S3-compatible), already provisioned and in use:
+
+- Endpoint `https://eu2.contabostorage.com` · bucket `era5-thermograph` ·
+  region `default` · ~2 TB · **private** (keep it that way).
+- ⚠️ **Contabo requires PATH-STYLE addressing.** Set
+  `force_path_style=true` / `s3.path-style-access=true` (whatever your engine's
+  FileIO calls it) and `region=default`. Virtual-host-style addressing **fails**.
+  This is validated, not theoretical.
+- ⚠️ **The `backups/` prefix is in use** by the nightly backup jobs (prod DB +
+  Forgejo). Put Iceberg data under its own prefix (e.g. `iceberg/` or
+  `warehouse/`). Do not write outside your prefix.
+- Credentials already exist — **reuse them, don't mint new ones silently**:
+  - SOPS vault: `infra/deploy/secrets/{prod,beta}.yaml` as `THERMOGRAPH_S3_*`
+    (rendered to `/etc/thermograph.env` at deploy by
+    `infra/deploy/render-secrets.sh`).
+  - Forgejo Actions secrets: `S3_ENDPOINT`, `S3_BUCKET`, `S3_ACCESS_KEY`,
+    `S3_SECRET_KEY`.
+  - If read-only S3 keys are needed for the query identity, ask the operator to
+    mint a scoped keypair rather than reusing the read-write one.
+
+## Secrets handling
+
+- Host-side: SOPS + age. Recipient `age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2`;
+  the private key is on each host at `/etc/thermograph/age.key` and on the
+  operator's machine at `~/.config/sops/age/keys.txt`.
+- CI-side: Forgejo Actions secrets.
+- ⚠️ **beta gotcha:** `/etc/thermograph.env` on beta is `agent:agent 0640` — the
+  `deploy` user **cannot read it**. If your service or job runs as `deploy` on
+  beta, pull credentials from Actions secrets instead, or change the ownership
+  deliberately and say so.
+
+## Decisions you must make — and report back
+
+1. **Catalog**: REST (Lakekeeper / Nessie / Polaris), JDBC-on-Postgres, Hive, or
+   filesystem/hadoop — and where it runs per environment.
+2. **Engine**: DuckDB + iceberg extension, Trino, Spark, or pyiceberg. Prefer the
+   lightest that satisfies the contract; it must run containerised and headless.
+3. **Warehouse location**: bucket + prefix, and whether environments are isolated
+   by separate prefixes, namespaces, or buckets.
+4. **Read-only mechanism**: scoped S3 keys? catalog RBAC? engine restricted mode?
+   State which, and how a write is refused.
+5. **Where the engine runs** for each env (local container on dev, on the box for
+   beta/prod, or attached to the overlay) — consistent with constraint #3.
+
+## Acceptance criteria
+
+The work is accepted when all of these are demonstrated with pasted output:
+
+- [ ] `infra/ops/iceberg.sh <env> -c "<select>"` returns rows for **each**
+      environment that has data (explicitly state any env that doesn't yet).
+- [ ] **A write/DDL attempt is refused by the system**, with the error pasted.
+      The Postgres bar to match:
+      `create table` on prod → `ERROR:  permission denied for schema public`.
+- [ ] **No new exposure**: `sudo ss -ltnp` on prod and beta shows no new
+      listening port; no new ufw rules; no Swarm `EndpointSpec` publish added.
+      State this explicitly.
+- [ ] Non-interactive: extra flags pass through, stdin (`-f -`) works.
+- [ ] Prod container/task resolution is dynamic (survives a redeploy).
+- [ ] `infra/ops/README.md` updated with an Iceberg section: usage examples and
+      the *why* (same style as the Postgres rationale).
+
+## Anti-goals
+
+- ❌ Publishing catalog/engine/DB ports publicly, or adding firewall holes.
+- ❌ Any design requiring a persistent SSH tunnel (**prod cannot be tunnelled**).
+- ❌ Querying with an admin/superuser/read-write credential.
+- ❌ Secrets committed to the repo or pasted into workflow files.
+- ❌ Writing outside your own S3 prefix (`backups/` is live and load-bearing).
+
+## Questions to route back to the operator
+
+Anything that would require: minting new cloud credentials, opening a port,
+changing firewall/Swarm configuration, or writing to a shared prefix. Don't
+improvise on those — ask.

infra/ops/README.md

@@ -48,6 +48,10 @@ GRANT pg_read_all_data TO thermograph_ro;
 
 ## Iceberg (planned)
 
+**Implementation spec: [`ICEBERG-HANDOFF.md`](./ICEBERG-HANDOFF.md)** — the
+contract, environment constraints, acceptance criteria and anti-goals for the
+Iceberg worker.
+
 Iceberg lands as a sibling in this directory following the same three
 conventions, so the query surface stays uniform: