From 151cf37dfa83c148399960727425ffba79b395c6 Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Thu, 23 Jul 2026 14:54:20 -0700 Subject: [PATCH 1/2] Add ops/dbq.sh: read-only Postgres queries across all environments Uniform read-only query access to the LAN dev, beta and prod databases, plus a thermograph_ro role in each to enforce it. None of the databases are exposed over TCP: each listens only on its private docker network, and prod's is a Swarm overlay the host cannot route to -- so ssh -L works for beta but is impossible for prod, and publishing 5432 would mean new ufw rules and a Swarm endpoint change on production. Running psql inside the db container works identically everywhere with no ports, tunnels or infra changes, so dbq.sh does that: local docker exec for dev, ssh for beta/prod. The prod container is a Swarm task whose name changes each redeploy, so it is resolved at call time rather than hardcoded. Queries connect as thermograph_ro (NOSUPERUSER, granted only pg_read_all_data), so a stray write fails with "permission denied" even against prod; the app's own superuser role is deliberately unused here. Extra args pass through to psql and stdin is forwarded. ops/README.md documents usage and the conventions Iceberg will follow when it lands as a sibling (env-keyed dispatch, run the engine where the data is reachable, dedicated read-only identity). --- infra/ops/README.md | 61 ++++++++++++++++++++++++++++++++++++++++ infra/ops/dbq.sh | 68 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 129 insertions(+) create mode 100644 infra/ops/README.md create mode 100755 infra/ops/dbq.sh diff --git a/infra/ops/README.md b/infra/ops/README.md new file mode 100644 index 0000000..9a6993e --- /dev/null +++ b/infra/ops/README.md @@ -0,0 +1,61 @@ +# ops/ — operator + agent query tooling + +Read-only query access to Thermograph data, uniform across environments. + +## `dbq.sh` — Postgres, any environment + +```sh +infra/ops/dbq.sh dev "select count(*) from climate_history" +infra/ops/dbq.sh prod -tA -c "select max(date) from climate_history" +infra/ops/dbq.sh beta -c '\dt' +echo "select 1" | infra/ops/dbq.sh prod -f - +``` + +Environments: `dev` (LAN compose stack on the dev machine), `beta` +(75.119.132.91), `prod` (169.58.46.181). Extra args pass straight through to +`psql` (`-tA`, `-x`, `--csv`, `-f`, …); stdin is forwarded. + +### Why it execs into the container instead of using a connection string + +None of the databases are exposed over TCP — each listens only on its private +docker network. Prod's is a Swarm **overlay** (`10.0.2.0/24`) that the host +cannot route to, so `ssh -L` works for beta but is *impossible* for prod; +publishing 5432 would mean new ufw rules plus a Swarm endpoint change on +production. Running `psql` inside the db container works identically in all +three environments with no ports, no tunnels, and no infra changes. + +The prod container name is a Swarm task name that changes on every redeploy, so +it is resolved at call time via `docker ps --filter name=…`, never hardcoded. + +### Safety + +Queries connect as **`thermograph_ro`** — a `NOSUPERUSER` role granted only +`pg_read_all_data`. Read-only is enforced by Postgres, not by convention: + +``` +$ infra/ops/dbq.sh prod -c "create table t(i int)" +ERROR: permission denied for schema public +``` + +The app's own `thermograph` role is a superuser and is deliberately **not** used +by this tool. If the role is ever missing (fresh database), recreate it with: + +```sql +CREATE ROLE thermograph_ro LOGIN; +GRANT CONNECT ON DATABASE thermograph TO thermograph_ro; +GRANT pg_read_all_data TO thermograph_ro; +``` + +## Iceberg (planned) + +Iceberg lands as a sibling in this directory following the same three +conventions, so the query surface stays uniform: + +1. **Env-keyed dispatch** — `iceberg.sh ""`, same argument + shape as `dbq.sh`. +2. **Run the engine where the data is reachable** — the catalog/warehouse + dictates it (a Trino/DuckDB/pyiceberg invocation, containerised the same way), + rather than exposing new network endpoints. +3. **A dedicated read-only identity** — the Iceberg equivalent of + `thermograph_ro`, so the same "read-only enforced by the system, not by + convention" property holds. diff --git a/infra/ops/dbq.sh b/infra/ops/dbq.sh new file mode 100755 index 0000000..c2c0cac --- /dev/null +++ b/infra/ops/dbq.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# dbq -- run READ-ONLY SQL against any Thermograph Postgres (LAN dev / beta / prod). +# +# infra/ops/dbq.sh dev "select count(*) from climate_history" +# infra/ops/dbq.sh prod -tA "select max(date) from climate_history" +# infra/ops/dbq.sh beta -c '\dt' +# echo "select 1" | infra/ops/dbq.sh prod -f - +# +# Why exec-into-the-container instead of a connection string: +# none of the databases are exposed over TCP. Each listens only on its private +# docker network -- and prod's is a Swarm *overlay* (10.0.2.0/24) that the host +# itself cannot route to, so `ssh -L` works for beta but is impossible for prod. +# Publishing 5432 would mean new firewall + Swarm endpoint changes on production. +# Running psql *inside* the db container works identically in all three +# environments with no ports, no tunnels and no infra changes -- locally for LAN +# dev, over SSH for beta/prod. +# +# Safety: always connects as `thermograph_ro`, a NOSUPERUSER role granted only +# pg_read_all_data. Read-only is enforced by Postgres itself, not by convention, +# so a stray INSERT/DDL fails with "permission denied" even against prod. (The +# app's own `thermograph` role is a superuser -- deliberately not used here.) +# +# Any extra arguments are passed straight through to psql, so -tA, -c, -f, -x, +# --csv etc. all work. Stdin is forwarded, so `-f -` reads piped SQL. +set -euo pipefail + +KEYFILE="${THERMOGRAPH_AGENT_KEY:-$HOME/.ssh/thermograph_agent_ed25519}" +DB_USER="${THERMOGRAPH_DB_QUERY_USER:-thermograph_ro}" +DB_NAME="${THERMOGRAPH_DB_NAME:-thermograph}" + +usage() { + sed -n '2,9p' "$0" | sed 's/^# \{0,1\}//' >&2 + echo "environments: dev | beta | prod" >&2 + exit 2 +} + +env_name="${1:-}" +[ -n "$env_name" ] || usage +shift + +case "$env_name" in + dev) filter=thermograph-dev-db ; ssh_target= ;; + beta) filter=thermograph-db-1 ; ssh_target=agent@75.119.132.91 ;; + prod) filter=thermograph_db ; ssh_target=agent@169.58.46.181 ;; + -h|--help) usage ;; + *) echo "dbq: unknown environment '$env_name' (want: dev|beta|prod)" >&2; exit 2 ;; +esac + +[ $# -gt 0 ] || usage + +# Quote the psql arguments so they survive the remote shell intact. +psql_args=$(printf '%q ' "$@") + +# The container name is resolved on the target host at call time: prod's Swarm +# task name changes on every redeploy, so it can never be hardcoded. +remote=$(cat <&2; exit 1; } +exec docker exec -i "\$cid" psql -U ${DB_USER} -d ${DB_NAME} -v ON_ERROR_STOP=1 ${psql_args} +REMOTE +) + +if [ -z "$ssh_target" ]; then + exec bash -c "$remote" +else + exec ssh -i "$KEYFILE" -o StrictHostKeyChecking=no -o ConnectTimeout=15 \ + "$ssh_target" "$remote" +fi -- 2.45.2 From bb7ce43902c5fd95c713f6498fad1476fcb18d52 Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Thu, 23 Jul 2026 15:00:51 -0700 Subject: [PATCH 2/2] Add ICEBERG-HANDOFF.md: implementation spec for the Iceberg query service Handoff for the Iceberg worker: the contract iceberg.sh must satisfy (same interface shape as dbq.sh, read-only enforced by the system, no new network endpoints, call-time container resolution), the environment constraints that will otherwise bite -- prod's Swarm overlay is unroutable from its own host so tunnels are impossible, Contabo object storage requires path-style addressing, the backups/ prefix is live, and beta's /etc/thermograph.env is unreadable by the deploy user -- plus the decisions to report back, acceptance criteria with a demonstrated refused write, and anti-goals. --- infra/ops/ICEBERG-HANDOFF.md | 133 +++++++++++++++++++++++++++++++++++ infra/ops/README.md | 4 ++ 2 files changed, 137 insertions(+) create mode 100644 infra/ops/ICEBERG-HANDOFF.md diff --git a/infra/ops/ICEBERG-HANDOFF.md b/infra/ops/ICEBERG-HANDOFF.md new file mode 100644 index 0000000..d27810b --- /dev/null +++ b/infra/ops/ICEBERG-HANDOFF.md @@ -0,0 +1,133 @@ +# Iceberg: implementing the agent-queryable service + +Handoff spec for whoever builds the Iceberg layer. The Postgres equivalent is +already shipped (`infra/ops/dbq.sh`) — **mirror it**. This document is the +contract, the environment facts you'll trip over, and how the result gets +verified. + +## Definition of done + +`infra/ops/iceberg.sh` exists and behaves like `dbq.sh`: + +```sh +infra/ops/iceberg.sh prod -c "select count(*) from ." +infra/ops/iceberg.sh dev -c "show tables" +echo "select 1" | infra/ops/iceberg.sh beta -f - +``` + +An agent (or operator) can query Iceberg in any environment, read-only, with no +new network exposure and no interactive steps. + +## The contract (non-negotiable) + +1. **Same interface shape as `dbq.sh`** — `iceberg.sh [flags] + ""`. Extra flags pass through to the engine; **stdin is forwarded** so + `-f -` works. Non-interactive, deterministic, safe to call from CI. +2. **Read-only enforced by the system, not by convention.** A dedicated + read-only identity — not an admin/superuser credential. You must be able to + *demonstrate* a refused write (see Acceptance). +3. **No new network endpoints.** Run the engine where the data is already + reachable (containerised), rather than publishing ports or opening firewall + rules. See the prod constraint below — this is not negotiable, it's physics. +4. **Env-keyed dispatch, names resolved at call time.** Prod is Docker Swarm; + task container names change on **every redeploy**. Resolve via + `docker ps --filter name=…`; never hardcode a container name. +5. **No secrets in the repo** or in workflow files. + +## Environment facts you need + +- Monorepo `emi/thermograph` (domain dirs: `backend/ frontend/ infra/ + observability/`). Ops tooling lives in `infra/ops/`. +- Environments: **dev** = LAN compose stack on the dev machine; **beta** = + `75.119.132.91`; **prod** = `169.58.46.181`. SSH as `agent` with + `~/.ssh/thermograph_agent_ed25519` (passwordless sudo on both boxes). +- **Prod runs Docker Swarm.** Its app database sits on the overlay network + `thermograph_internal` (`10.0.2.0/24`), which **the prod host itself cannot + route to**. Consequence, learned the hard way: `ssh -L` tunnelling works for + beta but is *impossible* for prod. Any design that depends on a tunnel or a + published port will fail on prod — that's why `dbq.sh` execs into the + container instead. Assume the same for anything you deploy there. +- The overlay is `attachable=true`, and the **dev machine is a Swarm worker** in + the prod cluster (NodeAddr `10.10.0.3`) — so `docker run --network + thermograph_internal …` from the dev box is a plausible route to prod-side + services. Untested; verify before relying on it. +- WireGuard mesh: prod `10.10.0.1`, beta `10.10.0.2`, dev `10.10.0.3`. +- `rclone` and `age` are already installed on prod and beta. +- Reference implementation to copy: **`infra/ops/dbq.sh`** + `infra/ops/README.md`. + +## Storage — almost certainly your warehouse + +Contabo Object Storage (S3-compatible), already provisioned and in use: + +- Endpoint `https://eu2.contabostorage.com` · bucket `era5-thermograph` · + region `default` · ~2 TB · **private** (keep it that way). +- ⚠️ **Contabo requires PATH-STYLE addressing.** Set + `force_path_style=true` / `s3.path-style-access=true` (whatever your engine's + FileIO calls it) and `region=default`. Virtual-host-style addressing **fails**. + This is validated, not theoretical. +- ⚠️ **The `backups/` prefix is in use** by the nightly backup jobs (prod DB + + Forgejo). Put Iceberg data under its own prefix (e.g. `iceberg/` or + `warehouse/`). Do not write outside your prefix. +- Credentials already exist — **reuse them, don't mint new ones silently**: + - SOPS vault: `infra/deploy/secrets/{prod,beta}.yaml` as `THERMOGRAPH_S3_*` + (rendered to `/etc/thermograph.env` at deploy by + `infra/deploy/render-secrets.sh`). + - Forgejo Actions secrets: `S3_ENDPOINT`, `S3_BUCKET`, `S3_ACCESS_KEY`, + `S3_SECRET_KEY`. + - If read-only S3 keys are needed for the query identity, ask the operator to + mint a scoped keypair rather than reusing the read-write one. + +## Secrets handling + +- Host-side: SOPS + age. Recipient `age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2`; + the private key is on each host at `/etc/thermograph/age.key` and on the + operator's machine at `~/.config/sops/age/keys.txt`. +- CI-side: Forgejo Actions secrets. +- ⚠️ **beta gotcha:** `/etc/thermograph.env` on beta is `agent:agent 0640` — the + `deploy` user **cannot read it**. If your service or job runs as `deploy` on + beta, pull credentials from Actions secrets instead, or change the ownership + deliberately and say so. + +## Decisions you must make — and report back + +1. **Catalog**: REST (Lakekeeper / Nessie / Polaris), JDBC-on-Postgres, Hive, or + filesystem/hadoop — and where it runs per environment. +2. **Engine**: DuckDB + iceberg extension, Trino, Spark, or pyiceberg. Prefer the + lightest that satisfies the contract; it must run containerised and headless. +3. **Warehouse location**: bucket + prefix, and whether environments are isolated + by separate prefixes, namespaces, or buckets. +4. **Read-only mechanism**: scoped S3 keys? catalog RBAC? engine restricted mode? + State which, and how a write is refused. +5. **Where the engine runs** for each env (local container on dev, on the box for + beta/prod, or attached to the overlay) — consistent with constraint #3. + +## Acceptance criteria + +The work is accepted when all of these are demonstrated with pasted output: + +- [ ] `infra/ops/iceberg.sh -c "