diff --git a/deploy/POSTGRES-MIGRATION.md b/deploy/POSTGRES-MIGRATION.md index 26e25d3..d512445 100644 --- a/deploy/POSTGRES-MIGRATION.md +++ b/deploy/POSTGRES-MIGRATION.md @@ -11,13 +11,19 @@ deploy to the compose stack, migrating the authoritative accounts data. tables. Managed by Alembic (`backend/alembic/`); `create_all` on a fresh DB. - **derived cache** (`store.py`) and **metrics** (`metrics.py`): now Postgres **UNLOGGED** tables (fast, non-durable — same throwaway semantics). No data to - migrate — the cache rebuilds from parquet, metrics start empty. + migrate — the cache rebuilds, metrics start empty. +- **climate record** (`climate.py`): the raw daily archive + recent/forecast bundle + moved from per-cell **parquet** files into **TimescaleDB hypertables** + (`climate_history` / `climate_recent` / `climate_sync`, managed by Alembic). The + app reads/writes them via `data/climate_store.py`. Backfilled from the existing + parquet cache with `migrate_cache_to_pg.py` (see below). - The app selects Postgres when `THERMOGRAPH_DATABASE_URL` is a `postgresql+asyncpg` - URL; unset ⇒ the old SQLite behavior (this is how the **test suite** stays on - SQLite — no Postgres needed in CI). -- `db` runs **pg_duckdb** (`deploy/db/Dockerfile.db`, genuine PG18): the app's - parquet cache is mounted read-only at `/parquet`, so you can query it directly, - e.g. `SELECT r['date'], r['tmax'] FROM read_parquet('/parquet/cache/*.parquet') r`. + URL; unset ⇒ the old SQLite/parquet behavior (this is how the **test suite** stays + Postgres-free — no Postgres needed in CI, and climate falls back to parquet). +- `db` runs the stock **`timescale/timescaledb:latest-pg18`** image (genuine PG18 + + TimescaleDB). The earlier pg_duckdb / `read_parquet('/parquet/…')` capability is + gone — the climate record is real tables now, queryable directly (see + `deploy/db/README.md`). ## Connection model @@ -64,10 +70,51 @@ unset/SQLite and restart). **Clean only during/immediately after the window** real users write to Postgres, those writes are lost on rollback (the copy is one-way), so keep the window short and writes frozen during the copy. +## TimescaleDB cutover (from the earlier pg_duckdb PG18 stack) + +Swapping the `db` image from `pgduckdb/pgduckdb:18` to +`timescale/timescaledb:latest-pg18` means a **fresh PGDATA volume** — and that +volume also holds the durable **accounts** data. So this cutover is not a plain +`docker compose pull`; it re-seeds both accounts and the climate record. + +1. **Back up first.** `pg_dump` the accounts tables from the *old* db container + (`docker compose exec db pg_dump -U thermograph -t user -t subscription \ + -t notification -t push_subscription -t pending_digest thermograph > accounts.sql`), + or keep the pre-Postgres `accounts.sqlite` as the source of truth. +2. **Replace the DB.** Deploy this branch (compose now points at the stock + TimescaleDB image, no `build:`). Bring down the stack and remove the old volume + so PGDATA re-inits on the new image: `docker compose down && docker volume rm + _pgdata`. +3. **Build the schema.** `docker compose up -d db`, then + `docker compose run --rm app alembic upgrade head` — this creates the accounts + tables *and* the climate hypertables, and `CREATE EXTENSION timescaledb`. +4. **Restore accounts.** Either `psql < accounts.sql`, or re-run + `migrate_accounts_to_pg.py --sqlite ` (§ the SQLite cutover + above). Verify per-table counts. +5. **Backfill the climate record** from the parquet cache (mount it and run the + backfill; idempotent, resumable): + ``` + docker compose run --rm \ + -v /opt/thermograph/data/cache:/app/data/cache:ro \ + app python migrate_cache_to_pg.py + ``` + It loads `climate_history` + `climate_recent` and sets each cell's + `climate_sync` to the parquet **file mtime**, so `recent_stamp` (hence every + derived-payload token) is unchanged across the cutover. Cells without a + schema-complete parquet record are skipped and refetch lazily on first request. +6. **Start serving:** `docker compose up -d`. Smoke test: login, a subscription, + a cached calendar/day request (served from `climate_history`), the metrics + dashboard. +7. Keep the pre-cutover DB backup (accounts dump + parquet cache) for a few days. + The climate record is now **durable** — make sure `pg_dump`/PITR covers it. + ## Notes - The two `deploy/migrations/*.sql` files are superseded by Alembic on Postgres (they were SQLite-specific `ALTER TABLE` column-adds for the old prod DB). -- The PG18 image's data dir is the versioned `…/18/docker` subdir; the compose +- The TimescaleDB image's data dir is `/var/lib/postgresql/data`; the compose volume is mounted at the parent `/var/lib/postgresql` so it persists without the initdb permission issue that pinning `PGDATA` to the mountpoint can trigger. +- Climate falls back to the parquet cache whenever `THERMOGRAPH_DATABASE_URL` is + not a Postgres URL (dev, tests, offline tooling) — the same dialect switch as the + accounts DB, so a DB outage in prod degrades to upstream refetch, never a 500. diff --git a/deploy/db/Dockerfile.db b/deploy/db/Dockerfile.db deleted file mode 100644 index d9951b4..0000000 --- a/deploy/db/Dockerfile.db +++ /dev/null @@ -1,25 +0,0 @@ -# Thermograph database image: PostgreSQL 18 with pg_duckdb. -# -# pg_duckdb embeds DuckDB inside Postgres, which lets the DB container read the -# app's Parquet climate cache directly for ad-hoc analytics: -# -# SELECT * FROM read_parquet('/parquet/1026_-2857.parquet'); -# -# Why FROM the official pg_duckdb image instead of `FROM postgres:18` + compile: -# pg_duckdb links a full DuckDB build, so compiling it from source in this -# Dockerfile would mean pulling the DuckDB toolchain and a long, fragile build. -# The maintainers (duckdb/pg_duckdb, MotherDuck) publish an official PG18 image -# that is genuine PostgreSQL 18.1 on Debian 12 bookworm -- the exact same base -# as the official `postgres:18` image, using the standard postgres -# docker-entrypoint.sh. So POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB / -# PGDATA / /docker-entrypoint-initdb.d / pg_isready all behave identically to -# `postgres:18`; this is a drop-in replacement for the compose `db` service. -# -# Pinned to a specific patch tag (not 18-main) for reproducible builds. -FROM pgduckdb/pgduckdb:18-v1.1.1 - -# Bake the parquet init script so the image enables the extension on first init -# even without the compose bind mount. It is CREATE EXTENSION IF NOT EXISTS, so -# it is idempotent with the base image's own 0001-install-pg_duckdb.sql. -# Build context is the repo root (see deploy/db/README.md for the compose snippet). -COPY deploy/db/init/ /docker-entrypoint-initdb.d/ diff --git a/deploy/db/README.md b/deploy/db/README.md index 905ab54..4f341c6 100644 --- a/deploy/db/README.md +++ b/deploy/db/README.md @@ -1,171 +1,87 @@ -# Thermograph DB image — Postgres 18 + parquet reads +# Thermograph DB — TimescaleDB (PostgreSQL 18) -The `db` service can read the app's Parquet climate cache -(`data/cache/*.parquet`) directly from SQL, for ad-hoc analytics, via -**pg_duckdb** — DuckDB embedded inside Postgres. +The `db` service runs the stock **`timescale/timescaledb:latest-pg18`** image +(TimescaleDB 2.24+, genuine PostgreSQL 18). The app's climate record lives here in +**hypertables** — the DB, not the filesystem, is the source of truth: -```sql -SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin -FROM read_parquet('/parquet/1026_-2857.parquet') r -ORDER BY r['date'] LIMIT 5; -``` +- **`climate_history`** — a hypertable of the full daily archive per grid cell + (`cell_id, date, tmax, tmin, precip, wind, gust, humid, fmax, fmin, feels`), back + to 1980. Durable/LOGGED (a 45-year, rate-limited refetch is expensive), + range-partitioned on `date` (5-year chunks), compressed for chunks older than a + year. +- **`climate_recent`** — the recent-observations + forward-forecast bundle (a plain + table: it holds future dates and is rewritten hourly). +- **`climate_sync`** — per-cell freshness (epoch seconds) that replaces the old + parquet file mtimes: it drives the hourly history top-up, the 1-hour forecast + TTL, and the `recent_stamp` token embedded in derived-payload validity. -## Chosen extension: pg_duckdb — and why +The schema is created by Alembic (`backend/alembic/versions/0002_climate_hypertables.py`, +run at app boot via `deploy/entrypoint.sh`). The app reads/writes it through +`backend/data/climate_store.py` (psycopg + polars). See +`deploy/POSTGRES-MIGRATION.md` for the parquet→hypertable cutover. -| Option | PG18? | Fit | Verdict | -| --- | --- | --- | --- | -| **pg_duckdb** (duckdb / MotherDuck) | **Yes** — official image `pgduckdb/pgduckdb:18-v1.1.1` is genuine PG18.1 | `read_parquet('…')` in plain SQL; globs, `union_by_name`, full DuckDB analytics engine | **Chosen** | -| pg_parquet (Crunchy Data) | Yes (14–18) | `COPY … TO/FROM '…' (format 'parquet')` — import/export, not a query engine | Viable, but COPY-oriented; no standalone official image (ships via Crunchy Bridge/CPK), so it'd need a Rust/pgrx source build | -| parquet_fdw | No prebuilt PG18 support; low activity | Foreign tables over parquet | Rejected — oldest, weakest PG18 story | +## Why the stock image (no custom Dockerfile) -pg_duckdb wins for the stated goal (ad-hoc analytics): it exposes DuckDB's -`read_parquet` directly in SQL, so you query cache files like tables — no -import step, no foreign-table DDL, and you get aggregation/joins/globs across -many cells at once. - -## PG-version reality for PG18 (empirically verified 2026-07-19) - -**No version delta.** PG18 support is real, not a fallback. The pulled image -reports: - -``` -PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2) on x86_64-pc-linux-gnu -pg_extension: pg_duckdb 1.1.0 -shared_preload_libraries: pg_duckdb -``` - -The image is built on Debian 12 bookworm — the **same base as the official -`postgres:18` image** — and uses the standard `docker-entrypoint.sh`. So -`POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` / `PGDATA` / -`/docker-entrypoint-initdb.d` / `pg_isready` all behave exactly as with -`postgres:18`. It is a drop-in replacement for the `db` service; nothing else -in the stack changes. - -We `FROM` the official pg_duckdb image (pinned to `18-v1.1.1`, not `18-main`) -rather than `FROM postgres:18` + compile, because pg_duckdb links a full DuckDB -build — compiling from source in the Dockerfile means the DuckDB toolchain and -a long, fragile build for no benefit over the maintainers' official PG18 image. +The previous DB image was a custom `pgduckdb/pgduckdb:18` build whose only purpose +was ad-hoc `read_parquet()` over the parquet cache. Now the climate record is in +real tables, so that capability is gone and the DB is the **stock TimescaleDB +image** — no build step. The image already sets +`shared_preload_libraries=timescaledb`; never `ALTER SYSTEM SET +shared_preload_libraries` (it would land in `postgresql.auto.conf` and override the +image's preload). ## Files here -- **`Dockerfile.db`** — `FROM pgduckdb/pgduckdb:18-v1.1.1`, plus `COPY` of the - init script so the image enables the extension on first init even without the - compose bind mount. -- **`init/10-parquet.sql`** — `CREATE EXTENSION IF NOT EXISTS pg_duckdb;` (runs - from `/docker-entrypoint-initdb.d` on first cluster init). +- **`init/10-timescaledb.sql`** — `CREATE EXTENSION IF NOT EXISTS timescaledb;` + (runs from `/docker-entrypoint-initdb.d` on first cluster init; Alembic also does + this idempotently at boot). +- **`init/20-tuning.sh`** — scales `shared_buffers` (25%), `effective_cache_size` + (75%), `work_mem`, and `maintenance_work_mem` from `DB_MEMORY` via `ALTER SYSTEM`. -## Compose snippet to merge into the `db` service - -Replace `image: postgres:18` with the `build:` block; add the read-only parquet -bind and the init mount. Everything else in the `db` service stays as-is. +## Compose `db` service ```yaml db: - # image: postgres:18 # <- remove; build the parquet-capable image - build: - context: . - dockerfile: deploy/db/Dockerfile.db + image: timescale/timescaledb:latest-pg18 environment: POSTGRES_USER: thermograph POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD} POSTGRES_DB: thermograph - PGDATA: /var/lib/postgresql/data + DB_MEMORY: ${DB_MEMORY:-8g} volumes: - - pgdata:/var/lib/postgresql/data - - ./data/cache:/parquet:ro # read-only parquet cache + - pgdata:/var/lib/postgresql - ./deploy/db/init:/docker-entrypoint-initdb.d - # healthcheck / cpus / deploy / restart: unchanged + # healthcheck / cpus / mem_limit / shm_size: unchanged ``` Notes: -- The bind mount at `/docker-entrypoint-initdb.d` **replaces** the base image's - own init scripts (its `0001-install-pg_duckdb.sql` and the MotherDuck-only - `0002-enable-md-pg_duckdb.sql`). That's intended: our `10-parquet.sql` still - runs `CREATE EXTENSION`, and we don't use MotherDuck. If you prefer to keep - the image's baked scripts, drop the `:/docker-entrypoint-initdb.d` line — the - `Dockerfile` already bakes `10-parquet.sql` in. -- `:ro` keeps the DB from ever mutating the app's cache. In prod the app writes - the cache to the `appdata` volume; point this bind at wherever that lives on - the host if you want the DB to see the live cache rather than the repo copy. -- Init scripts only run when `PGDATA` is empty. On an **existing** database, - enable it once by hand: +- The volume is mounted at the **parent** of the data dir; the image picks its own + PGDATA subdir (`/var/lib/postgresql/data`) under it. The whole tree persists on + the named volume. +- No host port on purpose: the app reaches Postgres as `db:5432` on the compose + network. Nothing outside the stack should touch the database. +- Init scripts only run when PGDATA is empty. On an **existing** database, enable + the extension once by hand: ``` docker compose exec db psql -U thermograph -d thermograph \ - -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;' + -c 'CREATE EXTENSION IF NOT EXISTS timescaledb;' ``` -## Usage — real cache file at `/parquet/…` - -Cache columns: `date, tmax, tmin, precip, wind, gust, humid, fmax, fmin, feels`. -pg_duckdb ≥ 0.3 uses the `r['colname']` subscript syntax (not -`AS t(col type, …)`). +## Inspecting the hypertable ```sql --- one cell, first rows -SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin, r['precip'] AS precip -FROM read_parquet('/parquet/1026_-2857.parquet') r -ORDER BY r['date'] LIMIT 5; +-- Chunk / compression overview +SELECT hypertable_name, num_chunks, compression_enabled +FROM timescaledb_information.hypertables; --- per-year analytics over one cell -SELECT EXTRACT(YEAR FROM r['date']::timestamp) AS yr, - ROUND(AVG(r['tmax'])::numeric, 1) AS avg_tmax, - MAX(r['tmax']) AS record_high -FROM read_parquet('/parquet/1026_-2857.parquet') r -WHERE r['date'] >= '2020-01-01' +-- One cell, most recent archived days +SELECT date, tmax, tmin, precip +FROM climate_history WHERE cell_id = '1026_-2857' +ORDER BY date DESC LIMIT 5; + +-- Per-year highs for one cell +SELECT EXTRACT(YEAR FROM date) AS yr, + ROUND(AVG(tmax)::numeric, 1) AS avg_tmax, MAX(tmax) AS record_high +FROM climate_history WHERE cell_id = '1026_-2857' AND date >= '2020-01-01' GROUP BY yr ORDER BY yr; - --- glob across every cached cell (union_by_name handles the _rf / _forecast --- files whose column sets differ) -SELECT COUNT(*) FROM read_parquet('/parquet/*.parquet', union_by_name := true) r; -``` - -## Proof of work (verified 2026-07-19) - -Built `deploy/db/Dockerfile.db`, ran a throwaway container with the real -`data/cache` bind-mounted read-only at `/parquet`, then: - -``` -$ psql -c "SELECT version();" - PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2) on x86_64-pc-linux-gnu ... - -$ psql -c "SELECT extname, extversion FROM pg_extension WHERE extname='pg_duckdb';" - extname | extversion ------------+------------ - pg_duckdb | 1.1.0 - -$ psql -c "SELECT COUNT(*) FROM read_parquet('/parquet/1026_-2857.parquet') r;" - row_count ------------ - 16982 - -$ psql -c "SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin, - r['precip'] AS precip, r['humid'] AS humid - FROM read_parquet('/parquet/1026_-2857.parquet') r - ORDER BY r['date'] LIMIT 5;" - date | tmax | tmin | precip | humid ----------------------+------+------+--------+------- - 1980-01-01 00:00:00 | 56.2 | 34.7 | 0 | 73 - 1980-01-02 00:00:00 | 63.8 | 38 | 0 | 79 - 1980-01-03 00:00:00 | 60.1 | 46.1 | 0.315 | 83 - 1980-01-04 00:00:00 | 51.7 | 40 | 0 | 68 - 1980-01-05 00:00:00 | 56.5 | 33.7 | 0 | 78 - -$ psql -c "SELECT EXTRACT(YEAR FROM r['date']::timestamp) AS yr, - ROUND(AVG(r['tmax'])::numeric,1) AS avg_tmax, MAX(r['tmax']) AS record_high - FROM read_parquet('/parquet/1026_-2857.parquet') r - WHERE r['date'] >= '2020-01-01' GROUP BY yr ORDER BY yr;" - yr | avg_tmax | record_high -------+----------+------------- - 2020 | 78.8 | 99 - 2021 | 77.0 | 93 - 2022 | 79.2 | 101.4 - 2023 | 80.8 | 107.1 - 2024 | 79.8 | 97.5 - 2025 | 80.0 | 99.4 - 2026 | 77.8 | 95.6 - -$ psql -c "SELECT COUNT(*) FROM read_parquet('/parquet/*.parquet', union_by_name := true) r;" - count --------- - 525421 -- rows across all 45 cached cells ``` diff --git a/deploy/db/init/10-parquet.sql b/deploy/db/init/10-parquet.sql deleted file mode 100644 index d9e16be..0000000 --- a/deploy/db/init/10-parquet.sql +++ /dev/null @@ -1,15 +0,0 @@ --- Enable pg_duckdb so the database can read the app's Parquet climate cache --- directly (ad-hoc analytics via DuckDB's read_parquet / read_csv / etc.). --- --- This runs once, on first cluster init (empty PGDATA), from --- /docker-entrypoint-initdb.d. Because the compose `db` service keeps its data --- on a persistent named volume, init scripts do NOT re-run on an existing --- database -- to enable pg_duckdb on a DB that was created before this image, --- run it by hand: --- --- docker compose exec db psql -U thermograph -d thermograph \ --- -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;' --- --- pg_duckdb requires shared_preload_libraries='pg_duckdb'; the base image's --- postgresql.conf.sample already sets it, so it is active on fresh init. -CREATE EXTENSION IF NOT EXISTS pg_duckdb; diff --git a/deploy/db/init/10-timescaledb.sql b/deploy/db/init/10-timescaledb.sql new file mode 100644 index 0000000..224c629 --- /dev/null +++ b/deploy/db/init/10-timescaledb.sql @@ -0,0 +1,14 @@ +-- Enable TimescaleDB so the app's climate record lives in hypertables +-- (climate_history) instead of parquet files. The stock timescale/timescaledb +-- image already preloads the extension (shared_preload_libraries=timescaledb); +-- this just runs CREATE EXTENSION on first cluster init. +-- +-- This runs once, on first cluster init (empty PGDATA), from +-- /docker-entrypoint-initdb.d. Because the compose `db` service keeps its data on +-- a persistent named volume, init scripts do NOT re-run on an existing database. +-- Alembic (backend/alembic) also runs `CREATE EXTENSION IF NOT EXISTS timescaledb` +-- at app boot, so an existing volume gets it there; to enable it by hand instead: +-- +-- docker compose exec db psql -U thermograph -d thermograph \ +-- -c 'CREATE EXTENSION IF NOT EXISTS timescaledb;' +CREATE EXTENSION IF NOT EXISTS timescaledb; diff --git a/deploy/db/init/20-tuning.sh b/deploy/db/init/20-tuning.sh index 0ed1748..eabc6fe 100755 --- a/deploy/db/init/20-tuning.sh +++ b/deploy/db/init/20-tuning.sh @@ -2,9 +2,12 @@ # Postgres memory / performance tuning, scaled to the container's DB_MEMORY budget so # the same init serves every host (beta 8g; prod 16g on the 48 GB box) with no # hardcoding. Runs once on a fresh data volume from /docker-entrypoint-initdb.d, after -# 10-parquet.sql enables pg_duckdb. Settings are written via ALTER SYSTEM (persisted to -# postgresql.auto.conf); the container's post-init restart brings restart-only settings -# (shared_buffers, …) into effect. Init scripts do NOT re-run on an existing volume — to +# 10-timescaledb.sql enables timescaledb. Settings are written via ALTER SYSTEM +# (persisted to postgresql.auto.conf, which the timescaledb image's own +# timescaledb-tune postgresql.conf defers to); the container's post-init restart brings +# restart-only settings (shared_buffers, …) into effect. NB: never ALTER SYSTEM SET +# shared_preload_libraries here — that would land in auto.conf and override the image's +# `timescaledb` preload. Init scripts do NOT re-run on an existing volume — to # re-tune later, set DB_MEMORY and run this by hand, then restart: # docker compose exec -e DB_MEMORY=16g db bash /docker-entrypoint-initdb.d/20-tuning.sh # docker compose restart db @@ -23,18 +26,17 @@ esac if [ "$mb" -lt 1024 ]; then mb=8192; fi # Derive settings from the budget. The ratios reproduce the historical 8 GB tuning -# (shared_buffers 2 GB, effective_cache_size 6 GB, duckdb 4 GB, work_mem 64 MB, +# (shared_buffers 2 GB, effective_cache_size 6 GB, work_mem 64 MB, # maintenance_work_mem 512 MB) and scale linearly on a bigger box. shared_buffers=$((mb / 4)) # 25% — the shared page cache effective_cache=$((mb * 3 / 4)) # 75% — planner's view of total cache (PG + OS) -duckdb_mem=$((mb / 2)) # 50% — pg_duckdb ceiling for parquet processing work_mem=$((mb / 128)) # ~64 MB at 8 GB (per-operation; kept modest) if [ "$work_mem" -lt 16 ]; then work_mem=16; fi maint_mem=$((mb / 16)) # 512 MB at 8 GB — index builds / VACUUM echo "[tuning] DB_MEMORY=${budget} -> ${mb}MB: shared_buffers=${shared_buffers}MB" \ "effective_cache_size=${effective_cache}MB work_mem=${work_mem}MB" \ - "maintenance_work_mem=${maint_mem}MB duckdb.max_memory=${duckdb_mem}MB" + "maintenance_work_mem=${maint_mem}MB" psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <