Jinemi/thermograph
4
0
Fork 0
Code Issues 2 Pull requests 3 Projects Releases Packages Wiki Activity Actions

Move the climate record from parquet to TimescaleDB hypertables (#227)

Browse source 
Replace the per-cell parquet cache with TimescaleDB hypertables as the
production backend for the raw daily climate record, and drop pg_duckdb.
Parquet stays the backend whenever THERMOGRAPH_DATABASE_URL is not a
Postgres URL (dev, tests, offline tooling), the same dialect switch the
accounts DB and derived store already use, so CI stays Postgres-free.

- data/climate_store.py: psycopg + polars bridge over climate_history
  (a hypertable), climate_recent, and climate_sync (per-cell freshness).
  Reads via pl.read_database, writes via COPY + ON CONFLICT upsert;
  fail-soft to a cache miss so a DB hiccup degrades to upstream refetch.
- data/climate.py: route every cache/mtime touchpoint through a backend
  dispatch. recent_stamp becomes int(recent_synced_at) on Postgres; the
  stale-serve path still avoids bumping it, so derived-payload tokens
  invalidate on exactly the same events as before.
- alembic 0002: CREATE EXTENSION timescaledb plus the hypertable schema
  (compression policy on year-old chunks), guarded to no-op off Postgres.
- migrate_cache_to_pg.py (make migrate-cache): idempotent backfill of the
  parquet cache into the hypertables, preserving file mtimes as sync
  timestamps so recent_stamp is unchanged across cutover.
- db image -> stock timescale/timescaledb:latest-pg18; drop the custom
  pg_duckdb Dockerfile, the read-only /parquet mount, and the duckdb
  tuning GUC. Docs updated for the new backend and cutover.

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Emi Griffith 2026-07-20 13:15:55 -07:00 committed by GitHub
parent dc4ee9a8db
commit 1f7e8552cf
9 changed files with 154 additions and 221 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

61
deploy/POSTGRES-MIGRATION.md
View file

@ -11,13 +11,19 @@ deploy to the compose stack, migrating the authoritative accounts data.
tables. Managed by Alembic (`backend/alembic/`); `create_all` on a fresh DB. tables. Managed by Alembic (`backend/alembic/`); `create_all` on a fresh DB.
- **derived cache** (`store.py`) and **metrics** (`metrics.py`): now Postgres - **derived cache** (`store.py`) and **metrics** (`metrics.py`): now Postgres
**UNLOGGED** tables (fast, non-durable — same throwaway semantics). No data to **UNLOGGED** tables (fast, non-durable — same throwaway semantics). No data to
migrate — the cache rebuilds from parquet, metrics start empty. migrate — the cache rebuilds, metrics start empty.
- **climate record** (`climate.py`): the raw daily archive + recent/forecast bundle
moved from per-cell **parquet** files into **TimescaleDB hypertables**
(`climate_history` / `climate_recent` / `climate_sync`, managed by Alembic). The
app reads/writes them via `data/climate_store.py`. Backfilled from the existing
parquet cache with `migrate_cache_to_pg.py` (see below).
- The app selects Postgres when `THERMOGRAPH_DATABASE_URL` is a `postgresql+asyncpg` - The app selects Postgres when `THERMOGRAPH_DATABASE_URL` is a `postgresql+asyncpg`
URL; unset ⇒ the old SQLite behavior (this is how the **test suite** stays on URL; unset ⇒ the old SQLite/parquet behavior (this is how the **test suite** stays
SQLite — no Postgres needed in CI). Postgres-free — no Postgres needed in CI, and climate falls back to parquet).
- `db` runs **pg_duckdb** (`deploy/db/Dockerfile.db`, genuine PG18): the app's - `db` runs the stock **`timescale/timescaledb:latest-pg18`** image (genuine PG18 +
parquet cache is mounted read-only at `/parquet`, so you can query it directly, TimescaleDB). The earlier pg_duckdb / `read_parquet('/parquet/…')` capability is
e.g. `SELECT r['date'], r['tmax'] FROM read_parquet('/parquet/cache/*.parquet') r`. gone — the climate record is real tables now, queryable directly (see
`deploy/db/README.md`).
## Connection model ## Connection model
@ -64,10 +70,51 @@ unset/SQLite and restart). **Clean only during/immediately after the window**
real users write to Postgres, those writes are lost on rollback (the copy is real users write to Postgres, those writes are lost on rollback (the copy is
one-way), so keep the window short and writes frozen during the copy. one-way), so keep the window short and writes frozen during the copy.
## TimescaleDB cutover (from the earlier pg_duckdb PG18 stack)
Swapping the `db` image from `pgduckdb/pgduckdb:18` to
`timescale/timescaledb:latest-pg18` means a **fresh PGDATA volume** — and that
volume also holds the durable **accounts** data. So this cutover is not a plain
`docker compose pull`; it re-seeds both accounts and the climate record.
1. **Back up first.** `pg_dump` the accounts tables from the *old* db container
(`docker compose exec db pg_dump -U thermograph -t user -t subscription \
-t notification -t push_subscription -t pending_digest thermograph > accounts.sql`),
or keep the pre-Postgres `accounts.sqlite` as the source of truth.
2. **Replace the DB.** Deploy this branch (compose now points at the stock
TimescaleDB image, no `build:`). Bring down the stack and remove the old volume
so PGDATA re-inits on the new image: `docker compose down && docker volume rm
<project>_pgdata`.
3. **Build the schema.** `docker compose up -d db`, then
`docker compose run --rm app alembic upgrade head` — this creates the accounts
tables *and* the climate hypertables, and `CREATE EXTENSION timescaledb`.
4. **Restore accounts.** Either `psql < accounts.sql`, or re-run
`migrate_accounts_to_pg.py --sqlite <accounts.sqlite>` (§ the SQLite cutover
above). Verify per-table counts.
5. **Backfill the climate record** from the parquet cache (mount it and run the
backfill; idempotent, resumable):
```
docker compose run --rm \
-v /opt/thermograph/data/cache:/app/data/cache:ro \
app python migrate_cache_to_pg.py
```
It loads `climate_history` + `climate_recent` and sets each cell's
`climate_sync` to the parquet **file mtime**, so `recent_stamp` (hence every
derived-payload token) is unchanged across the cutover. Cells without a
schema-complete parquet record are skipped and refetch lazily on first request.
6. **Start serving:** `docker compose up -d`. Smoke test: login, a subscription,
a cached calendar/day request (served from `climate_history`), the metrics
dashboard.
7. Keep the pre-cutover DB backup (accounts dump + parquet cache) for a few days.
The climate record is now **durable** — make sure `pg_dump`/PITR covers it.
## Notes ## Notes
- The two `deploy/migrations/*.sql` files are superseded by Alembic on Postgres - The two `deploy/migrations/*.sql` files are superseded by Alembic on Postgres
(they were SQLite-specific `ALTER TABLE` column-adds for the old prod DB). (they were SQLite-specific `ALTER TABLE` column-adds for the old prod DB).
- The PG18 image's data dir is the versioned `…/18/docker` subdir; the compose - The TimescaleDB image's data dir is `/var/lib/postgresql/data`; the compose
volume is mounted at the parent `/var/lib/postgresql` so it persists without the volume is mounted at the parent `/var/lib/postgresql` so it persists without the
initdb permission issue that pinning `PGDATA` to the mountpoint can trigger. initdb permission issue that pinning `PGDATA` to the mountpoint can trigger.
- Climate falls back to the parquet cache whenever `THERMOGRAPH_DATABASE_URL` is
not a Postgres URL (dev, tests, offline tooling) — the same dialect switch as the
accounts DB, so a DB outage in prod degrades to upstream refetch, never a 500.

25
deploy/db/Dockerfile.db
View file

@ -1,25 +0,0 @@
# Thermograph database image: PostgreSQL 18 with pg_duckdb.
#
# pg_duckdb embeds DuckDB inside Postgres, which lets the DB container read the
# app's Parquet climate cache directly for ad-hoc analytics:
#
# SELECT * FROM read_parquet('/parquet/1026_-2857.parquet');
#
# Why FROM the official pg_duckdb image instead of `FROM postgres:18` + compile:
# pg_duckdb links a full DuckDB build, so compiling it from source in this
# Dockerfile would mean pulling the DuckDB toolchain and a long, fragile build.
# The maintainers (duckdb/pg_duckdb, MotherDuck) publish an official PG18 image
# that is genuine PostgreSQL 18.1 on Debian 12 bookworm -- the exact same base
# as the official `postgres:18` image, using the standard postgres
# docker-entrypoint.sh. So POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB /
# PGDATA / /docker-entrypoint-initdb.d / pg_isready all behave identically to
# `postgres:18`; this is a drop-in replacement for the compose `db` service.
#
# Pinned to a specific patch tag (not 18-main) for reproducible builds.
FROM pgduckdb/pgduckdb:18-v1.1.1
# Bake the parquet init script so the image enables the extension on first init
# even without the compose bind mount. It is CREATE EXTENSION IF NOT EXISTS, so
# it is idempotent with the base image's own 0001-install-pg_duckdb.sql.
# Build context is the repo root (see deploy/db/README.md for the compose snippet).
COPY deploy/db/init/ /docker-entrypoint-initdb.d/

198
deploy/db/README.md
View file

@ -1,171 +1,87 @@
# Thermograph DB image — Postgres 18 + parquet reads # Thermograph DB — TimescaleDB (PostgreSQL 18)
The `db` service can read the app's Parquet climate cache The `db` service runs the stock **`timescale/timescaledb:latest-pg18`** image
(`data/cache/*.parquet`) directly from SQL, for ad-hoc analytics, via (TimescaleDB 2.24+, genuine PostgreSQL 18). The app's climate record lives here in
**pg_duckdb** — DuckDB embedded inside Postgres. **hypertables** — the DB, not the filesystem, is the source of truth:
```sql - **`climate_history`** — a hypertable of the full daily archive per grid cell
SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin (`cell_id, date, tmax, tmin, precip, wind, gust, humid, fmax, fmin, feels`), back
FROM read_parquet('/parquet/1026_-2857.parquet') r to 1980. Durable/LOGGED (a 45-year, rate-limited refetch is expensive),
ORDER BY r['date'] LIMIT 5; range-partitioned on `date` (5-year chunks), compressed for chunks older than a
``` year.
- **`climate_recent`** — the recent-observations + forward-forecast bundle (a plain
table: it holds future dates and is rewritten hourly).
- **`climate_sync`** — per-cell freshness (epoch seconds) that replaces the old
parquet file mtimes: it drives the hourly history top-up, the 1-hour forecast
TTL, and the `recent_stamp` token embedded in derived-payload validity.
## Chosen extension: pg_duckdb — and why The schema is created by Alembic (`backend/alembic/versions/0002_climate_hypertables.py`,
run at app boot via `deploy/entrypoint.sh`). The app reads/writes it through
`backend/data/climate_store.py` (psycopg + polars). See
`deploy/POSTGRES-MIGRATION.md` for the parquet→hypertable cutover.
| Option | PG18? | Fit | Verdict | ## Why the stock image (no custom Dockerfile)
| --- | --- | --- | --- |
| **pg_duckdb** (duckdb / MotherDuck) | **Yes** — official image `pgduckdb/pgduckdb:18-v1.1.1` is genuine PG18.1 | `read_parquet('…')` in plain SQL; globs, `union_by_name`, full DuckDB analytics engine | **Chosen** |
| pg_parquet (Crunchy Data) | Yes (14–18) | `COPY … TO/FROM '…' (format 'parquet')` — import/export, not a query engine | Viable, but COPY-oriented; no standalone official image (ships via Crunchy Bridge/CPK), so it'd need a Rust/pgrx source build |
| parquet_fdw | No prebuilt PG18 support; low activity | Foreign tables over parquet | Rejected — oldest, weakest PG18 story |
pg_duckdb wins for the stated goal (ad-hoc analytics): it exposes DuckDB's The previous DB image was a custom `pgduckdb/pgduckdb:18` build whose only purpose
`read_parquet` directly in SQL, so you query cache files like tables — no was ad-hoc `read_parquet()` over the parquet cache. Now the climate record is in
import step, no foreign-table DDL, and you get aggregation/joins/globs across real tables, so that capability is gone and the DB is the **stock TimescaleDB
many cells at once. image** — no build step. The image already sets
`shared_preload_libraries=timescaledb`; never `ALTER SYSTEM SET
## PG-version reality for PG18 (empirically verified 2026-07-19) shared_preload_libraries` (it would land in `postgresql.auto.conf` and override the
image's preload).
**No version delta.** PG18 support is real, not a fallback. The pulled image
reports:
```
PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2) on x86_64-pc-linux-gnu
pg_extension: pg_duckdb 1.1.0
shared_preload_libraries: pg_duckdb
```
The image is built on Debian 12 bookworm — the **same base as the official
`postgres:18` image** — and uses the standard `docker-entrypoint.sh`. So
`POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` / `PGDATA` /
`/docker-entrypoint-initdb.d` / `pg_isready` all behave exactly as with
`postgres:18`. It is a drop-in replacement for the `db` service; nothing else
in the stack changes.
We `FROM` the official pg_duckdb image (pinned to `18-v1.1.1`, not `18-main`)
rather than `FROM postgres:18` + compile, because pg_duckdb links a full DuckDB
build — compiling from source in the Dockerfile means the DuckDB toolchain and
a long, fragile build for no benefit over the maintainers' official PG18 image.
## Files here ## Files here
- **`Dockerfile.db`** — `FROM pgduckdb/pgduckdb:18-v1.1.1`, plus `COPY` of the - **`init/10-timescaledb.sql`** — `CREATE EXTENSION IF NOT EXISTS timescaledb;`
init script so the image enables the extension on first init even without the (runs from `/docker-entrypoint-initdb.d` on first cluster init; Alembic also does
compose bind mount. this idempotently at boot).
- **`init/10-parquet.sql`** — `CREATE EXTENSION IF NOT EXISTS pg_duckdb;` (runs - **`init/20-tuning.sh`** — scales `shared_buffers` (25%), `effective_cache_size`
from `/docker-entrypoint-initdb.d` on first cluster init). (75%), `work_mem`, and `maintenance_work_mem` from `DB_MEMORY` via `ALTER SYSTEM`.
## Compose snippet to merge into the `db` service ## Compose `db` service
Replace `image: postgres:18` with the `build:` block; add the read-only parquet
bind and the init mount. Everything else in the `db` service stays as-is.
```yaml ```yaml
db: db:
# image: postgres:18 # <- remove; build the parquet-capable image image: timescale/timescaledb:latest-pg18
build:
context: .
dockerfile: deploy/db/Dockerfile.db
environment: environment:
POSTGRES_USER: thermograph POSTGRES_USER: thermograph
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}
POSTGRES_DB: thermograph POSTGRES_DB: thermograph
PGDATA: /var/lib/postgresql/data DB_MEMORY: ${DB_MEMORY:-8g}
volumes: volumes:
- pgdata:/var/lib/postgresql/data - pgdata:/var/lib/postgresql
- ./data/cache:/parquet:ro # read-only parquet cache
- ./deploy/db/init:/docker-entrypoint-initdb.d - ./deploy/db/init:/docker-entrypoint-initdb.d
# healthcheck / cpus / deploy / restart: unchanged # healthcheck / cpus / mem_limit / shm_size: unchanged
``` ```
Notes: Notes:
- The bind mount at `/docker-entrypoint-initdb.d` **replaces** the base image's - The volume is mounted at the **parent** of the data dir; the image picks its own
own init scripts (its `0001-install-pg_duckdb.sql` and the MotherDuck-only PGDATA subdir (`/var/lib/postgresql/data`) under it. The whole tree persists on
`0002-enable-md-pg_duckdb.sql`). That's intended: our `10-parquet.sql` still the named volume.
runs `CREATE EXTENSION`, and we don't use MotherDuck. If you prefer to keep - No host port on purpose: the app reaches Postgres as `db:5432` on the compose
the image's baked scripts, drop the `:/docker-entrypoint-initdb.d` line — the network. Nothing outside the stack should touch the database.
`Dockerfile` already bakes `10-parquet.sql` in. - Init scripts only run when PGDATA is empty. On an **existing** database, enable
- `:ro` keeps the DB from ever mutating the app's cache. In prod the app writes the extension once by hand:
the cache to the `appdata` volume; point this bind at wherever that lives on
the host if you want the DB to see the live cache rather than the repo copy.
- Init scripts only run when `PGDATA` is empty. On an **existing** database,
enable it once by hand:
``` ```
docker compose exec db psql -U thermograph -d thermograph \ docker compose exec db psql -U thermograph -d thermograph \
-c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;' -c 'CREATE EXTENSION IF NOT EXISTS timescaledb;'
``` ```
## Usage — real cache file at `/parquet/…` ## Inspecting the hypertable
Cache columns: `date, tmax, tmin, precip, wind, gust, humid, fmax, fmin, feels`.
pg_duckdb ≥ 0.3 uses the `r['colname']` subscript syntax (not
`AS t(col type, …)`).
```sql ```sql
-- one cell, first rows -- Chunk / compression overview
SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin, r['precip'] AS precip SELECT hypertable_name, num_chunks, compression_enabled
FROM read_parquet('/parquet/1026_-2857.parquet') r FROM timescaledb_information.hypertables;
ORDER BY r['date'] LIMIT 5;
-- per-year analytics over one cell -- One cell, most recent archived days
SELECT EXTRACT(YEAR FROM r['date']::timestamp) AS yr, SELECT date, tmax, tmin, precip
ROUND(AVG(r['tmax'])::numeric, 1) AS avg_tmax, FROM climate_history WHERE cell_id = '1026_-2857'
MAX(r['tmax']) AS record_high ORDER BY date DESC LIMIT 5;
FROM read_parquet('/parquet/1026_-2857.parquet') r
WHERE r['date'] >= '2020-01-01' -- Per-year highs for one cell
SELECT EXTRACT(YEAR FROM date) AS yr,
ROUND(AVG(tmax)::numeric, 1) AS avg_tmax, MAX(tmax) AS record_high
FROM climate_history WHERE cell_id = '1026_-2857' AND date >= '2020-01-01'
GROUP BY yr ORDER BY yr; GROUP BY yr ORDER BY yr;
-- glob across every cached cell (union_by_name handles the _rf / _forecast
-- files whose column sets differ)
SELECT COUNT(*) FROM read_parquet('/parquet/*.parquet', union_by_name := true) r;
```
## Proof of work (verified 2026-07-19)
Built `deploy/db/Dockerfile.db`, ran a throwaway container with the real
`data/cache` bind-mounted read-only at `/parquet`, then:
```
$ psql -c "SELECT version();"
PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2) on x86_64-pc-linux-gnu ...
$ psql -c "SELECT extname, extversion FROM pg_extension WHERE extname='pg_duckdb';"
extname | extversion
-----------+------------
pg_duckdb | 1.1.0
$ psql -c "SELECT COUNT(*) FROM read_parquet('/parquet/1026_-2857.parquet') r;"
row_count
-----------
16982
$ psql -c "SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin,
r['precip'] AS precip, r['humid'] AS humid
FROM read_parquet('/parquet/1026_-2857.parquet') r
ORDER BY r['date'] LIMIT 5;"
date | tmax | tmin | precip | humid
---------------------+------+------+--------+-------
1980-01-01 00:00:00 | 56.2 | 34.7 | 0 | 73
1980-01-02 00:00:00 | 63.8 | 38 | 0 | 79
1980-01-03 00:00:00 | 60.1 | 46.1 | 0.315 | 83
1980-01-04 00:00:00 | 51.7 | 40 | 0 | 68
1980-01-05 00:00:00 | 56.5 | 33.7 | 0 | 78
$ psql -c "SELECT EXTRACT(YEAR FROM r['date']::timestamp) AS yr,
ROUND(AVG(r['tmax'])::numeric,1) AS avg_tmax, MAX(r['tmax']) AS record_high
FROM read_parquet('/parquet/1026_-2857.parquet') r
WHERE r['date'] >= '2020-01-01' GROUP BY yr ORDER BY yr;"
yr | avg_tmax | record_high
------+----------+-------------
2020 | 78.8 | 99
2021 | 77.0 | 93
2022 | 79.2 | 101.4
2023 | 80.8 | 107.1
2024 | 79.8 | 97.5
2025 | 80.0 | 99.4
2026 | 77.8 | 95.6
$ psql -c "SELECT COUNT(*) FROM read_parquet('/parquet/*.parquet', union_by_name := true) r;"
count
--------
525421 -- rows across all 45 cached cells
``` ```

15
deploy/db/init/10-parquet.sql
View file

@ -1,15 +0,0 @@
-- Enable pg_duckdb so the database can read the app's Parquet climate cache
-- directly (ad-hoc analytics via DuckDB's read_parquet / read_csv / etc.).
--
-- This runs once, on first cluster init (empty PGDATA), from
-- /docker-entrypoint-initdb.d. Because the compose `db` service keeps its data
-- on a persistent named volume, init scripts do NOT re-run on an existing
-- database -- to enable pg_duckdb on a DB that was created before this image,
-- run it by hand:
--
-- docker compose exec db psql -U thermograph -d thermograph \
-- -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;'
--
-- pg_duckdb requires shared_preload_libraries='pg_duckdb'; the base image's
-- postgresql.conf.sample already sets it, so it is active on fresh init.
CREATE EXTENSION IF NOT EXISTS pg_duckdb;

14
deploy/db/init/10-timescaledb.sql Normal file
View file

@ -0,0 +1,14 @@
-- Enable TimescaleDB so the app's climate record lives in hypertables
-- (climate_history) instead of parquet files. The stock timescale/timescaledb
-- image already preloads the extension (shared_preload_libraries=timescaledb);
-- this just runs CREATE EXTENSION on first cluster init.
--
-- This runs once, on first cluster init (empty PGDATA), from
-- /docker-entrypoint-initdb.d. Because the compose `db` service keeps its data on
-- a persistent named volume, init scripts do NOT re-run on an existing database.
-- Alembic (backend/alembic) also runs `CREATE EXTENSION IF NOT EXISTS timescaledb`
-- at app boot, so an existing volume gets it there; to enable it by hand instead:
--
-- docker compose exec db psql -U thermograph -d thermograph \
-- -c 'CREATE EXTENSION IF NOT EXISTS timescaledb;'
CREATE EXTENSION IF NOT EXISTS timescaledb;

18
deploy/db/init/20-tuning.sh
View file

@ -2,9 +2,12 @@
# Postgres memory / performance tuning, scaled to the container's DB_MEMORY budget so # Postgres memory / performance tuning, scaled to the container's DB_MEMORY budget so
# the same init serves every host (beta 8g; prod 16g on the 48 GB box) with no # the same init serves every host (beta 8g; prod 16g on the 48 GB box) with no
# hardcoding. Runs once on a fresh data volume from /docker-entrypoint-initdb.d, after # hardcoding. Runs once on a fresh data volume from /docker-entrypoint-initdb.d, after
# 10-parquet.sql enables pg_duckdb. Settings are written via ALTER SYSTEM (persisted to # 10-timescaledb.sql enables timescaledb. Settings are written via ALTER SYSTEM
# postgresql.auto.conf); the container's post-init restart brings restart-only settings # (persisted to postgresql.auto.conf, which the timescaledb image's own
# (shared_buffers, …) into effect. Init scripts do NOT re-run on an existing volume — to # timescaledb-tune postgresql.conf defers to); the container's post-init restart brings
# restart-only settings (shared_buffers, …) into effect. NB: never ALTER SYSTEM SET
# shared_preload_libraries here — that would land in auto.conf and override the image's
# `timescaledb` preload. Init scripts do NOT re-run on an existing volume — to
# re-tune later, set DB_MEMORY and run this by hand, then restart: # re-tune later, set DB_MEMORY and run this by hand, then restart:
# docker compose exec -e DB_MEMORY=16g db bash /docker-entrypoint-initdb.d/20-tuning.sh # docker compose exec -e DB_MEMORY=16g db bash /docker-entrypoint-initdb.d/20-tuning.sh
# docker compose restart db # docker compose restart db
@ -23,18 +26,17 @@ esac
if [ "$mb" -lt 1024 ]; then mb=8192; fi if [ "$mb" -lt 1024 ]; then mb=8192; fi
# Derive settings from the budget. The ratios reproduce the historical 8 GB tuning # Derive settings from the budget. The ratios reproduce the historical 8 GB tuning
# (shared_buffers 2 GB, effective_cache_size 6 GB, duckdb 4 GB, work_mem 64 MB, # (shared_buffers 2 GB, effective_cache_size 6 GB, work_mem 64 MB,
# maintenance_work_mem 512 MB) and scale linearly on a bigger box. # maintenance_work_mem 512 MB) and scale linearly on a bigger box.
shared_buffers=$((mb / 4)) # 25% — the shared page cache shared_buffers=$((mb / 4)) # 25% — the shared page cache
effective_cache=$((mb * 3 / 4)) # 75% — planner's view of total cache (PG + OS) effective_cache=$((mb * 3 / 4)) # 75% — planner's view of total cache (PG + OS)
duckdb_mem=$((mb / 2)) # 50% — pg_duckdb ceiling for parquet processing
work_mem=$((mb / 128)) # ~64 MB at 8 GB (per-operation; kept modest) work_mem=$((mb / 128)) # ~64 MB at 8 GB (per-operation; kept modest)
if [ "$work_mem" -lt 16 ]; then work_mem=16; fi if [ "$work_mem" -lt 16 ]; then work_mem=16; fi
maint_mem=$((mb / 16)) # 512 MB at 8 GB — index builds / VACUUM maint_mem=$((mb / 16)) # 512 MB at 8 GB — index builds / VACUUM
echo "[tuning] DB_MEMORY=${budget} -> ${mb}MB: shared_buffers=${shared_buffers}MB" \ echo "[tuning] DB_MEMORY=${budget} -> ${mb}MB: shared_buffers=${shared_buffers}MB" \
"effective_cache_size=${effective_cache}MB work_mem=${work_mem}MB" \ "effective_cache_size=${effective_cache}MB work_mem=${work_mem}MB" \
"maintenance_work_mem=${maint_mem}MB duckdb.max_memory=${duckdb_mem}MB" "maintenance_work_mem=${maint_mem}MB"
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<SQL psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<SQL
-- Caching: the shared page cache, and the planner's view of total cache (PG + OS). -- Caching: the shared page cache, and the planner's view of total cache (PG + OS).
@ -57,8 +59,4 @@ ALTER SYSTEM SET effective_io_concurrency = 200;
-- Room for parallel scans/aggregates on the bigger analytic queries. -- Room for parallel scans/aggregates on the bigger analytic queries.
ALTER SYSTEM SET max_parallel_workers_per_gather = 2; ALTER SYSTEM SET max_parallel_workers_per_gather = 2;
-- DuckDB (pg_duckdb) memory ceiling for parquet processing. pg_duckdb is preloaded
-- (shared_preload_libraries), so this GUC exists at ALTER SYSTEM time.
ALTER SYSTEM SET duckdb.max_memory = '${duckdb_mem}MB';
SQL SQL

2
docker-compose.openmeteo.yml
View file

@ -9,7 +9,7 @@
# containers at that mount; it defaults to ./data/om-archive so a local smoke test # containers at that mount; it defaults to ./data/om-archive so a local smoke test
# works against a plain directory. Object storage is never bind-mounted into the app # works against a plain directory. Object storage is never bind-mounted into the app
# — only Open-Meteo reads it; the app just talks HTTP to open-meteo-api and keeps its # — only Open-Meteo reads it; the app just talks HTTP to open-meteo-api and keeps its
# own small daily-per-cell parquet cache. # own daily-per-cell climate record in the TimescaleDB `db` service.
# #
# One-time backfill (writes ~1–1.5 TB of .om to object storage — run once before the # One-time backfill (writes ~1–1.5 TB of .om to object storage — run once before the
# app is flipped over): `make om-backfill`. # app is flipped over): `make om-backfill`.

34
docker-compose.yml
View file

@ -1,4 +1,5 @@
# Thermograph production stack: the FastAPI app plus its PostgreSQL 18 database. # Thermograph production stack: the FastAPI app plus its TimescaleDB (PostgreSQL 18)
# database.
# #
# docker compose up -d --build # or: make up # docker compose up -d --build # or: make up
# #
@ -10,13 +11,13 @@
services: services:
db: db:
# PostgreSQL 18 + pg_duckdb (deploy/db/Dockerfile.db, FROM pgduckdb/pgduckdb:18-…, # TimescaleDB on PostgreSQL 18 (the stock image already sets
# which IS PostgreSQL 18). Lets the DB read the app's parquet climate cache # shared_preload_libraries=timescaledb). The app's climate record — the full
# directly, e.g. SELECT * FROM read_parquet('/parquet/cache/*.parquet'). The init # daily archive and the hourly recent+forecast bundle — lives in hypertables
# script CREATE EXTENSIONs pg_duckdb on a fresh volume. # here (see backend/data/climate_store.py), so the DB, not the filesystem, is the
build: # source of truth. The init script CREATE EXTENSIONs timescaledb on a fresh volume
context: . # (Alembic also does, idempotently, at app boot).
dockerfile: deploy/db/Dockerfile.db image: timescale/timescaledb:latest-pg18
environment: environment:
POSTGRES_USER: thermograph POSTGRES_USER: thermograph
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}
@ -26,14 +27,11 @@ services:
# per host: Terraform sets DB_MEMORY (prod 16g), local/beta default 8g. # per host: Terraform sets DB_MEMORY (prod 16g), local/beta default 8g.
DB_MEMORY: ${DB_MEMORY:-8g} DB_MEMORY: ${DB_MEMORY:-8g}
volumes: volumes:
# Mount the volume at the PARENT of the data dir and let the PG18 image use # Mount the volume at the PARENT of the data dir and let the image pick its own
# its own versioned subdir (/var/lib/postgresql/18/docker). Pinning PGDATA # PGDATA subdir under it (the timescaledb image defaults to
# directly at the mountpoint trips an initdb chmod on some Docker setups; the # /var/lib/postgresql/data). Pinning PGDATA directly at the mountpoint trips an
# whole tree still persists on the named volume this way. # initdb chmod on some Docker setups; the whole tree still persists this way.
- pgdata:/var/lib/postgresql - pgdata:/var/lib/postgresql
# Read the app's live parquet cache (the same appdata volume the app writes,
# mounted read-only) — query it under /parquet/cache/*.parquet.
- appdata:/parquet:ro
- ./deploy/db/init:/docker-entrypoint-initdb.d - ./deploy/db/init:/docker-entrypoint-initdb.d
healthcheck: healthcheck:
test: ["CMD-SHELL", "pg_isready -U thermograph -d thermograph"] test: ["CMD-SHELL", "pg_isready -U thermograph -d thermograph"]
@ -42,9 +40,9 @@ services:
retries: 10 retries: 10
# Give Postgres room to cache + process. mem_limit is the hard ceiling; the actual # Give Postgres room to cache + process. mem_limit is the hard ceiling; the actual
# budget is tuned in deploy/db/init/20-tuning.sh, which scales shared_buffers (25%), # budget is tuned in deploy/db/init/20-tuning.sh, which scales shared_buffers (25%),
# effective_cache_size (75%), work_mem, maintenance_work_mem and duckdb.max_memory # effective_cache_size (75%), work_mem and maintenance_work_mem from DB_MEMORY — so
# (50%) from DB_MEMORY — so raising DB_MEMORY raises both the cap and the tuning # raising DB_MEMORY raises both the cap and the tuning together. shm_size backs
# together. shm_size backs parallel-query shared memory (the 64MB docker default is # parallel-query shared memory (the 64MB docker default is
# too small once shared_buffers/parallelism grow). # too small once shared_buffers/parallelism grow).
# Sized via env (Terraform sets DB_CPUS/DB_MEMORY per host); defaults match the # Sized via env (Terraform sets DB_CPUS/DB_MEMORY per host); defaults match the
# historical 2 CPU / 8 GB budget so a plain `docker compose up` is unchanged. # historical 2 CPU / 8 GB budget so a plain `docker compose up` is unchanged.

8
terraform/README.md
View file

@ -42,11 +42,11 @@ A change to the rendered env, the compose files, the branch, or the sizing flips
environment (defaults `4 / 4 / 2 / 8g`, identical to before). Terraform sets them per environment (defaults `4 / 4 / 2 / 8g`, identical to before). Terraform sets them per
host through `/etc/thermograph.env`, so the big prod box can run larger caps without a host through `/etc/thermograph.env`, so the big prod box can run larger caps without a
compose edit. The Postgres *internal* memory budget (`shared_buffers`, compose edit. The Postgres *internal* memory budget (`shared_buffers`,
`effective_cache_size`, `work_mem`, `duckdb.max_memory`) is derived from the same `effective_cache_size`, `work_mem`, `maintenance_work_mem`) is derived from the same
`DB_MEMORY` by `deploy/db/init/20-tuning.sh` — so raising `db_memory` scales the `DB_MEMORY` by `deploy/db/init/20-tuning.sh` — so raising `db_memory` scales the
container cap and the tuning together (prod 16g → shared_buffers 4 GB, duckdb 8 GB). The container cap and the tuning together (prod 16g → shared_buffers 4 GB). The tuning
tuning applies on a fresh DB volume; on an existing volume re-run it by hand (see the applies on a fresh DB volume; on an existing volume re-run it by hand (see the script
script header). header).
## Prerequisites ## Prerequisites