Move the climate record from parquet to TimescaleDB hypertables (#227)
Replace the per-cell parquet cache with TimescaleDB hypertables as the production backend for the raw daily climate record, and drop pg_duckdb. Parquet stays the backend whenever THERMOGRAPH_DATABASE_URL is not a Postgres URL (dev, tests, offline tooling), the same dialect switch the accounts DB and derived store already use, so CI stays Postgres-free. - data/climate_store.py: psycopg + polars bridge over climate_history (a hypertable), climate_recent, and climate_sync (per-cell freshness). Reads via pl.read_database, writes via COPY + ON CONFLICT upsert; fail-soft to a cache miss so a DB hiccup degrades to upstream refetch. - data/climate.py: route every cache/mtime touchpoint through a backend dispatch. recent_stamp becomes int(recent_synced_at) on Postgres; the stale-serve path still avoids bumping it, so derived-payload tokens invalidate on exactly the same events as before. - alembic 0002: CREATE EXTENSION timescaledb plus the hypertable schema (compression policy on year-old chunks), guarded to no-op off Postgres. - migrate_cache_to_pg.py (make migrate-cache): idempotent backfill of the parquet cache into the hypertables, preserving file mtimes as sync timestamps so recent_stamp is unchanged across cutover. - db image -> stock timescale/timescaledb:latest-pg18; drop the custom pg_duckdb Dockerfile, the read-only /parquet mount, and the duckdb tuning GUC. Docs updated for the new backend and cutover. Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Emi Griffith
GitHub
parent
dc4ee9a8db
commit
1f7e8552cf
9 changed files with 154 additions and 221 deletions
|
|
@ -11,13 +11,19 @@ deploy to the compose stack, migrating the authoritative accounts data.
|
|||
tables. Managed by Alembic (`backend/alembic/`); `create_all` on a fresh DB.
|
||||
- **derived cache** (`store.py`) and **metrics** (`metrics.py`): now Postgres
|
||||
**UNLOGGED** tables (fast, non-durable — same throwaway semantics). No data to
|
||||
migrate — the cache rebuilds from parquet, metrics start empty.
|
||||
migrate — the cache rebuilds, metrics start empty.
|
||||
- **climate record** (`climate.py`): the raw daily archive + recent/forecast bundle
|
||||
moved from per-cell **parquet** files into **TimescaleDB hypertables**
|
||||
(`climate_history` / `climate_recent` / `climate_sync`, managed by Alembic). The
|
||||
app reads/writes them via `data/climate_store.py`. Backfilled from the existing
|
||||
parquet cache with `migrate_cache_to_pg.py` (see below).
|
||||
- The app selects Postgres when `THERMOGRAPH_DATABASE_URL` is a `postgresql+asyncpg`
|
||||
URL; unset ⇒ the old SQLite behavior (this is how the **test suite** stays on
|
||||
SQLite — no Postgres needed in CI).
|
||||
- `db` runs **pg_duckdb** (`deploy/db/Dockerfile.db`, genuine PG18): the app's
|
||||
parquet cache is mounted read-only at `/parquet`, so you can query it directly,
|
||||
e.g. `SELECT r['date'], r['tmax'] FROM read_parquet('/parquet/cache/*.parquet') r`.
|
||||
URL; unset ⇒ the old SQLite/parquet behavior (this is how the **test suite** stays
|
||||
Postgres-free — no Postgres needed in CI, and climate falls back to parquet).
|
||||
- `db` runs the stock **`timescale/timescaledb:latest-pg18`** image (genuine PG18 +
|
||||
TimescaleDB). The earlier pg_duckdb / `read_parquet('/parquet/…')` capability is
|
||||
gone — the climate record is real tables now, queryable directly (see
|
||||
`deploy/db/README.md`).
|
||||
|
||||
## Connection model
|
||||
|
||||
|
|
@ -64,10 +70,51 @@ unset/SQLite and restart). **Clean only during/immediately after the window**
|
|||
real users write to Postgres, those writes are lost on rollback (the copy is
|
||||
one-way), so keep the window short and writes frozen during the copy.
|
||||
|
||||
## TimescaleDB cutover (from the earlier pg_duckdb PG18 stack)
|
||||
|
||||
Swapping the `db` image from `pgduckdb/pgduckdb:18` to
|
||||
`timescale/timescaledb:latest-pg18` means a **fresh PGDATA volume** — and that
|
||||
volume also holds the durable **accounts** data. So this cutover is not a plain
|
||||
`docker compose pull`; it re-seeds both accounts and the climate record.
|
||||
|
||||
1. **Back up first.** `pg_dump` the accounts tables from the *old* db container
|
||||
(`docker compose exec db pg_dump -U thermograph -t user -t subscription \
|
||||
-t notification -t push_subscription -t pending_digest thermograph > accounts.sql`),
|
||||
or keep the pre-Postgres `accounts.sqlite` as the source of truth.
|
||||
2. **Replace the DB.** Deploy this branch (compose now points at the stock
|
||||
TimescaleDB image, no `build:`). Bring down the stack and remove the old volume
|
||||
so PGDATA re-inits on the new image: `docker compose down && docker volume rm
|
||||
<project>_pgdata`.
|
||||
3. **Build the schema.** `docker compose up -d db`, then
|
||||
`docker compose run --rm app alembic upgrade head` — this creates the accounts
|
||||
tables *and* the climate hypertables, and `CREATE EXTENSION timescaledb`.
|
||||
4. **Restore accounts.** Either `psql < accounts.sql`, or re-run
|
||||
`migrate_accounts_to_pg.py --sqlite <accounts.sqlite>` (§ the SQLite cutover
|
||||
above). Verify per-table counts.
|
||||
5. **Backfill the climate record** from the parquet cache (mount it and run the
|
||||
backfill; idempotent, resumable):
|
||||
```
|
||||
docker compose run --rm \
|
||||
-v /opt/thermograph/data/cache:/app/data/cache:ro \
|
||||
app python migrate_cache_to_pg.py
|
||||
```
|
||||
It loads `climate_history` + `climate_recent` and sets each cell's
|
||||
`climate_sync` to the parquet **file mtime**, so `recent_stamp` (hence every
|
||||
derived-payload token) is unchanged across the cutover. Cells without a
|
||||
schema-complete parquet record are skipped and refetch lazily on first request.
|
||||
6. **Start serving:** `docker compose up -d`. Smoke test: login, a subscription,
|
||||
a cached calendar/day request (served from `climate_history`), the metrics
|
||||
dashboard.
|
||||
7. Keep the pre-cutover DB backup (accounts dump + parquet cache) for a few days.
|
||||
The climate record is now **durable** — make sure `pg_dump`/PITR covers it.
|
||||
|
||||
## Notes
|
||||
|
||||
- The two `deploy/migrations/*.sql` files are superseded by Alembic on Postgres
|
||||
(they were SQLite-specific `ALTER TABLE` column-adds for the old prod DB).
|
||||
- The PG18 image's data dir is the versioned `…/18/docker` subdir; the compose
|
||||
- The TimescaleDB image's data dir is `/var/lib/postgresql/data`; the compose
|
||||
volume is mounted at the parent `/var/lib/postgresql` so it persists without the
|
||||
initdb permission issue that pinning `PGDATA` to the mountpoint can trigger.
|
||||
- Climate falls back to the parquet cache whenever `THERMOGRAPH_DATABASE_URL` is
|
||||
not a Postgres URL (dev, tests, offline tooling) — the same dialect switch as the
|
||||
accounts DB, so a DB outage in prod degrades to upstream refetch, never a 500.
|
||||
|
|
|
|||
|
|
@ -1,25 +0,0 @@
|
|||
# Thermograph database image: PostgreSQL 18 with pg_duckdb.
|
||||
#
|
||||
# pg_duckdb embeds DuckDB inside Postgres, which lets the DB container read the
|
||||
# app's Parquet climate cache directly for ad-hoc analytics:
|
||||
#
|
||||
# SELECT * FROM read_parquet('/parquet/1026_-2857.parquet');
|
||||
#
|
||||
# Why FROM the official pg_duckdb image instead of `FROM postgres:18` + compile:
|
||||
# pg_duckdb links a full DuckDB build, so compiling it from source in this
|
||||
# Dockerfile would mean pulling the DuckDB toolchain and a long, fragile build.
|
||||
# The maintainers (duckdb/pg_duckdb, MotherDuck) publish an official PG18 image
|
||||
# that is genuine PostgreSQL 18.1 on Debian 12 bookworm -- the exact same base
|
||||
# as the official `postgres:18` image, using the standard postgres
|
||||
# docker-entrypoint.sh. So POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB /
|
||||
# PGDATA / /docker-entrypoint-initdb.d / pg_isready all behave identically to
|
||||
# `postgres:18`; this is a drop-in replacement for the compose `db` service.
|
||||
#
|
||||
# Pinned to a specific patch tag (not 18-main) for reproducible builds.
|
||||
FROM pgduckdb/pgduckdb:18-v1.1.1
|
||||
|
||||
# Bake the parquet init script so the image enables the extension on first init
|
||||
# even without the compose bind mount. It is CREATE EXTENSION IF NOT EXISTS, so
|
||||
# it is idempotent with the base image's own 0001-install-pg_duckdb.sql.
|
||||
# Build context is the repo root (see deploy/db/README.md for the compose snippet).
|
||||
COPY deploy/db/init/ /docker-entrypoint-initdb.d/
|
||||
|
|
@ -1,171 +1,87 @@
|
|||
# Thermograph DB image — Postgres 18 + parquet reads
|
||||
# Thermograph DB — TimescaleDB (PostgreSQL 18)
|
||||
|
||||
The `db` service can read the app's Parquet climate cache
|
||||
(`data/cache/*.parquet`) directly from SQL, for ad-hoc analytics, via
|
||||
**pg_duckdb** — DuckDB embedded inside Postgres.
|
||||
The `db` service runs the stock **`timescale/timescaledb:latest-pg18`** image
|
||||
(TimescaleDB 2.24+, genuine PostgreSQL 18). The app's climate record lives here in
|
||||
**hypertables** — the DB, not the filesystem, is the source of truth:
|
||||
|
||||
```sql
|
||||
SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin
|
||||
FROM read_parquet('/parquet/1026_-2857.parquet') r
|
||||
ORDER BY r['date'] LIMIT 5;
|
||||
```
|
||||
- **`climate_history`** — a hypertable of the full daily archive per grid cell
|
||||
(`cell_id, date, tmax, tmin, precip, wind, gust, humid, fmax, fmin, feels`), back
|
||||
to 1980. Durable/LOGGED (a 45-year, rate-limited refetch is expensive),
|
||||
range-partitioned on `date` (5-year chunks), compressed for chunks older than a
|
||||
year.
|
||||
- **`climate_recent`** — the recent-observations + forward-forecast bundle (a plain
|
||||
table: it holds future dates and is rewritten hourly).
|
||||
- **`climate_sync`** — per-cell freshness (epoch seconds) that replaces the old
|
||||
parquet file mtimes: it drives the hourly history top-up, the 1-hour forecast
|
||||
TTL, and the `recent_stamp` token embedded in derived-payload validity.
|
||||
|
||||
## Chosen extension: pg_duckdb — and why
|
||||
The schema is created by Alembic (`backend/alembic/versions/0002_climate_hypertables.py`,
|
||||
run at app boot via `deploy/entrypoint.sh`). The app reads/writes it through
|
||||
`backend/data/climate_store.py` (psycopg + polars). See
|
||||
`deploy/POSTGRES-MIGRATION.md` for the parquet→hypertable cutover.
|
||||
|
||||
| Option | PG18? | Fit | Verdict |
|
||||
| --- | --- | --- | --- |
|
||||
| **pg_duckdb** (duckdb / MotherDuck) | **Yes** — official image `pgduckdb/pgduckdb:18-v1.1.1` is genuine PG18.1 | `read_parquet('…')` in plain SQL; globs, `union_by_name`, full DuckDB analytics engine | **Chosen** |
|
||||
| pg_parquet (Crunchy Data) | Yes (14–18) | `COPY … TO/FROM '…' (format 'parquet')` — import/export, not a query engine | Viable, but COPY-oriented; no standalone official image (ships via Crunchy Bridge/CPK), so it'd need a Rust/pgrx source build |
|
||||
| parquet_fdw | No prebuilt PG18 support; low activity | Foreign tables over parquet | Rejected — oldest, weakest PG18 story |
|
||||
## Why the stock image (no custom Dockerfile)
|
||||
|
||||
pg_duckdb wins for the stated goal (ad-hoc analytics): it exposes DuckDB's
|
||||
`read_parquet` directly in SQL, so you query cache files like tables — no
|
||||
import step, no foreign-table DDL, and you get aggregation/joins/globs across
|
||||
many cells at once.
|
||||
|
||||
## PG-version reality for PG18 (empirically verified 2026-07-19)
|
||||
|
||||
**No version delta.** PG18 support is real, not a fallback. The pulled image
|
||||
reports:
|
||||
|
||||
```
|
||||
PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2) on x86_64-pc-linux-gnu
|
||||
pg_extension: pg_duckdb 1.1.0
|
||||
shared_preload_libraries: pg_duckdb
|
||||
```
|
||||
|
||||
The image is built on Debian 12 bookworm — the **same base as the official
|
||||
`postgres:18` image** — and uses the standard `docker-entrypoint.sh`. So
|
||||
`POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` / `PGDATA` /
|
||||
`/docker-entrypoint-initdb.d` / `pg_isready` all behave exactly as with
|
||||
`postgres:18`. It is a drop-in replacement for the `db` service; nothing else
|
||||
in the stack changes.
|
||||
|
||||
We `FROM` the official pg_duckdb image (pinned to `18-v1.1.1`, not `18-main`)
|
||||
rather than `FROM postgres:18` + compile, because pg_duckdb links a full DuckDB
|
||||
build — compiling from source in the Dockerfile means the DuckDB toolchain and
|
||||
a long, fragile build for no benefit over the maintainers' official PG18 image.
|
||||
The previous DB image was a custom `pgduckdb/pgduckdb:18` build whose only purpose
|
||||
was ad-hoc `read_parquet()` over the parquet cache. Now the climate record is in
|
||||
real tables, so that capability is gone and the DB is the **stock TimescaleDB
|
||||
image** — no build step. The image already sets
|
||||
`shared_preload_libraries=timescaledb`; never `ALTER SYSTEM SET
|
||||
shared_preload_libraries` (it would land in `postgresql.auto.conf` and override the
|
||||
image's preload).
|
||||
|
||||
## Files here
|
||||
|
||||
- **`Dockerfile.db`** — `FROM pgduckdb/pgduckdb:18-v1.1.1`, plus `COPY` of the
|
||||
init script so the image enables the extension on first init even without the
|
||||
compose bind mount.
|
||||
- **`init/10-parquet.sql`** — `CREATE EXTENSION IF NOT EXISTS pg_duckdb;` (runs
|
||||
from `/docker-entrypoint-initdb.d` on first cluster init).
|
||||
- **`init/10-timescaledb.sql`** — `CREATE EXTENSION IF NOT EXISTS timescaledb;`
|
||||
(runs from `/docker-entrypoint-initdb.d` on first cluster init; Alembic also does
|
||||
this idempotently at boot).
|
||||
- **`init/20-tuning.sh`** — scales `shared_buffers` (25%), `effective_cache_size`
|
||||
(75%), `work_mem`, and `maintenance_work_mem` from `DB_MEMORY` via `ALTER SYSTEM`.
|
||||
|
||||
## Compose snippet to merge into the `db` service
|
||||
|
||||
Replace `image: postgres:18` with the `build:` block; add the read-only parquet
|
||||
bind and the init mount. Everything else in the `db` service stays as-is.
|
||||
## Compose `db` service
|
||||
|
||||
```yaml
|
||||
db:
|
||||
# image: postgres:18 # <- remove; build the parquet-capable image
|
||||
build:
|
||||
context: .
|
||||
dockerfile: deploy/db/Dockerfile.db
|
||||
image: timescale/timescaledb:latest-pg18
|
||||
environment:
|
||||
POSTGRES_USER: thermograph
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}
|
||||
POSTGRES_DB: thermograph
|
||||
PGDATA: /var/lib/postgresql/data
|
||||
DB_MEMORY: ${DB_MEMORY:-8g}
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
- ./data/cache:/parquet:ro # read-only parquet cache
|
||||
- pgdata:/var/lib/postgresql
|
||||
- ./deploy/db/init:/docker-entrypoint-initdb.d
|
||||
# healthcheck / cpus / deploy / restart: unchanged
|
||||
# healthcheck / cpus / mem_limit / shm_size: unchanged
|
||||
```
|
||||
|
||||
Notes:
|
||||
- The bind mount at `/docker-entrypoint-initdb.d` **replaces** the base image's
|
||||
own init scripts (its `0001-install-pg_duckdb.sql` and the MotherDuck-only
|
||||
`0002-enable-md-pg_duckdb.sql`). That's intended: our `10-parquet.sql` still
|
||||
runs `CREATE EXTENSION`, and we don't use MotherDuck. If you prefer to keep
|
||||
the image's baked scripts, drop the `:/docker-entrypoint-initdb.d` line — the
|
||||
`Dockerfile` already bakes `10-parquet.sql` in.
|
||||
- `:ro` keeps the DB from ever mutating the app's cache. In prod the app writes
|
||||
the cache to the `appdata` volume; point this bind at wherever that lives on
|
||||
the host if you want the DB to see the live cache rather than the repo copy.
|
||||
- Init scripts only run when `PGDATA` is empty. On an **existing** database,
|
||||
enable it once by hand:
|
||||
- The volume is mounted at the **parent** of the data dir; the image picks its own
|
||||
PGDATA subdir (`/var/lib/postgresql/data`) under it. The whole tree persists on
|
||||
the named volume.
|
||||
- No host port on purpose: the app reaches Postgres as `db:5432` on the compose
|
||||
network. Nothing outside the stack should touch the database.
|
||||
- Init scripts only run when PGDATA is empty. On an **existing** database, enable
|
||||
the extension once by hand:
|
||||
```
|
||||
docker compose exec db psql -U thermograph -d thermograph \
|
||||
-c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;'
|
||||
-c 'CREATE EXTENSION IF NOT EXISTS timescaledb;'
|
||||
```
|
||||
|
||||
## Usage — real cache file at `/parquet/…`
|
||||
|
||||
Cache columns: `date, tmax, tmin, precip, wind, gust, humid, fmax, fmin, feels`.
|
||||
pg_duckdb ≥ 0.3 uses the `r['colname']` subscript syntax (not
|
||||
`AS t(col type, …)`).
|
||||
## Inspecting the hypertable
|
||||
|
||||
```sql
|
||||
-- one cell, first rows
|
||||
SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin, r['precip'] AS precip
|
||||
FROM read_parquet('/parquet/1026_-2857.parquet') r
|
||||
ORDER BY r['date'] LIMIT 5;
|
||||
-- Chunk / compression overview
|
||||
SELECT hypertable_name, num_chunks, compression_enabled
|
||||
FROM timescaledb_information.hypertables;
|
||||
|
||||
-- per-year analytics over one cell
|
||||
SELECT EXTRACT(YEAR FROM r['date']::timestamp) AS yr,
|
||||
ROUND(AVG(r['tmax'])::numeric, 1) AS avg_tmax,
|
||||
MAX(r['tmax']) AS record_high
|
||||
FROM read_parquet('/parquet/1026_-2857.parquet') r
|
||||
WHERE r['date'] >= '2020-01-01'
|
||||
-- One cell, most recent archived days
|
||||
SELECT date, tmax, tmin, precip
|
||||
FROM climate_history WHERE cell_id = '1026_-2857'
|
||||
ORDER BY date DESC LIMIT 5;
|
||||
|
||||
-- Per-year highs for one cell
|
||||
SELECT EXTRACT(YEAR FROM date) AS yr,
|
||||
ROUND(AVG(tmax)::numeric, 1) AS avg_tmax, MAX(tmax) AS record_high
|
||||
FROM climate_history WHERE cell_id = '1026_-2857' AND date >= '2020-01-01'
|
||||
GROUP BY yr ORDER BY yr;
|
||||
|
||||
-- glob across every cached cell (union_by_name handles the _rf / _forecast
|
||||
-- files whose column sets differ)
|
||||
SELECT COUNT(*) FROM read_parquet('/parquet/*.parquet', union_by_name := true) r;
|
||||
```
|
||||
|
||||
## Proof of work (verified 2026-07-19)
|
||||
|
||||
Built `deploy/db/Dockerfile.db`, ran a throwaway container with the real
|
||||
`data/cache` bind-mounted read-only at `/parquet`, then:
|
||||
|
||||
```
|
||||
$ psql -c "SELECT version();"
|
||||
PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2) on x86_64-pc-linux-gnu ...
|
||||
|
||||
$ psql -c "SELECT extname, extversion FROM pg_extension WHERE extname='pg_duckdb';"
|
||||
extname | extversion
|
||||
-----------+------------
|
||||
pg_duckdb | 1.1.0
|
||||
|
||||
$ psql -c "SELECT COUNT(*) FROM read_parquet('/parquet/1026_-2857.parquet') r;"
|
||||
row_count
|
||||
-----------
|
||||
16982
|
||||
|
||||
$ psql -c "SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin,
|
||||
r['precip'] AS precip, r['humid'] AS humid
|
||||
FROM read_parquet('/parquet/1026_-2857.parquet') r
|
||||
ORDER BY r['date'] LIMIT 5;"
|
||||
date | tmax | tmin | precip | humid
|
||||
---------------------+------+------+--------+-------
|
||||
1980-01-01 00:00:00 | 56.2 | 34.7 | 0 | 73
|
||||
1980-01-02 00:00:00 | 63.8 | 38 | 0 | 79
|
||||
1980-01-03 00:00:00 | 60.1 | 46.1 | 0.315 | 83
|
||||
1980-01-04 00:00:00 | 51.7 | 40 | 0 | 68
|
||||
1980-01-05 00:00:00 | 56.5 | 33.7 | 0 | 78
|
||||
|
||||
$ psql -c "SELECT EXTRACT(YEAR FROM r['date']::timestamp) AS yr,
|
||||
ROUND(AVG(r['tmax'])::numeric,1) AS avg_tmax, MAX(r['tmax']) AS record_high
|
||||
FROM read_parquet('/parquet/1026_-2857.parquet') r
|
||||
WHERE r['date'] >= '2020-01-01' GROUP BY yr ORDER BY yr;"
|
||||
yr | avg_tmax | record_high
|
||||
------+----------+-------------
|
||||
2020 | 78.8 | 99
|
||||
2021 | 77.0 | 93
|
||||
2022 | 79.2 | 101.4
|
||||
2023 | 80.8 | 107.1
|
||||
2024 | 79.8 | 97.5
|
||||
2025 | 80.0 | 99.4
|
||||
2026 | 77.8 | 95.6
|
||||
|
||||
$ psql -c "SELECT COUNT(*) FROM read_parquet('/parquet/*.parquet', union_by_name := true) r;"
|
||||
count
|
||||
--------
|
||||
525421 -- rows across all 45 cached cells
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,15 +0,0 @@
|
|||
-- Enable pg_duckdb so the database can read the app's Parquet climate cache
|
||||
-- directly (ad-hoc analytics via DuckDB's read_parquet / read_csv / etc.).
|
||||
--
|
||||
-- This runs once, on first cluster init (empty PGDATA), from
|
||||
-- /docker-entrypoint-initdb.d. Because the compose `db` service keeps its data
|
||||
-- on a persistent named volume, init scripts do NOT re-run on an existing
|
||||
-- database -- to enable pg_duckdb on a DB that was created before this image,
|
||||
-- run it by hand:
|
||||
--
|
||||
-- docker compose exec db psql -U thermograph -d thermograph \
|
||||
-- -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;'
|
||||
--
|
||||
-- pg_duckdb requires shared_preload_libraries='pg_duckdb'; the base image's
|
||||
-- postgresql.conf.sample already sets it, so it is active on fresh init.
|
||||
CREATE EXTENSION IF NOT EXISTS pg_duckdb;
|
||||
14
deploy/db/init/10-timescaledb.sql
Normal file
14
deploy/db/init/10-timescaledb.sql
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
-- Enable TimescaleDB so the app's climate record lives in hypertables
|
||||
-- (climate_history) instead of parquet files. The stock timescale/timescaledb
|
||||
-- image already preloads the extension (shared_preload_libraries=timescaledb);
|
||||
-- this just runs CREATE EXTENSION on first cluster init.
|
||||
--
|
||||
-- This runs once, on first cluster init (empty PGDATA), from
|
||||
-- /docker-entrypoint-initdb.d. Because the compose `db` service keeps its data on
|
||||
-- a persistent named volume, init scripts do NOT re-run on an existing database.
|
||||
-- Alembic (backend/alembic) also runs `CREATE EXTENSION IF NOT EXISTS timescaledb`
|
||||
-- at app boot, so an existing volume gets it there; to enable it by hand instead:
|
||||
--
|
||||
-- docker compose exec db psql -U thermograph -d thermograph \
|
||||
-- -c 'CREATE EXTENSION IF NOT EXISTS timescaledb;'
|
||||
CREATE EXTENSION IF NOT EXISTS timescaledb;
|
||||
|
|
@ -2,9 +2,12 @@
|
|||
# Postgres memory / performance tuning, scaled to the container's DB_MEMORY budget so
|
||||
# the same init serves every host (beta 8g; prod 16g on the 48 GB box) with no
|
||||
# hardcoding. Runs once on a fresh data volume from /docker-entrypoint-initdb.d, after
|
||||
# 10-parquet.sql enables pg_duckdb. Settings are written via ALTER SYSTEM (persisted to
|
||||
# postgresql.auto.conf); the container's post-init restart brings restart-only settings
|
||||
# (shared_buffers, …) into effect. Init scripts do NOT re-run on an existing volume — to
|
||||
# 10-timescaledb.sql enables timescaledb. Settings are written via ALTER SYSTEM
|
||||
# (persisted to postgresql.auto.conf, which the timescaledb image's own
|
||||
# timescaledb-tune postgresql.conf defers to); the container's post-init restart brings
|
||||
# restart-only settings (shared_buffers, …) into effect. NB: never ALTER SYSTEM SET
|
||||
# shared_preload_libraries here — that would land in auto.conf and override the image's
|
||||
# `timescaledb` preload. Init scripts do NOT re-run on an existing volume — to
|
||||
# re-tune later, set DB_MEMORY and run this by hand, then restart:
|
||||
# docker compose exec -e DB_MEMORY=16g db bash /docker-entrypoint-initdb.d/20-tuning.sh
|
||||
# docker compose restart db
|
||||
|
|
@ -23,18 +26,17 @@ esac
|
|||
if [ "$mb" -lt 1024 ]; then mb=8192; fi
|
||||
|
||||
# Derive settings from the budget. The ratios reproduce the historical 8 GB tuning
|
||||
# (shared_buffers 2 GB, effective_cache_size 6 GB, duckdb 4 GB, work_mem 64 MB,
|
||||
# (shared_buffers 2 GB, effective_cache_size 6 GB, work_mem 64 MB,
|
||||
# maintenance_work_mem 512 MB) and scale linearly on a bigger box.
|
||||
shared_buffers=$((mb / 4)) # 25% — the shared page cache
|
||||
effective_cache=$((mb * 3 / 4)) # 75% — planner's view of total cache (PG + OS)
|
||||
duckdb_mem=$((mb / 2)) # 50% — pg_duckdb ceiling for parquet processing
|
||||
work_mem=$((mb / 128)) # ~64 MB at 8 GB (per-operation; kept modest)
|
||||
if [ "$work_mem" -lt 16 ]; then work_mem=16; fi
|
||||
maint_mem=$((mb / 16)) # 512 MB at 8 GB — index builds / VACUUM
|
||||
|
||||
echo "[tuning] DB_MEMORY=${budget} -> ${mb}MB: shared_buffers=${shared_buffers}MB" \
|
||||
"effective_cache_size=${effective_cache}MB work_mem=${work_mem}MB" \
|
||||
"maintenance_work_mem=${maint_mem}MB duckdb.max_memory=${duckdb_mem}MB"
|
||||
"maintenance_work_mem=${maint_mem}MB"
|
||||
|
||||
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<SQL
|
||||
-- Caching: the shared page cache, and the planner's view of total cache (PG + OS).
|
||||
|
|
@ -57,8 +59,4 @@ ALTER SYSTEM SET effective_io_concurrency = 200;
|
|||
|
||||
-- Room for parallel scans/aggregates on the bigger analytic queries.
|
||||
ALTER SYSTEM SET max_parallel_workers_per_gather = 2;
|
||||
|
||||
-- DuckDB (pg_duckdb) memory ceiling for parquet processing. pg_duckdb is preloaded
|
||||
-- (shared_preload_libraries), so this GUC exists at ALTER SYSTEM time.
|
||||
ALTER SYSTEM SET duckdb.max_memory = '${duckdb_mem}MB';
|
||||
SQL
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
# containers at that mount; it defaults to ./data/om-archive so a local smoke test
|
||||
# works against a plain directory. Object storage is never bind-mounted into the app
|
||||
# — only Open-Meteo reads it; the app just talks HTTP to open-meteo-api and keeps its
|
||||
# own small daily-per-cell parquet cache.
|
||||
# own daily-per-cell climate record in the TimescaleDB `db` service.
|
||||
#
|
||||
# One-time backfill (writes ~1–1.5 TB of .om to object storage — run once before the
|
||||
# app is flipped over): `make om-backfill`.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# Thermograph production stack: the FastAPI app plus its PostgreSQL 18 database.
|
||||
# Thermograph production stack: the FastAPI app plus its TimescaleDB (PostgreSQL 18)
|
||||
# database.
|
||||
#
|
||||
# docker compose up -d --build # or: make up
|
||||
#
|
||||
|
|
@ -10,13 +11,13 @@
|
|||
|
||||
services:
|
||||
db:
|
||||
# PostgreSQL 18 + pg_duckdb (deploy/db/Dockerfile.db, FROM pgduckdb/pgduckdb:18-…,
|
||||
# which IS PostgreSQL 18). Lets the DB read the app's parquet climate cache
|
||||
# directly, e.g. SELECT * FROM read_parquet('/parquet/cache/*.parquet'). The init
|
||||
# script CREATE EXTENSIONs pg_duckdb on a fresh volume.
|
||||
build:
|
||||
context: .
|
||||
dockerfile: deploy/db/Dockerfile.db
|
||||
# TimescaleDB on PostgreSQL 18 (the stock image already sets
|
||||
# shared_preload_libraries=timescaledb). The app's climate record — the full
|
||||
# daily archive and the hourly recent+forecast bundle — lives in hypertables
|
||||
# here (see backend/data/climate_store.py), so the DB, not the filesystem, is the
|
||||
# source of truth. The init script CREATE EXTENSIONs timescaledb on a fresh volume
|
||||
# (Alembic also does, idempotently, at app boot).
|
||||
image: timescale/timescaledb:latest-pg18
|
||||
environment:
|
||||
POSTGRES_USER: thermograph
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}
|
||||
|
|
@ -26,14 +27,11 @@ services:
|
|||
# per host: Terraform sets DB_MEMORY (prod 16g), local/beta default 8g.
|
||||
DB_MEMORY: ${DB_MEMORY:-8g}
|
||||
volumes:
|
||||
# Mount the volume at the PARENT of the data dir and let the PG18 image use
|
||||
# its own versioned subdir (/var/lib/postgresql/18/docker). Pinning PGDATA
|
||||
# directly at the mountpoint trips an initdb chmod on some Docker setups; the
|
||||
# whole tree still persists on the named volume this way.
|
||||
# Mount the volume at the PARENT of the data dir and let the image pick its own
|
||||
# PGDATA subdir under it (the timescaledb image defaults to
|
||||
# /var/lib/postgresql/data). Pinning PGDATA directly at the mountpoint trips an
|
||||
# initdb chmod on some Docker setups; the whole tree still persists this way.
|
||||
- pgdata:/var/lib/postgresql
|
||||
# Read the app's live parquet cache (the same appdata volume the app writes,
|
||||
# mounted read-only) — query it under /parquet/cache/*.parquet.
|
||||
- appdata:/parquet:ro
|
||||
- ./deploy/db/init:/docker-entrypoint-initdb.d
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U thermograph -d thermograph"]
|
||||
|
|
@ -42,9 +40,9 @@ services:
|
|||
retries: 10
|
||||
# Give Postgres room to cache + process. mem_limit is the hard ceiling; the actual
|
||||
# budget is tuned in deploy/db/init/20-tuning.sh, which scales shared_buffers (25%),
|
||||
# effective_cache_size (75%), work_mem, maintenance_work_mem and duckdb.max_memory
|
||||
# (50%) from DB_MEMORY — so raising DB_MEMORY raises both the cap and the tuning
|
||||
# together. shm_size backs parallel-query shared memory (the 64MB docker default is
|
||||
# effective_cache_size (75%), work_mem and maintenance_work_mem from DB_MEMORY — so
|
||||
# raising DB_MEMORY raises both the cap and the tuning together. shm_size backs
|
||||
# parallel-query shared memory (the 64MB docker default is
|
||||
# too small once shared_buffers/parallelism grow).
|
||||
# Sized via env (Terraform sets DB_CPUS/DB_MEMORY per host); defaults match the
|
||||
# historical 2 CPU / 8 GB budget so a plain `docker compose up` is unchanged.
|
||||
|
|
|
|||
|
|
@ -42,11 +42,11 @@ A change to the rendered env, the compose files, the branch, or the sizing flips
|
|||
environment (defaults `4 / 4 / 2 / 8g`, identical to before). Terraform sets them per
|
||||
host through `/etc/thermograph.env`, so the big prod box can run larger caps without a
|
||||
compose edit. The Postgres *internal* memory budget (`shared_buffers`,
|
||||
`effective_cache_size`, `work_mem`, `duckdb.max_memory`) is derived from the same
|
||||
`effective_cache_size`, `work_mem`, `maintenance_work_mem`) is derived from the same
|
||||
`DB_MEMORY` by `deploy/db/init/20-tuning.sh` — so raising `db_memory` scales the
|
||||
container cap and the tuning together (prod 16g → shared_buffers 4 GB, duckdb 8 GB). The
|
||||
tuning applies on a fresh DB volume; on an existing volume re-run it by hand (see the
|
||||
script header).
|
||||
container cap and the tuning together (prod 16g → shared_buffers 4 GB). The tuning
|
||||
applies on a fresh DB volume; on an existing volume re-run it by hand (see the script
|
||||
header).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue