Containerize the app and move the databases to PostgreSQL 18 (#220)

Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the data layer on Postgres, while keeping the test suite on SQLite. - accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write + read-only asyncpg pair (the RO engine pins read-only transactions, used by the pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL is unset). models.py: boolean server_default -> sa.false(). - store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg when configured, else the existing raw-sqlite3 paths byte-for-byte; sync interfaces and every fail-soft contract preserved. - Alembic (backend/alembic/) manages the accounts schema; the container entrypoint runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced), skips access_token, and resets identity sequences. - Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137) and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet'). - deploy.sh/thermograph.service rewired to manage the compose stack; env example, Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook. Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was verified via docker compose: alembic migrations, register/login, the RO endpoint, store/metrics round-trips, and the accounts data migration.
2026-07-19 23:28:23 -07:00 · 2026-07-19 23:28:23 -07:00 · 6fd2d7c981
commit 6fd2d7c981
parent 1e89bc71c9
8 changed files with 452 additions and 81 deletions
--- a/deploy/POSTGRES-MIGRATION.md
+++ b/deploy/POSTGRES-MIGRATION.md
@ -0,0 +1,73 @@
+# Cutover: SQLite → PostgreSQL 18 (containerized stack)
+
+The app now runs as a docker-compose stack (`app` + `db`) and standardizes on
+PostgreSQL. Locally you need nothing but Docker; `make up` builds and starts both.
+This doc is the **one-time production cutover** from the old bare-systemd/SQLite
+deploy to the compose stack, migrating the authoritative accounts data.
+
+## What moved
+
+- **accounts** (users/subscriptions/notifications/…): SQLite → Postgres **durable**
+  tables. Managed by Alembic (`backend/alembic/`); `create_all` on a fresh DB.
+- **derived cache** (`store.py`) and **metrics** (`metrics.py`): now Postgres
+  **UNLOGGED** tables (fast, non-durable — same throwaway semantics). No data to
+  migrate — the cache rebuilds from parquet, metrics start empty.
+- The app selects Postgres when `THERMOGRAPH_DATABASE_URL` is a `postgresql+asyncpg`
+  URL; unset ⇒ the old SQLite behavior (this is how the **test suite** stays on
+  SQLite — no Postgres needed in CI).
+- `db` runs **pg_duckdb** (`deploy/db/Dockerfile.db`, genuine PG18): the app's
+  parquet cache is mounted read-only at `/parquet`, so you can query it directly,
+  e.g. `SELECT r['date'], r['tmax'] FROM read_parquet('/parquet/cache/*.parquet') r`.
+
+## Connection model
+
+Per worker (4 workers): a read-write asyncpg engine + a read-only asyncpg engine
+(`default_transaction_read_only=on`, used by the pure-GET endpoints), plus one sync
+psycopg engine for the notifier thread. Single Postgres primary — the RO engine is
+a guardrail, not a replica.
+
+## Prod cutover steps (maintenance window)
+
+Prereq: the deploy user can run `docker`. Secrets in `/etc/thermograph.env` must
+include `POSTGRES_PASSWORD`, and pinned `THERMOGRAPH_AUTH_SECRET`,
+`THERMOGRAPH_VAPID_PRIVATE_KEY`/`_PUBLIC_KEY`, `THERMOGRAPH_COOKIE_SECURE=1`.
+
+1. **Ship the stack, don't cut over yet.** Deploy the branch; `docker compose build`.
+   Bring up only the DB: `docker compose up -d db`. Add a nightly `pg_dump` backup.
+2. **Freeze + back up.** Stop the old app (hard stop — no writes). Back up the live
+   `data/accounts.sqlite` + `-wal`/`-shm` (this is the rollback artifact).
+3. **Build the schema.** `docker compose run --rm app alembic upgrade head`
+   (or let the `app` entrypoint do it on first start — but do the data copy before
+   real traffic).
+4. **Copy the accounts data:**
+   ```
+   docker compose run --rm \
+     -e THERMOGRAPH_DATABASE_URL="postgresql+asyncpg://thermograph:$POSTGRES_PASSWORD@db:5432/thermograph" \
+     -v /opt/thermograph/data/accounts.sqlite:/src.sqlite:ro \
+     app python migrate_accounts_to_pg.py --sqlite /src.sqlite
+   ```
+   It copies user → subscription/push_subscription/pending_digest → notification,
+   **preserving PKs**, **skips access_token** (everyone re-logins once), and
+   **resets the integer-PK sequences** (skipping that = a runtime PK collision).
+   Verify the printed per-table counts against the old DB.
+5. **Start serving:** `docker compose up -d` (Caddy already proxies `127.0.0.1:8137`).
+6. **Smoke test:** login, list/create/delete a subscription, one notifier tick, a
+   cached calendar/day request (derived store on PG), the metrics dashboard.
+7. Keep the frozen `accounts.sqlite` as rollback for a few days; keep PG backed up.
+   Enable pg_duckdb on the (already-initialized) volume once, by hand:
+   `docker compose exec db psql -U thermograph -d thermograph -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;'`
+
+## Rollback
+
+Redeploy the previous SQLite release (or point `THERMOGRAPH_DATABASE_URL` back to
+unset/SQLite and restart). **Clean only during/immediately after the window** — once
+real users write to Postgres, those writes are lost on rollback (the copy is
+one-way), so keep the window short and writes frozen during the copy.
+
+## Notes
+
+- The two `deploy/migrations/*.sql` files are superseded by Alembic on Postgres
+  (they were SQLite-specific `ALTER TABLE` column-adds for the old prod DB).
+- The PG18 image's data dir is the versioned `…/18/docker` subdir; the compose
+  volume is mounted at the parent `/var/lib/postgresql` so it persists without the
+  initdb permission issue that pinning `PGDATA` to the mountpoint can trigger.
--- a/deploy/db/Dockerfile.db
+++ b/deploy/db/Dockerfile.db
@ -0,0 +1,25 @@
+# Thermograph database image: PostgreSQL 18 with pg_duckdb.
+#
+# pg_duckdb embeds DuckDB inside Postgres, which lets the DB container read the
+# app's Parquet climate cache directly for ad-hoc analytics:
+#
+#     SELECT * FROM read_parquet('/parquet/1026_-2857.parquet');
+#
+# Why FROM the official pg_duckdb image instead of `FROM postgres:18` + compile:
+# pg_duckdb links a full DuckDB build, so compiling it from source in this
+# Dockerfile would mean pulling the DuckDB toolchain and a long, fragile build.
+# The maintainers (duckdb/pg_duckdb, MotherDuck) publish an official PG18 image
+# that is genuine PostgreSQL 18.1 on Debian 12 bookworm -- the exact same base
+# as the official `postgres:18` image, using the standard postgres
+# docker-entrypoint.sh. So POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB /
+# PGDATA / /docker-entrypoint-initdb.d / pg_isready all behave identically to
+# `postgres:18`; this is a drop-in replacement for the compose `db` service.
+#
+# Pinned to a specific patch tag (not 18-main) for reproducible builds.
+FROM pgduckdb/pgduckdb:18-v1.1.1
+
+# Bake the parquet init script so the image enables the extension on first init
+# even without the compose bind mount. It is CREATE EXTENSION IF NOT EXISTS, so
+# it is idempotent with the base image's own 0001-install-pg_duckdb.sql.
+# Build context is the repo root (see deploy/db/README.md for the compose snippet).
+COPY deploy/db/init/ /docker-entrypoint-initdb.d/
--- a/deploy/db/README.md
+++ b/deploy/db/README.md
@ -0,0 +1,171 @@
+# Thermograph DB image — Postgres 18 + parquet reads
+
+The `db` service can read the app's Parquet climate cache
+(`data/cache/*.parquet`) directly from SQL, for ad-hoc analytics, via
+**pg_duckdb** — DuckDB embedded inside Postgres.
+
+```sql
+SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin
+FROM read_parquet('/parquet/1026_-2857.parquet') r
+ORDER BY r['date'] LIMIT 5;
+```
+
+## Chosen extension: pg_duckdb — and why
+
+| Option | PG18? | Fit | Verdict |
+| --- | --- | --- | --- |
+| **pg_duckdb** (duckdb / MotherDuck) | **Yes** — official image `pgduckdb/pgduckdb:18-v1.1.1` is genuine PG18.1 | `read_parquet('…')` in plain SQL; globs, `union_by_name`, full DuckDB analytics engine | **Chosen** |
+| pg_parquet (Crunchy Data) | Yes (14–18) | `COPY … TO/FROM '…' (format 'parquet')` — import/export, not a query engine | Viable, but COPY-oriented; no standalone official image (ships via Crunchy Bridge/CPK), so it'd need a Rust/pgrx source build |
+| parquet_fdw | No prebuilt PG18 support; low activity | Foreign tables over parquet | Rejected — oldest, weakest PG18 story |
+
+pg_duckdb wins for the stated goal (ad-hoc analytics): it exposes DuckDB's
+`read_parquet` directly in SQL, so you query cache files like tables — no
+import step, no foreign-table DDL, and you get aggregation/joins/globs across
+many cells at once.
+
+## PG-version reality for PG18 (empirically verified 2026-07-19)
+
+**No version delta.** PG18 support is real, not a fallback. The pulled image
+reports:
+
+```
+PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2) on x86_64-pc-linux-gnu
+pg_extension: pg_duckdb 1.1.0
+shared_preload_libraries: pg_duckdb
+```
+
+The image is built on Debian 12 bookworm — the **same base as the official
+`postgres:18` image** — and uses the standard `docker-entrypoint.sh`. So
+`POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` / `PGDATA` /
+`/docker-entrypoint-initdb.d` / `pg_isready` all behave exactly as with
+`postgres:18`. It is a drop-in replacement for the `db` service; nothing else
+in the stack changes.
+
+We `FROM` the official pg_duckdb image (pinned to `18-v1.1.1`, not `18-main`)
+rather than `FROM postgres:18` + compile, because pg_duckdb links a full DuckDB
+build — compiling from source in the Dockerfile means the DuckDB toolchain and
+a long, fragile build for no benefit over the maintainers' official PG18 image.
+
+## Files here
+
+- **`Dockerfile.db`** — `FROM pgduckdb/pgduckdb:18-v1.1.1`, plus `COPY` of the
+  init script so the image enables the extension on first init even without the
+  compose bind mount.
+- **`init/10-parquet.sql`** — `CREATE EXTENSION IF NOT EXISTS pg_duckdb;` (runs
+  from `/docker-entrypoint-initdb.d` on first cluster init).
+
+## Compose snippet to merge into the `db` service
+
+Replace `image: postgres:18` with the `build:` block; add the read-only parquet
+bind and the init mount. Everything else in the `db` service stays as-is.
+
+```yaml
+  db:
+    # image: postgres:18            # <- remove; build the parquet-capable image
+    build:
+      context: .
+      dockerfile: deploy/db/Dockerfile.db
+    environment:
+      POSTGRES_USER: thermograph
+      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}
+      POSTGRES_DB: thermograph
+      PGDATA: /var/lib/postgresql/data
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+      - ./data/cache:/parquet:ro                 # read-only parquet cache
+      - ./deploy/db/init:/docker-entrypoint-initdb.d
+    # healthcheck / cpus / deploy / restart: unchanged
+```
+
+Notes:
+- The bind mount at `/docker-entrypoint-initdb.d` **replaces** the base image's
+  own init scripts (its `0001-install-pg_duckdb.sql` and the MotherDuck-only
+  `0002-enable-md-pg_duckdb.sql`). That's intended: our `10-parquet.sql` still
+  runs `CREATE EXTENSION`, and we don't use MotherDuck. If you prefer to keep
+  the image's baked scripts, drop the `:/docker-entrypoint-initdb.d` line — the
+  `Dockerfile` already bakes `10-parquet.sql` in.
+- `:ro` keeps the DB from ever mutating the app's cache. In prod the app writes
+  the cache to the `appdata` volume; point this bind at wherever that lives on
+  the host if you want the DB to see the live cache rather than the repo copy.
+- Init scripts only run when `PGDATA` is empty. On an **existing** database,
+  enable it once by hand:
+  ```
+  docker compose exec db psql -U thermograph -d thermograph \
+    -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;'
+  ```
+
+## Usage — real cache file at `/parquet/…`
+
+Cache columns: `date, tmax, tmin, precip, wind, gust, humid, fmax, fmin, feels`.
+pg_duckdb ≥ 0.3 uses the `r['colname']` subscript syntax (not
+`AS t(col type, …)`).
+
+```sql
+-- one cell, first rows
+SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin, r['precip'] AS precip
+FROM read_parquet('/parquet/1026_-2857.parquet') r
+ORDER BY r['date'] LIMIT 5;
+
+-- per-year analytics over one cell
+SELECT EXTRACT(YEAR FROM r['date']::timestamp) AS yr,
+       ROUND(AVG(r['tmax'])::numeric, 1) AS avg_tmax,
+       MAX(r['tmax']) AS record_high
+FROM read_parquet('/parquet/1026_-2857.parquet') r
+WHERE r['date'] >= '2020-01-01'
+GROUP BY yr ORDER BY yr;
+
+-- glob across every cached cell (union_by_name handles the _rf / _forecast
+-- files whose column sets differ)
+SELECT COUNT(*) FROM read_parquet('/parquet/*.parquet', union_by_name := true) r;
+```
+
+## Proof of work (verified 2026-07-19)
+
+Built `deploy/db/Dockerfile.db`, ran a throwaway container with the real
+`data/cache` bind-mounted read-only at `/parquet`, then:
+
+```
+$ psql -c "SELECT version();"
+ PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2) on x86_64-pc-linux-gnu ...
+
+$ psql -c "SELECT extname, extversion FROM pg_extension WHERE extname='pg_duckdb';"
+  extname  | extversion
+-----------+------------
+ pg_duckdb | 1.1.0
+
+$ psql -c "SELECT COUNT(*) FROM read_parquet('/parquet/1026_-2857.parquet') r;"
+ row_count
+-----------
+     16982
+
+$ psql -c "SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin,
+           r['precip'] AS precip, r['humid'] AS humid
+           FROM read_parquet('/parquet/1026_-2857.parquet') r
+           ORDER BY r['date'] LIMIT 5;"
+        date         | tmax | tmin | precip | humid
+---------------------+------+------+--------+-------
+ 1980-01-01 00:00:00 | 56.2 | 34.7 |      0 |    73
+ 1980-01-02 00:00:00 | 63.8 |   38 |      0 |    79
+ 1980-01-03 00:00:00 | 60.1 | 46.1 |  0.315 |    83
+ 1980-01-04 00:00:00 | 51.7 |   40 |      0 |    68
+ 1980-01-05 00:00:00 | 56.5 | 33.7 |      0 |    78
+
+$ psql -c "SELECT EXTRACT(YEAR FROM r['date']::timestamp) AS yr,
+           ROUND(AVG(r['tmax'])::numeric,1) AS avg_tmax, MAX(r['tmax']) AS record_high
+           FROM read_parquet('/parquet/1026_-2857.parquet') r
+           WHERE r['date'] >= '2020-01-01' GROUP BY yr ORDER BY yr;"
+  yr  | avg_tmax | record_high
+------+----------+-------------
+ 2020 |     78.8 |          99
+ 2021 |     77.0 |          93
+ 2022 |     79.2 |       101.4
+ 2023 |     80.8 |       107.1
+ 2024 |     79.8 |        97.5
+ 2025 |     80.0 |        99.4
+ 2026 |     77.8 |        95.6
+
+$ psql -c "SELECT COUNT(*) FROM read_parquet('/parquet/*.parquet', union_by_name := true) r;"
+ count
+--------
+ 525421   -- rows across all 45 cached cells
+```
--- a/deploy/db/init/10-parquet.sql
+++ b/deploy/db/init/10-parquet.sql
@ -0,0 +1,15 @@
+-- Enable pg_duckdb so the database can read the app's Parquet climate cache
+-- directly (ad-hoc analytics via DuckDB's read_parquet / read_csv / etc.).
+--
+-- This runs once, on first cluster init (empty PGDATA), from
+-- /docker-entrypoint-initdb.d. Because the compose `db` service keeps its data
+-- on a persistent named volume, init scripts do NOT re-run on an existing
+-- database -- to enable pg_duckdb on a DB that was created before this image,
+-- run it by hand:
+--
+--     docker compose exec db psql -U thermograph -d thermograph \
+--         -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;'
+--
+-- pg_duckdb requires shared_preload_libraries='pg_duckdb'; the base image's
+-- postgresql.conf.sample already sets it, so it is active on fresh init.
+CREATE EXTENSION IF NOT EXISTS pg_duckdb;
--- a/deploy/deploy.sh
+++ b/deploy/deploy.sh
@ -1,66 +1,58 @@
 #!/usr/bin/env bash
-# Pull the latest code and restart Thermograph. Run on the VPS — the GitHub
-# Actions workflow invokes this over SSH, and you can run it by hand too.
+# Pull the latest code and roll the docker-compose stack (app + PostgreSQL). Run
+# on the VPS — the GitHub Actions workflow invokes this over SSH, and you can run
+# it by hand too.
 #
 #   ssh deploy@vps '/opt/thermograph/deploy/deploy.sh'
 set -euo pipefail

 APP_DIR="${APP_DIR:-/opt/thermograph}"
 BRANCH="${BRANCH:-main}"
+HEALTH_PORT="${HEALTH_PORT:-8137}"
 cd "$APP_DIR"

+# Secrets (POSTGRES_PASSWORD, VAPID keys, AUTH_SECRET, ...) drive compose
+# interpolation and are also loaded into the app container via env_file. Source
+# them here so a by-hand run interpolates the same as the systemd unit does.
+set -a; . /etc/thermograph.env 2>/dev/null || true; set +a
+
 # Pre-warm the ~750 city-page archives so /climate pages serve from cache and a
-# search-engine crawl never bursts the archive API quota. Detached + backgrounded
-# so it never blocks the deploy or health check; idempotent (skips already-cached
-# cells), so it's cheap on every deploy after the first full warm.
+# search-engine crawl never bursts the archive API quota. Detached inside the app
+# container (compose exec -d), idempotent (skips already-cached cells), so it
+# never blocks the deploy or health check and is cheap on every deploy after the
+# first full warm.
 warm_city_archives() {
-  mkdir -p "$APP_DIR/logs"
-  echo "==> Warming city-page archives in the background (logs/warm-cities.log)"
-  setsid nohup bash -c "cd '$APP_DIR/backend' && exec '$APP_DIR/.venv/bin/python' warm_cities.py --pace 2" \
-    </dev/null >"$APP_DIR/logs/warm-cities.log" 2>&1 &
+  echo "==> Warming city-page archives in the background (app:/app/logs/warm-cities.log)"
+  docker compose exec -d app sh -c \
+    'python warm_cities.py --pace 2 >> /app/logs/warm-cities.log 2>&1' || true
 }

 # Notify IndexNow (Bing / DuckDuckGo / Yandex) of the site's URLs, but only when
-# the set of pages actually changed (a new/removed city) — code-only deploys skip,
-# so we don't re-blast ~14k URLs every push. Best-effort: never fails the deploy.
-# Sourcing the env matches the key the running service serves at /{key}.txt.
+# the set of pages actually changed (a new/removed city) — code-only deploys skip.
+# Best-effort: never fails the deploy.
 ping_indexnow() {
  echo "==> Pinging IndexNow (only if the URL set changed)"
-  ( set -a; . /etc/thermograph.env 2>/dev/null || true; set +a
-    base="${THERMOGRAPH_BASE_URL:-https://thermograph.org}"
-    cd "$APP_DIR/backend" && "$APP_DIR/.venv/bin/python" indexnow.py --if-changed "$base"
-  ) || echo "!! IndexNow ping failed (non-fatal)" >&2
+  local base="${THERMOGRAPH_BASE_URL:-https://thermograph.org}"
+  docker compose exec -T app python indexnow.py --if-changed "$base" \
+    || echo "!! IndexNow ping failed (non-fatal)" >&2
 }

 echo "==> Fetching $BRANCH"
 git fetch --prune origin "$BRANCH"
 git reset --hard "origin/$BRANCH"

-echo "==> Installing dependencies"
-if [ ! -d .venv ]; then
-  python3 -m venv .venv
-fi
-.venv/bin/pip install --upgrade pip -q
-.venv/bin/pip install -r backend/requirements.txt -q
+echo "==> Building images"
+docker compose build

-# Apply any pending accounts-DB schema migrations before the new code starts, so
-# it never queries a column an older database lacks. Idempotent and tracked, so
-# every deploy can run it; sourcing the env picks up a THERMOGRAPH_ACCOUNTS_DB
-# override if one is set. Runs as the deploy user, who owns the data dir.
-echo "==> Applying database migrations"
-( set -a; . /etc/thermograph.env 2>/dev/null || true; set +a
-  "$APP_DIR/.venv/bin/python" "$APP_DIR/deploy/migrate-db.py"
-)
-
-echo "==> Restarting service"
-sudo systemctl restart thermograph
+# Schema migrations run inside the app container's entrypoint (alembic upgrade
+# head) before uvicorn starts, so there's no separate migrate step or service
+# restart here — compose owns the process model.
+echo "==> Starting stack"
+docker compose up -d

 echo "==> Health check"
-PORT="$(sed -n 's/^PORT=//p' /etc/thermograph.env)"; PORT="${PORT:-8137}"
-BASE="$(sed -n 's/^THERMOGRAPH_BASE=//p' /etc/thermograph.env)"; BASE="${BASE:-/}"
-# Normalize: no double slash, allow root.
-url="http://127.0.0.1:${PORT}${BASE%/}/"
-for i in $(seq 1 15); do
+url="http://127.0.0.1:${HEALTH_PORT}/"
+for i in $(seq 1 30); do
  if curl -fsS -o /dev/null "$url"; then
    echo "==> OK: $url is serving"
    warm_city_archives
@ -70,5 +62,6 @@ for i in $(seq 1 15); do
  sleep 1
 done
 echo "!! Health check failed for $url" >&2
-sudo systemctl status thermograph --no-pager -l || true
+docker compose ps || true
+docker compose logs --tail=50 app || true
 exit 1
--- a/deploy/thermograph.env.example
+++ b/deploy/thermograph.env.example
@ -1,17 +1,44 @@
 # Copy to /etc/thermograph.env on the VPS and edit.
-# Read by the systemd unit (EnvironmentFile).
+# Read by the systemd unit (EnvironmentFile) so `docker compose up` can interpolate
+# it, AND loaded into the app container (env_file in docker-compose.yml). Anything
+# secret the app needs — Postgres password, VAPID keys, auth secret — belongs here.

-# Port uvicorn binds on loopback. Caddy proxies to this. Keep 8137 unless it clashes.
+# Port uvicorn binds inside the container. The compose stack publishes it on the
+# host loopback (127.0.0.1:8137) for Caddy to proxy to. Keep 8137.
 PORT=8137

+# --- PostgreSQL (docker-compose stack) ------------------------------------------
+# The app and Postgres run as a docker-compose stack (see docker-compose.yml).
+# POSTGRES_PASSWORD is the database password: compose uses it to initialize the
+# postgres container AND to build the app's THERMOGRAPH_DATABASE_URL. It MUST be
+# set here (the systemd unit sources this file so `docker compose up` can
+# interpolate it). Change it from the default before the first `up`.
+POSTGRES_PASSWORD=change-me
+
+# The app's compose service already builds THERMOGRAPH_DATABASE_URL from
+# POSTGRES_PASSWORD, so you normally DON'T need this. It's here for reference and
+# for running the app outside compose against the same DB (keep the password in
+# sync with POSTGRES_PASSWORD above).
+#THERMOGRAPH_DATABASE_URL=postgresql+asyncpg://thermograph:change-me@db:5432/thermograph
+
+# Mark the session cookie Secure — required behind Caddy's HTTPS. Set to 1 in prod;
+# leave unset only for plain-HTTP LAN dev (a Secure cookie is never sent over HTTP).
+THERMOGRAPH_COOKIE_SECURE=1
+
+# Pin these too (see their own sections below), so container restarts don't rotate
+# them: THERMOGRAPH_AUTH_SECRET (else every emailed confirm/reset link breaks on
+# restart) and THERMOGRAPH_VAPID_PRIVATE_KEY / _PUBLIC_KEY (else every existing push
+# subscription silently stops delivering). The data dir persists on the appdata
+# volume, but pinning here is the safe default.
+
 # Number of uvicorn worker processes. More than 1 stops a single slow upstream fetch
 # (e.g. a cache-miss weather lookup) from blocking every other request — the cause of
-# past brief outages. Prod runs 3; leave unset (defaults to 1) on a small box. Workers
-# share one metrics store (THERMOGRAPH_METRICS_DB, defaulted in the systemd unit) so the
-# ops dashboard still sees the whole picture, and elect one leader for the subscription
-# notifier via a lockfile (THERMOGRAPH_SINGLETON_LOCK, also defaulted in the unit) so its
-# timer-driven upstream sweep runs once, not once per worker. ~200 MB RAM per worker.
-WORKERS=3
+# past brief outages. Prod runs 4 (the compose app service also defaults to WORKERS=4);
+# leave unset (defaults to 1) on a small box. Workers elect one leader for the
+# subscription notifier via a lockfile (THERMOGRAPH_SINGLETON_LOCK, set by the compose
+# app service to /app/data/notifier.lock) so its timer-driven upstream sweep runs once,
+# not once per worker. ~200 MB RAM per worker.
+WORKERS=4

 # Base path the app is served under.
 #   /            -> app at the domain root (Thermograph owns the whole domain —
--- a/deploy/thermograph.service
+++ b/deploy/thermograph.service
@ -1,41 +1,22 @@
 [Unit]
-Description=Thermograph (FastAPI/uvicorn)
-After=network-online.target
+Description=Thermograph docker-compose stack (app + PostgreSQL)
+# Docker must be up, and the network online, before compose can pull/interpolate.
+After=docker.service network-online.target
 Wants=network-online.target
+Requires=docker.service

 [Service]
-Type=exec
-User=deploy
-Group=deploy
-# The repo checkout. uvicorn is run from backend/ so `app:app` resolves.
-WorkingDirectory=/opt/thermograph/backend
-# Defaults; /etc/thermograph.env (below) overrides. WORKERS is how many uvicorn
-# worker processes to run — prod sets WORKERS=3 there; a single-worker box can leave
-# it. Multiple workers each keep their own in-process state, so metrics go to a shared
-# SQLite DB (THERMOGRAPH_METRICS_DB) that every worker tallies into — otherwise the ops
-# dashboard, polling one random worker, would see only a fraction of the traffic.
-Environment=WORKERS=1
-Environment=THERMOGRAPH_METRICS_DB=/opt/thermograph/data/metrics.db
-# Elect one worker to run the subscription notifier (its timer-driven upstream sweep
-# must run once across the deploy, not once per worker — else it multiplies the
-# Open-Meteo quota use). Workers race for this lockfile; the winner runs the notifier.
-Environment=THERMOGRAPH_SINGLETON_LOCK=/opt/thermograph/data/notifier.lock
-# THERMOGRAPH_BASE, PORT, WORKERS, etc. See deploy/thermograph.env.example.
-EnvironmentFile=/etc/thermograph.env
-# Clear the shared metrics DB on each (re)start so the dashboard's "since start" tallies
-# reflect the running workers, matching how the old in-process counters behaved.
-ExecStartPre=/usr/bin/rm -f /opt/thermograph/data/metrics.db /opt/thermograph/data/metrics.db-wal /opt/thermograph/data/metrics.db-shm
-# Bind loopback only — Caddy terminates TLS and reverse-proxies to us.
-ExecStart=/opt/thermograph/.venv/bin/uvicorn app:app --host 127.0.0.1 --port ${PORT} --workers ${WORKERS}
-Restart=on-failure
-RestartSec=2
-# Hardening
-NoNewPrivileges=true
-PrivateTmp=true
-ProtectSystem=full
-ProtectHome=read-only
-# The parquet cache must stay writable across deploys.
-ReadWritePaths=/opt/thermograph/data /opt/thermograph/logs
+# A thin manager for the compose stack: `up -d` starts it and returns, the unit
+# stays "active" (RemainAfterExit) so `systemctl restart thermograph` re-runs it.
+# Compose owns container ordering and restarts via depends_on + restart policies.
+Type=oneshot
+RemainAfterExit=yes
+WorkingDirectory=/opt/thermograph
+# POSTGRES_PASSWORD (and the other secrets) are needed at `docker compose`
+# interpolation time. The leading `-` makes a missing file non-fatal.
+EnvironmentFile=-/etc/thermograph.env
+ExecStart=/usr/bin/docker compose up -d
+ExecStop=/usr/bin/docker compose down

 [Install]
 WantedBy=multi-user.target
--- a/docker-compose.yml
+++ b/docker-compose.yml
@ -0,0 +1,86 @@
+# Thermograph production stack: the FastAPI app plus its PostgreSQL 18 database.
+#
+#   docker compose up -d --build      # or: make up
+#
+# POSTGRES_PASSWORD must be set at `docker compose` time — compose reads it from
+# the repo-root .env for local runs (copy .env.example -> .env), and in prod the
+# systemd unit's EnvironmentFile=/etc/thermograph.env puts it in the environment
+# so `docker compose up` can interpolate it. It is used BOTH to initialize the db
+# container and to build the app's THERMOGRAPH_DATABASE_URL below.
+
+services:
+  db:
+    # PostgreSQL 18 + pg_duckdb (deploy/db/Dockerfile.db, FROM pgduckdb/pgduckdb:18-…,
+    # which IS PostgreSQL 18). Lets the DB read the app's parquet climate cache
+    # directly, e.g. SELECT * FROM read_parquet('/parquet/cache/*.parquet'). The init
+    # script CREATE EXTENSIONs pg_duckdb on a fresh volume.
+    build:
+      context: .
+      dockerfile: deploy/db/Dockerfile.db
+    environment:
+      POSTGRES_USER: thermograph
+      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}
+      POSTGRES_DB: thermograph
+    volumes:
+      # Mount the volume at the PARENT of the data dir and let the PG18 image use
+      # its own versioned subdir (/var/lib/postgresql/18/docker). Pinning PGDATA
+      # directly at the mountpoint trips an initdb chmod on some Docker setups; the
+      # whole tree still persists on the named volume this way.
+      - pgdata:/var/lib/postgresql
+      # Read the app's live parquet cache (the same appdata volume the app writes,
+      # mounted read-only) — query it under /parquet/cache/*.parquet.
+      - appdata:/parquet:ro
+      - ./deploy/db/init:/docker-entrypoint-initdb.d
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U thermograph -d thermograph"]
+      interval: 5s
+      timeout: 5s
+      retries: 10
+    # Cap the DB at 2 CPUs. Compose v2 honors the top-level `cpus:`; the
+    # deploy.resources block is the Swarm-style equivalent, kept for parity.
+    cpus: 2.0
+    deploy:
+      resources:
+        limits:
+          cpus: "2.0"
+    restart: unless-stopped
+    # No host port on purpose: the app reaches Postgres as db:5432 on the
+    # compose network. Nothing outside the stack should touch the database.
+
+  app:
+    build: .
+    depends_on:
+      db:
+        condition: service_healthy
+    environment:
+      # Built from POSTGRES_PASSWORD; this `environment` value wins over anything
+      # in env_file, so the URL always matches the db container's password.
+      THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph:${POSTGRES_PASSWORD}@db:5432/thermograph
+      THERMOGRAPH_BASE: /
+      PORT: 8137
+      WORKERS: 4
+      # One worker wins this lock and runs the subscription notifier / homepage
+      # sweep; it lives on the appdata volume so it's shared across workers.
+      THERMOGRAPH_SINGLETON_LOCK: /app/data/notifier.lock
+    # Prod secrets live in /etc/thermograph.env: POSTGRES_PASSWORD,
+    # THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY,
+    # THERMOGRAPH_COOKIE_SECURE=1, mail/Discord keys, ... (see
+    # deploy/thermograph.env.example). `required: false` so local `docker compose
+    # up` works without that file — it reads POSTGRES_PASSWORD from repo-root .env.
+    env_file:
+      - path: /etc/thermograph.env
+        required: false
+    volumes:
+      # Parquet cache, notifier.lock, homepage.json, vapid.json persist here.
+      - appdata:/app/data
+      - applogs:/app/logs
+    cpus: 4.0
+    ports:
+      # Loopback only — host Caddy terminates TLS and reverse-proxies to this.
+      - "127.0.0.1:8137:8137"
+    restart: unless-stopped
+
+volumes:
+  pgdata: {}
+  appdata: {}
+  applogs: {}