thermograph/deploy/POSTGRES-MIGRATION.md

# Cutover: SQLite → PostgreSQL 18 (containerized stack)

The app now runs as a docker-compose stack (`app` + `db`) and standardizes on
PostgreSQL. Locally you need nothing but Docker; `make up` builds and starts both.
This doc is the **one-time production cutover** from the old bare-systemd/SQLite
deploy to the compose stack, migrating the authoritative accounts data.

## What moved

- **accounts** (users/subscriptions/notifications/…): SQLite → Postgres **durable**
  tables. Managed by Alembic (`backend/alembic/`); `create_all` on a fresh DB.
- **derived cache** (`store.py`) and **metrics** (`metrics.py`): now Postgres
  **UNLOGGED** tables (fast, non-durable — same throwaway semantics). No data to
  migrate — the cache rebuilds from parquet, metrics start empty.
- The app selects Postgres when `THERMOGRAPH_DATABASE_URL` is a `postgresql+asyncpg`
  URL; unset ⇒ the old SQLite behavior (this is how the **test suite** stays on
  SQLite — no Postgres needed in CI).
- `db` runs **pg_duckdb** (`deploy/db/Dockerfile.db`, genuine PG18): the app's
  parquet cache is mounted read-only at `/parquet`, so you can query it directly,
  e.g. `SELECT r['date'], r['tmax'] FROM read_parquet('/parquet/cache/*.parquet') r`.

## Connection model

Per worker (4 workers): a read-write asyncpg engine + a read-only asyncpg engine
(`default_transaction_read_only=on`, used by the pure-GET endpoints), plus one sync
psycopg engine for the notifier thread. Single Postgres primary — the RO engine is
a guardrail, not a replica.

## Prod cutover steps (maintenance window)

Prereq: the deploy user can run `docker`. Secrets in `/etc/thermograph.env` must
include `POSTGRES_PASSWORD`, and pinned `THERMOGRAPH_AUTH_SECRET`,
`THERMOGRAPH_VAPID_PRIVATE_KEY`/`_PUBLIC_KEY`, `THERMOGRAPH_COOKIE_SECURE=1`.

1. **Ship the stack, don't cut over yet.** Deploy the branch; `docker compose build`.
   Bring up only the DB: `docker compose up -d db`. Add a nightly `pg_dump` backup.
2. **Freeze + back up.** Stop the old app (hard stop — no writes). Back up the live
   `data/accounts.sqlite` + `-wal`/`-shm` (this is the rollback artifact).
3. **Build the schema.** `docker compose run --rm app alembic upgrade head`
   (or let the `app` entrypoint do it on first start — but do the data copy before
   real traffic).
4. **Copy the accounts data:**
   ```
   docker compose run --rm \
     -e THERMOGRAPH_DATABASE_URL="postgresql+asyncpg://thermograph:$POSTGRES_PASSWORD@db:5432/thermograph" \
     -v /opt/thermograph/data/accounts.sqlite:/src.sqlite:ro \
     app python migrate_accounts_to_pg.py --sqlite /src.sqlite
   ```
   It copies user → subscription/push_subscription/pending_digest → notification,
   **preserving PKs**, **skips access_token** (everyone re-logins once), and
   **resets the integer-PK sequences** (skipping that = a runtime PK collision).
   Verify the printed per-table counts against the old DB.
5. **Start serving:** `docker compose up -d` (Caddy already proxies `127.0.0.1:8137`).
6. **Smoke test:** login, list/create/delete a subscription, one notifier tick, a
   cached calendar/day request (derived store on PG), the metrics dashboard.
7. Keep the frozen `accounts.sqlite` as rollback for a few days; keep PG backed up.
   Enable pg_duckdb on the (already-initialized) volume once, by hand:
   `docker compose exec db psql -U thermograph -d thermograph -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;'`

## Rollback

Redeploy the previous SQLite release (or point `THERMOGRAPH_DATABASE_URL` back to
unset/SQLite and restart). **Clean only during/immediately after the window** — once
real users write to Postgres, those writes are lost on rollback (the copy is
one-way), so keep the window short and writes frozen during the copy.

## Notes

- The two `deploy/migrations/*.sql` files are superseded by Alembic on Postgres
  (they were SQLite-specific `ALTER TABLE` column-adds for the old prod DB).
- The PG18 image's data dir is the versioned `…/18/docker` subdir; the compose
  volume is mounted at the parent `/var/lib/postgresql` so it persists without the
  initdb permission issue that pinning `PGDATA` to the mountpoint can trigger.