thermograph/deploy/POSTGRES-MIGRATION.md

Cutover: SQLite → PostgreSQL 18 (containerized stack)

The app now runs as a docker-compose stack (app + db) and standardizes on PostgreSQL. Locally you need nothing but Docker; make up builds and starts both. This doc is the one-time production cutover from the old bare-systemd/SQLite deploy to the compose stack, migrating the authoritative accounts data.

What moved

• accounts (users/subscriptions/notifications/…): SQLite → Postgres durable tables. Managed by Alembic (backend/alembic/); create_all on a fresh DB.
• derived cache (store.py) and metrics (metrics.py): now Postgres UNLOGGED tables (fast, non-durable — same throwaway semantics). No data to migrate — the cache rebuilds from parquet, metrics start empty.
• The app selects Postgres when THERMOGRAPH_DATABASE_URL is a postgresql+asyncpg URL; unset ⇒ the old SQLite behavior (this is how the test suite stays on SQLite — no Postgres needed in CI).
• db runs pg_duckdb (deploy/db/Dockerfile.db, genuine PG18): the app's parquet cache is mounted read-only at /parquet, so you can query it directly, e.g. SELECT r['date'], r['tmax'] FROM read_parquet('/parquet/cache/*.parquet') r.

Connection model

Per worker (4 workers): a read-write asyncpg engine + a read-only asyncpg engine (default_transaction_read_only=on, used by the pure-GET endpoints), plus one sync psycopg engine for the notifier thread. Single Postgres primary — the RO engine is a guardrail, not a replica.

Prod cutover steps (maintenance window)

Prereq: the deploy user can run docker. Secrets in /etc/thermograph.env must include POSTGRES_PASSWORD, and pinned THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY, THERMOGRAPH_COOKIE_SECURE=1.

1. Ship the stack, don't cut over yet. Deploy the branch; docker compose build. Bring up only the DB: docker compose up -d db. Add a nightly pg_dump backup.
2. Freeze + back up. Stop the old app (hard stop — no writes). Back up the live data/accounts.sqlite + -wal/-shm (this is the rollback artifact).
3. Build the schema. docker compose run --rm app alembic upgrade head (or let the app entrypoint do it on first start — but do the data copy before real traffic).
4. Copy the accounts data:

   docker compose run --rm \
     -e THERMOGRAPH_DATABASE_URL="postgresql+asyncpg://thermograph:$POSTGRES_PASSWORD@db:5432/thermograph" \
     -v /opt/thermograph/data/accounts.sqlite:/src.sqlite:ro \
     app python migrate_accounts_to_pg.py --sqlite /src.sqlite
   

   It copies user → subscription/push_subscription/pending_digest → notification, preserving PKs, skips access_token (everyone re-logins once), and resets the integer-PK sequences (skipping that = a runtime PK collision). Verify the printed per-table counts against the old DB.
5. Start serving: docker compose up -d (Caddy already proxies 127.0.0.1:8137).
6. Smoke test: login, list/create/delete a subscription, one notifier tick, a cached calendar/day request (derived store on PG), the metrics dashboard.
7. Keep the frozen accounts.sqlite as rollback for a few days; keep PG backed up. Enable pg_duckdb on the (already-initialized) volume once, by hand: docker compose exec db psql -U thermograph -d thermograph -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;'

Rollback

Redeploy the previous SQLite release (or point THERMOGRAPH_DATABASE_URL back to unset/SQLite and restart). Clean only during/immediately after the window — once real users write to Postgres, those writes are lost on rollback (the copy is one-way), so keep the window short and writes frozen during the copy.

Notes

• The two deploy/migrations/*.sql files are superseded by Alembic on Postgres (they were SQLite-specific ALTER TABLE column-adds for the old prod DB).
• The PG18 image's data dir is the versioned …/18/docker subdir; the compose volume is mounted at the parent /var/lib/postgresql so it persists without the initdb permission issue that pinning PGDATA to the mountpoint can trigger.