Containerize the app and move the databases to PostgreSQL 18 (#220)
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
|
|
|
"""The accounts DB engine layer: dialect selection and the read/read-write split.
|
|
|
|
|
|
|
|
|
|
These run on the SQLite fallback (no THERMOGRAPH_DATABASE_URL in the test env);
|
|
|
|
|
the Postgres path (asyncpg RO/RW engines) is exercised by the docker-compose stack,
|
|
|
|
|
not the unit suite, so tests stay Postgres-free and fast."""
|
|
|
|
|
import asyncio
|
|
|
|
|
|
|
|
|
|
from sqlalchemy import select
|
|
|
|
|
|
|
|
|
|
from accounts import db
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_defaults_to_sqlite_when_no_database_url():
|
|
|
|
|
assert db.IS_POSTGRES is False
|
|
|
|
|
# Both engines exist; on SQLite the RO engine is an alias of the RW one.
|
|
|
|
|
assert db.async_engine is not None
|
|
|
|
|
assert db.async_engine_ro is db.async_engine
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_read_and_write_session_makers_are_distinct_dependencies():
|
|
|
|
|
# Distinct FastAPI deps so pure-read endpoints can opt into the RO engine.
|
|
|
|
|
assert db.get_async_session is not db.get_read_session
|
|
|
|
|
assert callable(db.get_async_session) and callable(db.get_read_session)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_sync_url_derivation():
|
|
|
|
|
assert db._sync_url("postgresql+asyncpg://u:p@h:5432/d") == "postgresql+psycopg://u:p@h:5432/d"
|
|
|
|
|
assert db._sync_url("sqlite+aiosqlite:////tmp/x.sqlite") == "sqlite:////tmp/x.sqlite"
|
|
|
|
|
|
|
|
|
|
|
2026-07-23 04:41:26 +00:00
|
|
|
def test_postgres_pool_sizing_and_sync_timeouts():
|
|
|
|
|
"""The Postgres engine-construction kwargs, defined at module scope so they're
|
|
|
|
|
testable without a real Postgres connection (the engines built from them are
|
|
|
|
|
only actually constructed when IS_POSTGRES, exercised by the docker-compose
|
|
|
|
|
stack -- see the module docstring). pool_size=1/max_overflow=1 (the old
|
|
|
|
|
setting) let a 3rd concurrent request per worker wait out pool_timeout for a
|
|
|
|
|
slot; both async engines need real headroom, and the sync engine (the
|
|
|
|
|
notifier's, driven off-event-loop) must bound connect + statement time so a
|
|
|
|
|
wedged Postgres can't hang it forever under its leader flock."""
|
|
|
|
|
assert db._ASYNC_POOL_KWARGS["pool_size"] >= 5
|
|
|
|
|
assert db._ASYNC_POOL_KWARGS["max_overflow"] >= 5
|
|
|
|
|
|
|
|
|
|
assert db._SYNC_POOL_KWARGS["pool_size"] < db._ASYNC_POOL_KWARGS["pool_size"]
|
|
|
|
|
connect_args = db._SYNC_POOL_KWARGS["connect_args"]
|
|
|
|
|
assert connect_args["connect_timeout"] > 0
|
|
|
|
|
assert "statement_timeout" in connect_args["options"]
|
|
|
|
|
|
|
|
|
|
# Build a real (unconnected -- SQLAlchemy engines are lazy) engine from these
|
|
|
|
|
# exact kwargs to confirm they're accepted by the psycopg dialect.
|
|
|
|
|
import sqlalchemy
|
|
|
|
|
engine = sqlalchemy.create_engine("postgresql+psycopg://u:p@h:5432/d", **db._SYNC_POOL_KWARGS)
|
|
|
|
|
assert engine.pool.size() == db._SYNC_POOL_KWARGS["pool_size"]
|
|
|
|
|
engine.dispose()
|
|
|
|
|
|
|
|
|
|
|
Containerize the app and move the databases to PostgreSQL 18 (#220)
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
|
|
|
def test_read_session_yields_a_working_session():
|
|
|
|
|
# On SQLite the read session is fully usable (no read-only enforcement); this
|
|
|
|
|
# just confirms the dependency wiring resolves to a live AsyncSession.
|
|
|
|
|
db.Base.metadata.create_all(db.sync_engine)
|
|
|
|
|
|
|
|
|
|
async def _use():
|
|
|
|
|
async for session in db.get_read_session():
|
|
|
|
|
return (await session.execute(select(1))).scalar_one()
|
|
|
|
|
|
|
|
|
|
assert asyncio.run(_use()) == 1
|