Two bugs surfaced wiring dev auto-deploy:
1. Sourcing .image-tags.env clobbered the caller's incoming *_IMAGE_TAG. The
first backend-only deploy persists FRONTEND_IMAGE_TAG=local (sibling unknown);
the next frontend deploy then sourced that and pulled :local -> 'manifest
unknown'. Now the incoming env is captured before sourcing and re-applied.
2. Health-checked services via a host-port curl, but the dev overlay leaves the
frontend port unpublished (reached via the backend proxy) -> false failure.
Now polls each container's own HEALTHCHECK status via docker inspect.
The CI/SSH deploy paths (deploy.yml/deploy-prod.yml over SSH, deploy-dev on the
LAN runner) don't pass REGISTRY_TOKEN and rely on the host already being
docker-logged-in; an unconditional login with an empty token aborts deploy.sh
under set -e. Skip the login when no token is set and trust the host cred;
pull still fails loudly on a real auth problem.
backend and frontend now reference their own images (BACKEND_IMAGE_PATH/TAG,
FRONTEND_IMAGE_PATH/TAG) instead of the shared emi/thermograph/app. deploy.sh
takes SERVICE=backend|frontend|all, rolls just that service with --no-deps,
persists each service's live tag in deploy/.image-tags.env so a single-service
deploy never disturbs the sibling, and fixes the old IMAGE_TAG:? guard that
made every workflow deploy hard-fail. Health-checks per service (8137/8080).
Claude-Session: https://claude.ai/code/session_01RdARHDJaYC1wSQRTYM6t3Z
Content-change pass following the extraction from the app monorepo (this repo
now stands alone, sourced via git filter-repo to preserve history):
- terraform/variables.tf, secrets.tf, modules/thermograph-host: remove every
app-secret Terraform variable (postgres_password, auth_secret, VAPID keys,
registry_token, Discord/SMTP creds, ...) and the random_password/random_id
generators. The SOPS+age vault (deploy/secrets/*.yaml) is now the sole
source of app secrets, rendered at deploy time by deploy/render-secrets.sh;
Terraform renders only a non-secret /etc/thermograph-topology.env (sizing,
routing) via the renamed thermograph-topology.env.tftpl template.
- hosts gains a required app_image_tag field: the host's own checkout is now
this infra repo, not the app repo, so there is no "current commit" to
derive an image tag from — every host pins one explicitly. repo_url now
points at this repo (private; typically needs an embedded read token).
- deploy.sh: IMAGE_TAG is now required from the environment instead of
derived via `git rev-parse HEAD` of the (now infra-repo) checkout, which
would have silently resolved to the wrong or a nonexistent tag.
- New terraform/modules/gcp-host: creates a GCE VM + minimal VPC/firewall
only, then feeds its IP into the same thermograph-host module every
SSH-managed host already uses — one provisioning path regardless of how a
host came to exist. var.gcp_hosts defaults to {}, so no google_* resource
is planned and the provider is never invoked without it (verified: plan
and validate succeed with no GCP credentials configured).
- terraform/README.md, ACCESS.md (renamed from INFRA.md), README.md: updated
for the new secrets model, the GCP scaffold, and this repo's own identity.
Verified: terraform fmt/validate/init clean; plan succeeds against realistic
dummy hosts (prod+beta shape) and against a populated gcp_hosts entry (plans
6 resources with no live credentials, confirming the composition wires
correctly end to end).
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
The deploy scripts pulled code and restarted but never applied the
hand-written SQL migrations in deploy/migrations/, so column additions to
the long-lived accounts DB had to be run by hand.
Add deploy/migrate-db.py, an idempotent runner that resolves the accounts
DB the same way the app does (THERMOGRAPH_ACCOUNTS_DB or
<repo>/data/accounts.sqlite), applies any files not yet recorded in a
schema_migrations table, and:
- backs the DB up before touching it;
- baselines a fresh DB (no file, or no user table) rather than ALTERing
a table create_all() will build with the current schema on next start;
- tolerates an already-present column/index (hand-applied or model-built)
by recording it instead of failing.
Wire it into deploy.sh (prod) and deploy-dev.sh (dev) just before the
service restarts, so new code never queries a column an older DB lacks.
No env or systemd changes: the migration runs as the deploy user, who owns
the data dir.
Claude-Session: https://claude.ai/code/session_01XXxmNFy9cZ6Gh8Y9thZn62
Hook the production deploy (deploy/deploy.sh) to ping IndexNow after the health
check passes, so new/changed pages reach Bing/DuckDuckGo/Yandex without a manual
run. Best-effort — wrapped so it can never fail a deploy — and it sources
/etc/thermograph.env so its key matches the one the running service serves.
To avoid re-blasting ~14k URLs on every code push, indexnow.py gains a
--if-changed mode gated by a signature of the URL set (persisted to
data/indexnow_state.txt): code-only deploys skip; a deploy that adds or removes a
city submits. `make indexnow` still forces a full submit.
The population-ranked global top-500 skewed to Asian megacities and missed
high-English-search-demand cities. gen_cities.py now tops up with the top ~250
cities from English-speaking countries (US/GB/CA/AU/NZ/IE/ZA) not already in the
global set, so US coverage goes 13->146, GB 2->42, CA 3->29, etc. (Seattle, Boston,
Manchester, Melbourne, Auckland, Dublin, ...). cities.json regenerated to 750.
Both deploy scripts now launch warm_cities.py automatically after the health check,
detached (dev: a systemd --user transient unit; prod: setsid/nohup), so the city
pages serve from cache without a manual step; idempotent, so only the first deploy
does the full warm. DEPLOY.md updated.
