Scale DB tuning from DB_MEMORY; order Docker after the rclone mount (#226)

Two prod-readiness hardening changes:

DB tuning scales with the container budget. Replace the fixed 8 GB
20-tuning.sql with 20-tuning.sh, which derives shared_buffers (25%),
effective_cache_size (75%), work_mem, maintenance_work_mem and
duckdb.max_memory (50%) from the DB_MEMORY the compose db service now passes in.
The ratios reproduce the historical 8 GB tuning exactly and scale linearly, so
the 48 GB prod box (db_memory 16g) gets shared_buffers 4 GB / duckdb 8 GB with
no separate edit. Beta/local (8g default) are unchanged. Docs that told
operators to raise the tuning by hand are updated.

Boot ordering for the self-hosted archive. On an openmeteo host, install a
docker.service drop-in (Wants/After rclone-om.service) so Docker starts after
the object-storage mount is ready on every boot — the restart-policy containers
never bind an empty mount point. rclone-om is Type=notify, so After waits for
the mount to actually be ready. Cleaned up when openmeteo is toggled off.

deploy/db/init/20-tuning.sh

@@ -0,0 +1,64 @@
+#!/bin/bash
+# Postgres memory / performance tuning, scaled to the container's DB_MEMORY budget so
+# the same init serves every host (beta 8g; prod 16g on the 48 GB box) with no
+# hardcoding. Runs once on a fresh data volume from /docker-entrypoint-initdb.d, after
+# 10-parquet.sql enables pg_duckdb. Settings are written via ALTER SYSTEM (persisted to
+# postgresql.auto.conf); the container's post-init restart brings restart-only settings
+# (shared_buffers, …) into effect. Init scripts do NOT re-run on an existing volume — to
+# re-tune later, set DB_MEMORY and run this by hand, then restart:
+#   docker compose exec -e DB_MEMORY=16g db bash /docker-entrypoint-initdb.d/20-tuning.sh
+#   docker compose restart db
+set -euo pipefail
+
+# Parse DB_MEMORY ("16g" / "8192m" / plain MB) into whole MB; default + floor at 8 GB.
+budget="${DB_MEMORY:-8g}"
+num="${budget//[!0-9]/}"
+num="${num:-8}"
+unit="$(printf '%s' "$budget" | tr -dc '[:alpha:]' | tr '[:upper:]' '[:lower:]')"
+case "$unit" in
+  g | gb) mb=$((num * 1024)) ;;
+  m | mb | "") mb="$num" ;;
+  *) mb=8192 ;;
+esac
+if [ "$mb" -lt 1024 ]; then mb=8192; fi
+
+# Derive settings from the budget. The ratios reproduce the historical 8 GB tuning
+# (shared_buffers 2 GB, effective_cache_size 6 GB, duckdb 4 GB, work_mem 64 MB,
+# maintenance_work_mem 512 MB) and scale linearly on a bigger box.
+shared_buffers=$((mb / 4))      # 25% — the shared page cache
+effective_cache=$((mb * 3 / 4)) # 75% — planner's view of total cache (PG + OS)
+duckdb_mem=$((mb / 2))          # 50% — pg_duckdb ceiling for parquet processing
+work_mem=$((mb / 128))          # ~64 MB at 8 GB (per-operation; kept modest)
+if [ "$work_mem" -lt 16 ]; then work_mem=16; fi
+maint_mem=$((mb / 16)) # 512 MB at 8 GB — index builds / VACUUM
+
+echo "[tuning] DB_MEMORY=${budget} -> ${mb}MB: shared_buffers=${shared_buffers}MB" \
+  "effective_cache_size=${effective_cache}MB work_mem=${work_mem}MB" \
+  "maintenance_work_mem=${maint_mem}MB duckdb.max_memory=${duckdb_mem}MB"
+
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<SQL
+-- Caching: the shared page cache, and the planner's view of total cache (PG + OS).
+ALTER SYSTEM SET shared_buffers = '${shared_buffers}MB';
+ALTER SYSTEM SET effective_cache_size = '${effective_cache}MB';
+
+-- Processing: per-operation sort/hash memory, and maintenance (index builds, VACUUM).
+ALTER SYSTEM SET work_mem = '${work_mem}MB';
+ALTER SYSTEM SET maintenance_work_mem = '${maint_mem}MB';
+
+-- Write throughput: fewer, larger checkpoints.
+ALTER SYSTEM SET wal_buffers = '16MB';
+ALTER SYSTEM SET min_wal_size = '1GB';
+ALTER SYSTEM SET max_wal_size = '4GB';
+ALTER SYSTEM SET checkpoint_completion_target = 0.9;
+
+-- SSD-friendly planner + IO concurrency.
+ALTER SYSTEM SET random_page_cost = 1.1;
+ALTER SYSTEM SET effective_io_concurrency = 200;
+
+-- Room for parallel scans/aggregates on the bigger analytic queries.
+ALTER SYSTEM SET max_parallel_workers_per_gather = 2;
+
+-- DuckDB (pg_duckdb) memory ceiling for parquet processing. pg_duckdb is preloaded
+-- (shared_preload_libraries), so this GUC exists at ALTER SYSTEM time.
+ALTER SYSTEM SET duckdb.max_memory = '${duckdb_mem}MB';
+SQL

deploy/db/init/20-tuning.sql

@@ -1,31 +0,0 @@
--- Memory / performance tuning for an ~8 GB budget. Applied once, on a fresh data
--- volume, via ALTER SYSTEM (persists to postgresql.auto.conf); the container's
--- post-init restart brings the restart-only settings (shared_buffers, …) into
--- effect. To re-apply on an existing volume, run these by hand and restart:
---   docker compose exec db psql -U thermograph -d thermograph -f /docker-entrypoint-initdb.d/20-tuning.sql
---   docker compose restart db
-
--- Caching: the shared page cache, and the planner's view of total cache (PG + OS).
-ALTER SYSTEM SET shared_buffers = '2GB';
-ALTER SYSTEM SET effective_cache_size = '6GB';
-
--- Processing: per-operation sort/hash memory, and maintenance (index builds, VACUUM).
-ALTER SYSTEM SET work_mem = '64MB';
-ALTER SYSTEM SET maintenance_work_mem = '512MB';
-
--- Write throughput: fewer, larger checkpoints.
-ALTER SYSTEM SET wal_buffers = '16MB';
-ALTER SYSTEM SET min_wal_size = '1GB';
-ALTER SYSTEM SET max_wal_size = '4GB';
-ALTER SYSTEM SET checkpoint_completion_target = 0.9;
-
--- SSD-friendly planner + IO concurrency.
-ALTER SYSTEM SET random_page_cost = 1.1;
-ALTER SYSTEM SET effective_io_concurrency = 200;
-
--- Some room for parallel scans/aggregates on the bigger analytic queries.
-ALTER SYSTEM SET max_parallel_workers_per_gather = 2;
-
--- DuckDB (pg_duckdb) memory ceiling for parquet processing. pg_duckdb is
--- preloaded, so this GUC exists at ALTER SYSTEM time.
-ALTER SYSTEM SET duckdb.max_memory = '4GB';

deploy/openmeteo/README.md

@@ -84,6 +84,23 @@ local disk after first read so repeat range reads don't go back to the bucket,
 and the size cap keeps that cache inside the 400 GB disk budget by evicting
 cold data.
 
+**Order Docker after the mount (reboots).** So the `restart: unless-stopped`
+containers never start against an empty mount point, make Docker wait for the
+mount (`rclone-om` is `Type=notify`, so this waits until the mount is actually
+ready). Terraform installs this automatically; for a manual setup:
+
+```sh
+sudo install -d /etc/systemd/system/docker.service.d
+printf '[Unit]\nWants=rclone-om.service\nAfter=rclone-om.service\n' \
+  | sudo tee /etc/systemd/system/docker.service.d/10-wait-rclone.conf
+sudo systemctl daemon-reload
+```
+
+(If the mount ever drops and remounts *while* the containers are running, the
+existing bind won't see the new mount — restart the Open-Meteo containers to
+re-bind. The app stays safe either way: it rejects a short/empty archive and
+falls back to NASA rather than caching a gap. See `make om-up`.)
+
 ## 4. Point the overlay at the mount
 
 `OM_DATA_DIR` is read from the environment at `docker compose` time; in prod

docker-compose.yml

@@ -21,6 +21,10 @@ services:
       POSTGRES_USER: thermograph
       POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}
       POSTGRES_DB: thermograph
+      # The init tuning script (deploy/db/init/20-tuning.sh) scales the Postgres +
+      # DuckDB memory GUCs from this budget, matching the mem_limit below. One knob
+      # per host: Terraform sets DB_MEMORY (prod 16g), local/beta default 8g.
+      DB_MEMORY: ${DB_MEMORY:-8g}
     volumes:
       # Mount the volume at the PARENT of the data dir and let the PG18 image use
       # its own versioned subdir (/var/lib/postgresql/18/docker). Pinning PGDATA
@@ -36,11 +40,12 @@ services:
       interval: 5s
       timeout: 5s
       retries: 10
-    # Give Postgres room to cache + process against ~8 GB. mem_limit is the ceiling;
-    # the actual budget is tuned in deploy/db/init/20-tuning.sql (shared_buffers 2GB,
-    # effective_cache_size 6GB, work_mem/maintenance_work_mem, plus duckdb.max_memory
-    # 4GB for parquet queries). shm_size backs parallel-query shared memory (the 64MB
-    # docker default is too small once shared_buffers/parallelism grow).
+    # Give Postgres room to cache + process. mem_limit is the hard ceiling; the actual
+    # budget is tuned in deploy/db/init/20-tuning.sh, which scales shared_buffers (25%),
+    # effective_cache_size (75%), work_mem, maintenance_work_mem and duckdb.max_memory
+    # (50%) from DB_MEMORY — so raising DB_MEMORY raises both the cap and the tuning
+    # together. shm_size backs parallel-query shared memory (the 64MB docker default is
+    # too small once shared_buffers/parallelism grow).
     # Sized via env (Terraform sets DB_CPUS/DB_MEMORY per host); defaults match the
     # historical 2 CPU / 8 GB budget so a plain `docker compose up` is unchanged.
     mem_limit: ${DB_MEMORY:-8g}

terraform/README.md

@@ -41,10 +41,12 @@ A change to the rendered env, the compose files, the branch, or the sizing flips
 `docker-compose.yml` reads `WORKERS`, `APP_CPUS`, `DB_CPUS`, and `DB_MEMORY` from the
 environment (defaults `4 / 4 / 2 / 8g`, identical to before). Terraform sets them per
 host through `/etc/thermograph.env`, so the big prod box can run larger caps without a
-compose edit. **Note:** this only sizes the *containers*. The Postgres *internal* memory
-budget (`shared_buffers`, `effective_cache_size`, `work_mem`, `duckdb.max_memory`) lives
-in `deploy/db/init/20-tuning.sql` and must be raised **separately** to exploit the 48 GB
-prod box — those settings apply on a fresh DB volume and are not driven by Terraform.
+compose edit. The Postgres *internal* memory budget (`shared_buffers`,
+`effective_cache_size`, `work_mem`, `duckdb.max_memory`) is derived from the same
+`DB_MEMORY` by `deploy/db/init/20-tuning.sh` — so raising `db_memory` scales the
+container cap and the tuning together (prod 16g → shared_buffers 4 GB, duckdb 8 GB). The
+tuning applies on a fresh DB volume; on an existing volume re-run it by hand (see the
+script header).
 
 ## Prerequisites
 

terraform/modules/thermograph-host/main.tf

@@ -184,6 +184,12 @@ resource "null_resource" "host" {
         set -eu
         if [ "${var.openmeteo}" != "true" ]; then
           rm -f /tmp/thermograph.rclone.conf /tmp/rclone-om.service
+          # Undo any prior openmeteo setup so a toggled-off host doesn't wait on a mount.
+          if [ -e /etc/systemd/system/docker.service.d/10-wait-rclone.conf ]; then
+            sudo rm -f /etc/systemd/system/docker.service.d/10-wait-rclone.conf
+            sudo systemctl disable --now rclone-om >/dev/null 2>&1 || true
+            sudo systemctl daemon-reload
+          fi
           echo "[${var.name}] openmeteo: disabled"
           exit 0
         fi
@@ -214,6 +220,13 @@ resource "null_resource" "host" {
           sudo systemctl status rclone-om --no-pager || true
           exit 1
         fi
+        # Order Docker after the mount on every boot, so the restart-policy containers
+        # never bind an empty mount point. rclone-om is Type=notify, so `After` waits
+        # until the mount is actually ready — not merely that the unit was launched.
+        sudo install -d -m 0755 /etc/systemd/system/docker.service.d
+        printf '[Unit]\nWants=rclone-om.service\nAfter=rclone-om.service\n' \
+          | sudo tee /etc/systemd/system/docker.service.d/10-wait-rclone.conf >/dev/null
+        sudo systemctl daemon-reload
         echo "[${var.name}] openmeteo: mounted at ${var.om_data_dir}"
       EOT
     ]

terraform/terraform.tfvars.example

@@ -19,8 +19,8 @@ hosts = {
     domain               = "thermograph.org" # Caddy TLS in front, app on loopback
     compose_files        = ["docker-compose.yml"]
     app_dir              = "/opt/thermograph"
-    # Sized up for the big box — tune freely. Also raise the Postgres internal budget
-    # in deploy/db/init/20-tuning.sql (shared_buffers etc.) to actually use the RAM.
+    # Sized up for the big box — tune freely. The Postgres internal budget scales from
+    # db_memory automatically (deploy/db/init/20-tuning.sh); no separate tuning edit.
     workers   = 8
     app_cpus  = 8
     db_cpus   = 4

terraform/variables.tf

@@ -13,8 +13,9 @@
 #
 # Resource sizing (workers / app_cpus / db_cpus / db_memory) defaults to the historical
 # 4 / 4 / 2 / 8g budget (right for beta). The prod box is 48 GB / 12 cores — raise these
-# there (the example uses app_cpus 8, db_cpus 4, db_memory "16g"); also raise the
-# Postgres *internal* budget in deploy/db/init/20-tuning.sql to actually exploit the RAM.
+# there (the example uses app_cpus 8, db_cpus 4, db_memory "16g"). The Postgres internal
+# budget scales from db_memory automatically (deploy/db/init/20-tuning.sh), so no
+# separate tuning edit is needed to exploit the RAM.
 variable "hosts" {
   description = "Map of hosts to manage, keyed by a short name (e.g. \"prod\", \"dev\")."
   type = map(object({