thermograph/infra/deploy/backup/README.md

Off-box backups → Contabo Object Storage

Durable, encrypted, off-box copies of the state that would otherwise die with a single VPS. Driven by .forgejo/workflows/ops-cron.yml (03:00 UTC daily + workflow_dispatch).

What is backed up

Prefix in bucket era5-thermograph	Source	Job	Host / secrets	Retention off-box
backups/db/prod/	prod's thermograph database (pg_dump --format=custom)	backup	vps2, VPS2_SSH_*	30 days
backups/db/beta/	beta's thermograph_beta database, same shared instance	backup	vps2, VPS2_SSH_*	30 days
backups/forgejo/db/	Forgejo Postgres (forgejo_db)	forgejo-backup	vps1, VPS1_SSH_*	30 days
backups/forgejo/data/	Forgejo data volume (repos/LFS/config)	forgejo-backup	vps1, VPS1_SSH_*	30 days

Both application databases live on the ONE shared TimescaleDB instance on vps2 now (separate roles/databases, not separate boxes) — the backup job dumps each by name so beta isn't silently left uncovered. Forgejo moved to vps1 with the vps1/vps2 rename (it used to be co-located with beta on the box now called vps2's predecessor); its backup targets vps1 accordingly. See ops-cron.yml's own header comment for the history: an earlier revision keyed these secrets by environment name rather than host, which is what let "the prod backup" silently dump the wrong box for a while.

Everything is streamed through age encryption before upload — encrypted to the vault recipient age1xx4dz…, so the private key each host already has at /etc/thermograph/age.key (and the operator's ~/.config/sops/age/keys.txt) decrypts it. Nothing plaintext leaves the box.

Access layer

• S3 endpoint/bucket/keys are Forgejo Actions secrets (S3_ENDPOINT, S3_BUCKET, S3_ACCESS_KEY, S3_SECRET_KEY) — the CI job's credential home (also mirrored into the SOPS vault deploy/secrets/{prod,beta}.yaml for host-side use). Contabo needs path-style addressing (force_path_style=true, provider=Other, region=default).
• rclone + age must be installed on vps1 and vps2 (one-time: apt-get install -y rclone age). The prod/beta job on vps2 apt-installs them if missing; check whether the Forgejo-backup job's user on vps1 has the sudo to do the same, or whether they need pre-installing there.

Restore (DR runbook)

Prereq on the restoring host: rclone + age + the age key at /etc/thermograph/age.key (or the operator key), and rclone configured for the archive: remote (env vars RCLONE_CONFIG_ARCHIVE_* or /etc/rclone/rclone.conf).

Prod / beta database

Both restore into the SAME shared instance on vps2 (<db-container> is that one instance's container/task, resolved the same way dbq.sh does) — only the bucket prefix and the target database/role differ:

# list available encrypted dumps
rclone ls archive:era5-thermograph/backups/db/prod/
rclone ls archive:era5-thermograph/backups/db/beta/
# download newest, decrypt, restore into a fresh db
OBJ=archive:era5-thermograph/backups/db/prod/<name>.dump.age
rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \
  | docker exec -i <db-container> pg_restore -U thermograph -d thermograph --clean --if-exists
# beta: same instance, its own role/database
OBJ=archive:era5-thermograph/backups/db/beta/<name>.dump.age
rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \
  | docker exec -i <db-container> pg_restore -U thermograph -d thermograph_beta --clean --if-exists

Forgejo (restore on vps1)

# database
rclone cat archive:era5-thermograph/backups/forgejo/db/<name>.dump.age \
  | sudo age -d -i /etc/thermograph/age.key \
  | docker exec -i forgejo_db sh -c 'PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U $POSTGRES_USER -d $POSTGRES_DB --clean --if-exists'
# data volume (repos/lfs/config) — restore into the volume with Forgejo stopped
rclone cat archive:era5-thermograph/backups/forgejo/data/<name>.tar.age \
  | sudo age -d -i /etc/thermograph/age.key \
  | docker run --rm -i -v forgejo_forgejo_data:/data busybox tar -C /data -xf -

Verify a backup is restorable (quick)

rclone cat archive:era5-thermograph/backups/db/prod/<name>.dump.age \
  | sudo age -d -i /etc/thermograph/age.key | head -c 5   # -> PGDMP = valid custom-format dump

Manual run

workflow_dispatch the "Ops cron (backup + IndexNow)" workflow, or run the same age | rclone rcat pipeline by hand (see the workflow script).