Jinemi/thermograph
4
0
Fork 0
Code Issues 2 Pull requests 10 Projects Releases Packages Wiki Activity Actions 1
thermograph/infra/deploy/backup/README.md
Emi Griffith 6086b40fe6
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 7s
Details
PR build (required check) / changes (pull_request) Successful in 8s
Details
PR build (required check) / build-backend (pull_request) Has been skipped
Details
PR build (required check) / build-frontend (pull_request) Has been skipped
Details
PR build (required check) / validate-observability (pull_request) Has been skipped
Details
PR build (required check) / gate (pull_request) Successful in 3s
Details
Add off-box encrypted backups to Contabo Object Storage 
Prod DB dumps and Forgejo (db + data volume) are now copied off-box to the
era5-thermograph S3 bucket, age-encrypted to the vault recipient so the host
age key at /etc/thermograph/age.key decrypts them for restore. Nothing plaintext
leaves the box; no local intermediate (streamed via age | rclone rcat).

ops-cron: the prod pg_dump job gains an off-box push; a new forgejo-backup job
dumps Forgejo's Postgres + tars its data volume on beta (the git host had NO
backup at all). 30-day off-box retention on both. S3 creds are Forgejo Actions
secrets (S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY), mirrored into the SOPS vault
(prod.yaml/beta.yaml) for host-side use. Contabo needs path-style addressing.

Adds deploy/backup/README.md with the DR restore runbook.
2026-07-23 13:42:09 -07:00

3 KiB
Raw Blame History

Off-box backups → Contabo Object Storage

Durable, encrypted, off-box copies of the state that would otherwise die with a single VPS. Driven by .forgejo/workflows/ops-cron.yml (03:00 UTC daily + workflow_dispatch).

What is backed up

Prefix in bucket era5-thermograph Source Job Retention off-box
backups/db/prod/ prod Postgres/TimescaleDB (pg_dump --format=custom) backup (prod, PROD_SSH_*) 30 days
backups/forgejo/db/ Forgejo Postgres (forgejo_db) forgejo-backup (beta, SSH_*) 30 days
backups/forgejo/data/ Forgejo data volume (repos/LFS/config) forgejo-backup 30 days

Everything is streamed through age encryption before upload — encrypted to the vault recipient age1xx4dz…, so the private key each host already has at /etc/thermograph/age.key (and the operator's ~/.config/sops/age/keys.txt) decrypts it. Nothing plaintext leaves the box.

Access layer

  • S3 endpoint/bucket/keys are Forgejo Actions secrets (S3_ENDPOINT, S3_BUCKET, S3_ACCESS_KEY, S3_SECRET_KEY) — the CI job's credential home (also mirrored into the SOPS vault deploy/secrets/{prod,beta}.yaml for host-side use). Contabo needs path-style addressing (force_path_style=true, provider=Other, region=default).
  • rclone + age must be installed on prod and beta (one-time: apt-get install -y rclone age). The prod job apt-installs them if missing; the beta user has no sudo, so they are pre-installed there.

Restore (DR runbook)

Prereq on the restoring host: rclone + age + the age key at /etc/thermograph/age.key (or the operator key), and rclone configured for the archive: remote (env vars RCLONE_CONFIG_ARCHIVE_* or /etc/rclone/rclone.conf).

Prod database

# list available encrypted dumps
rclone ls archive:era5-thermograph/backups/db/prod/
# download newest, decrypt, restore into a fresh db
OBJ=archive:era5-thermograph/backups/db/prod/<name>.dump.age
rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \
  | docker exec -i <db-container> pg_restore -U thermograph -d thermograph --clean --if-exists

Forgejo

# database
rclone cat archive:era5-thermograph/backups/forgejo/db/<name>.dump.age \
  | sudo age -d -i /etc/thermograph/age.key \
  | docker exec -i forgejo_db sh -c 'PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U $POSTGRES_USER -d $POSTGRES_DB --clean --if-exists'
# data volume (repos/lfs/config) — restore into the volume with Forgejo stopped
rclone cat archive:era5-thermograph/backups/forgejo/data/<name>.tar.age \
  | sudo age -d -i /etc/thermograph/age.key \
  | docker run --rm -i -v forgejo_forgejo_data:/data busybox tar -C /data -xf -

Verify a backup is restorable (quick)

rclone cat archive:era5-thermograph/backups/db/prod/<name>.dump.age \
  | sudo age -d -i /etc/thermograph/age.key | head -c 5   # -> PGDMP = valid custom-format dump

Manual run

workflow_dispatch the "Ops cron (backup + IndexNow)" workflow, or run the same age | rclone rcat pipeline by hand (see the workflow script).