thermograph/infra/deploy/backup/README.md

# Off-box backups → Contabo Object Storage

Durable, encrypted, off-box copies of the state that would otherwise die with a
single VPS. Driven by `.forgejo/workflows/ops-cron.yml` (03:00 UTC daily +
`workflow_dispatch`).

## What is backed up

| Prefix in bucket `era5-thermograph` | Source | Job | Host / secrets | Retention off-box |
|---|---|---|---|---|
| `backups/db/prod/` | prod's `thermograph` database (`pg_dump --format=custom`) | `backup` | vps2, `VPS2_SSH_*` | 30 days |
| `backups/db/beta/` | beta's `thermograph_beta` database, same shared instance | `backup` | vps2, `VPS2_SSH_*` | 30 days |
| `backups/forgejo/db/` | Forgejo Postgres (`forgejo_db`) | `forgejo-backup` | vps1, `VPS1_SSH_*` | 30 days |
| `backups/forgejo/data/` | Forgejo data volume (repos/LFS/config) | `forgejo-backup` | vps1, `VPS1_SSH_*` | 30 days |

Both application databases live on the ONE shared TimescaleDB instance on
vps2 now (separate roles/databases, not separate boxes) — the `backup` job
dumps each by name so beta isn't silently left uncovered. Forgejo moved to
**vps1** with the vps1/vps2 rename (it used to be co-located with beta on the
box now called vps2's predecessor); its backup targets vps1 accordingly. See
`ops-cron.yml`'s own header comment for the history: an earlier revision keyed
these secrets by environment name rather than host, which is what let "the
prod backup" silently dump the wrong box for a while.

Everything is streamed through **`age`** encryption before upload — encrypted to the
vault recipient `age1xx4dz…`, so the private key each host already has at
`/etc/thermograph/age.key` (and the operator's `~/.config/sops/age/keys.txt`)
decrypts it. Nothing plaintext leaves the box.

## Access layer
- S3 endpoint/bucket/keys are **Forgejo Actions secrets** (`S3_ENDPOINT`,
  `S3_BUCKET`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`) — the CI job's credential home
  (also mirrored into the SOPS vault `deploy/secrets/{prod,beta}.yaml` for host-side
  use). Contabo needs **path-style** addressing (`force_path_style=true`,
  `provider=Other`, `region=default`).
- `rclone` + `age` must be installed on vps1 and vps2 (one-time: `apt-get install -y
  rclone age`). The prod/beta job on vps2 apt-installs them if missing; check
  whether the Forgejo-backup job's user on vps1 has the sudo to do the same, or
  whether they need pre-installing there.

## Restore (DR runbook)

Prereq on the restoring host: `rclone` + `age` + the age key at
`/etc/thermograph/age.key` (or the operator key), and rclone configured for the
`archive:` remote (env vars `RCLONE_CONFIG_ARCHIVE_*` or `/etc/rclone/rclone.conf`).

### Prod / beta database

Both restore into the SAME shared instance on vps2 (`<db-container>` is that
one instance's container/task, resolved the same way `dbq.sh` does) — only
the bucket prefix and the target database/role differ:

```sh
# list available encrypted dumps
rclone ls archive:era5-thermograph/backups/db/prod/
rclone ls archive:era5-thermograph/backups/db/beta/
# download newest, decrypt, restore into a fresh db
OBJ=archive:era5-thermograph/backups/db/prod/<name>.dump.age
rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \
  | docker exec -i <db-container> pg_restore -U thermograph -d thermograph --clean --if-exists
# beta: same instance, its own role/database
OBJ=archive:era5-thermograph/backups/db/beta/<name>.dump.age
rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \
  | docker exec -i <db-container> pg_restore -U thermograph -d thermograph_beta --clean --if-exists
```

### Forgejo (restore on vps1)
```sh
# database
rclone cat archive:era5-thermograph/backups/forgejo/db/<name>.dump.age \
  | sudo age -d -i /etc/thermograph/age.key \
  | docker exec -i forgejo_db sh -c 'PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U $POSTGRES_USER -d $POSTGRES_DB --clean --if-exists'
# data volume (repos/lfs/config) — restore into the volume with Forgejo stopped
rclone cat archive:era5-thermograph/backups/forgejo/data/<name>.tar.age \
  | sudo age -d -i /etc/thermograph/age.key \
  | docker run --rm -i -v forgejo_forgejo_data:/data busybox tar -C /data -xf -
```

## Verify a backup is restorable (quick)
```sh
rclone cat archive:era5-thermograph/backups/db/prod/<name>.dump.age \
  | sudo age -d -i /etc/thermograph/age.key | head -c 5   # -> PGDMP = valid custom-format dump
```

## Manual run
`workflow_dispatch` the "Ops cron (backup + IndexNow)" workflow, or run the same
`age | rclone rcat` pipeline by hand (see the workflow script).