# Off-box backups → Contabo Object Storage

Durable, encrypted, off-box copies of the state that would otherwise die with a
single VPS. Driven by `.forgejo/workflows/ops-cron.yml` (03:00 UTC daily +
`workflow_dispatch`).

## What is backed up

| Prefix in bucket `era5-thermograph` | Source | Job | Retention off-box |
|---|---|---|---|
| `backups/db/prod/` | prod Postgres/TimescaleDB (`pg_dump --format=custom`) | `backup` (prod, `PROD_SSH_*`) | 30 days |
| `backups/forgejo/db/` | Forgejo Postgres (`forgejo_db`) | `forgejo-backup` (beta, `SSH_*`) | 30 days |
| `backups/forgejo/data/` | Forgejo data volume (repos/LFS/config) | `forgejo-backup` | 30 days |

Everything is streamed through **`age`** encryption before upload — encrypted to the
vault recipient `age1xx4dz…`, so the private key each host already has at
`/etc/thermograph/age.key` (and the operator's `~/.config/sops/age/keys.txt`)
decrypts it. Nothing plaintext leaves the box.

## Access layer
- S3 endpoint/bucket/keys are **Forgejo Actions secrets** (`S3_ENDPOINT`,
  `S3_BUCKET`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`) — the CI job's credential home
  (also mirrored into the SOPS vault `deploy/secrets/{prod,beta}.yaml` for host-side
  use). Contabo needs **path-style** addressing (`force_path_style=true`,
  `provider=Other`, `region=default`).
- `rclone` + `age` must be installed on prod and beta (one-time: `apt-get install -y
  rclone age`). The prod job apt-installs them if missing; the beta user has no sudo,
  so they are pre-installed there.

## Restore (DR runbook)

Prereq on the restoring host: `rclone` + `age` + the age key at
`/etc/thermograph/age.key` (or the operator key), and rclone configured for the
`archive:` remote (env vars `RCLONE_CONFIG_ARCHIVE_*` or `/etc/rclone/rclone.conf`).

### Prod database
```sh
# list available encrypted dumps
rclone ls archive:era5-thermograph/backups/db/prod/
# download newest, decrypt, restore into a fresh db
OBJ=archive:era5-thermograph/backups/db/prod/<name>.dump.age
rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \
  | docker exec -i <db-container> pg_restore -U thermograph -d thermograph --clean --if-exists
```

### Forgejo
```sh
# database
rclone cat archive:era5-thermograph/backups/forgejo/db/<name>.dump.age \
  | sudo age -d -i /etc/thermograph/age.key \
  | docker exec -i forgejo_db sh -c 'PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U $POSTGRES_USER -d $POSTGRES_DB --clean --if-exists'
# data volume (repos/lfs/config) — restore into the volume with Forgejo stopped
rclone cat archive:era5-thermograph/backups/forgejo/data/<name>.tar.age \
  | sudo age -d -i /etc/thermograph/age.key \
  | docker run --rm -i -v forgejo_forgejo_data:/data busybox tar -C /data -xf -
```

## Verify a backup is restorable (quick)
```sh
rclone cat archive:era5-thermograph/backups/db/prod/<name>.dump.age \
  | sudo age -d -i /etc/thermograph/age.key | head -c 5   # -> PGDMP = valid custom-format dump
```

## Manual run
`workflow_dispatch` the "Ops cron (backup + IndexNow)" workflow, or run the same
`age | rclone rcat` pipeline by hand (see the workflow script).