67 lines
3 KiB
Markdown
67 lines
3 KiB
Markdown
|
# Off-box backups → Contabo Object Storage
|
||
|
|
|
||
|
|
Durable, encrypted, off-box copies of the state that would otherwise die with a
|
||
|
|
single VPS. Driven by `.forgejo/workflows/ops-cron.yml` (03:00 UTC daily +
|
||
|
|
`workflow_dispatch`).
|
||
|
|
|
||
|
|
## What is backed up
|
||
|
|
|
||
|
|
| Prefix in bucket `era5-thermograph` | Source | Job | Retention off-box |
|
||
|
|
|---|---|---|---|
|
||
|
|
| `backups/db/prod/` | prod Postgres/TimescaleDB (`pg_dump --format=custom`) | `backup` (prod, `PROD_SSH_*`) | 30 days |
|
||
|
|
| `backups/forgejo/db/` | Forgejo Postgres (`forgejo_db`) | `forgejo-backup` (beta, `SSH_*`) | 30 days |
|
||
|
|
| `backups/forgejo/data/` | Forgejo data volume (repos/LFS/config) | `forgejo-backup` | 30 days |
|
||
|
|
|
||
|
|
Everything is streamed through **`age`** encryption before upload — encrypted to the
|
||
|
|
vault recipient `age1xx4dz…`, so the private key each host already has at
|
||
|
|
`/etc/thermograph/age.key` (and the operator's `~/.config/sops/age/keys.txt`)
|
||
|
|
decrypts it. Nothing plaintext leaves the box.
|
||
|
|
|
||
|
|
## Access layer
|
||
|
|
- S3 endpoint/bucket/keys are **Forgejo Actions secrets** (`S3_ENDPOINT`,
|
||
|
|
`S3_BUCKET`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`) — the CI job's credential home
|
||
|
|
(also mirrored into the SOPS vault `deploy/secrets/{prod,beta}.yaml` for host-side
|
||
|
|
use). Contabo needs **path-style** addressing (`force_path_style=true`,
|
||
|
|
`provider=Other`, `region=default`).
|
||
|
|
- `rclone` + `age` must be installed on prod and beta (one-time: `apt-get install -y
|
||
|
|
rclone age`). The prod job apt-installs them if missing; the beta user has no sudo,
|
||
|
|
so they are pre-installed there.
|
||
|
|
|
||
|
|
## Restore (DR runbook)
|
||
|
|
|
||
|
|
Prereq on the restoring host: `rclone` + `age` + the age key at
|
||
|
|
`/etc/thermograph/age.key` (or the operator key), and rclone configured for the
|
||
|
|
`archive:` remote (env vars `RCLONE_CONFIG_ARCHIVE_*` or `/etc/rclone/rclone.conf`).
|
||
|
|
|
||
|
|
### Prod database
|
||
|
|
```sh
|
||
|
|
# list available encrypted dumps
|
||
|
|
rclone ls archive:era5-thermograph/backups/db/prod/
|
||
|
|
# download newest, decrypt, restore into a fresh db
|
||
|
|
OBJ=archive:era5-thermograph/backups/db/prod/<name>.dump.age
|
||
|
|
rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \
|
||
|
|
| docker exec -i <db-container> pg_restore -U thermograph -d thermograph --clean --if-exists
|
||
|
|
```
|
||
|
|
|
||
|
|
### Forgejo
|
||
|
|
```sh
|
||
|
|
# database
|
||
|
|
rclone cat archive:era5-thermograph/backups/forgejo/db/<name>.dump.age \
|
||
|
|
| sudo age -d -i /etc/thermograph/age.key \
|
||
|
|
| docker exec -i forgejo_db sh -c 'PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U $POSTGRES_USER -d $POSTGRES_DB --clean --if-exists'
|
||
|
|
# data volume (repos/lfs/config) — restore into the volume with Forgejo stopped
|
||
|
|
rclone cat archive:era5-thermograph/backups/forgejo/data/<name>.tar.age \
|
||
|
|
| sudo age -d -i /etc/thermograph/age.key \
|
||
|
|
| docker run --rm -i -v forgejo_forgejo_data:/data busybox tar -C /data -xf -
|
||
|
|
```
|
||
|
|
|
||
|
|
## Verify a backup is restorable (quick)
|
||
|
|
```sh
|
||
|
|
rclone cat archive:era5-thermograph/backups/db/prod/<name>.dump.age \
|
||
|
|
| sudo age -d -i /etc/thermograph/age.key | head -c 5 # -> PGDMP = valid custom-format dump
|
||
|
|
```
|
||
|
|
|
||
|
|
## Manual run
|
||
|
|
`workflow_dispatch` the "Ops cron (backup + IndexNow)" workflow, or run the same
|
||
|
|
`age | rclone rcat` pipeline by hand (see the workflow script).
|