Backup container

Facilitates automated backups with restic in a kubernetes cluster. The following documentation is a subsection of the readme from jupyterhub-rci-dev.

DB Backups

If the database gets nuked the backupcontainer can help restore it to a specific state. Backups are made using pg_dump and then stored in an s3 bucket with restic; restic dump along with pg_restore can bring the database back. Two scripts are in the container for convenience: backup.sh and restore.sh, both are stored in /bin so they can be run anywhere.

The s3 bucket is under the tidesupport user with the bucket name sdsu-restic. The credentials for the sdsu-restic user along with the restic repository password are stored in the secret s3secret.

Any manual intervention requires remoting onto the backup container to utilize its scripts. Backups are automated and are set to run at 2am.

Backups (entirely automated in backup.sh)

1. Any file currently stored at /backup/<db_name>_dump.tar will be moved with a version number for archiving purposes.
2. pg_dump with the custom format -F c flags is used to dump the state of the DB.
3. restic backup /backup/<db_name>_dump.tar stores the information in the restic repository, in the s3 bucket.

Restoration (manual)

Restoration is a partly manual process. The admin is required to get the proper version of the /backup/<db_name>_dump.tar file in place that they want on the database, restore.sh will do the rest.

1. Remote onto the container with kubectl get pods and kubectl exec -it <backup pod name> -- sh.
2. See what backups are available with restic snapshots.
3. Restore the snapshot you want with restic dump <snapshot id> /backup/<db_name>_dump.tar > /backup/<db_name>_dump.tar. Note: pg_restore may throw some errors, if you see "Restore.sh complete" that means it exited with a standard exit code and should have worked.
4. Call restore.sh to install the backup onto the database (uses pg_restore).

Variable configuration

Environment variables are used to pass configuration information to the container. Three parts must be configured: the postgres information, the S3 key id and access key, and finally the restic repository and password.

PGUSER: The username for the database. PGPASSWORD: The password for the database. PGDATABASE: The name of the database to save. PGHOST: The name of the host, this is passed as the -h flag to pg_dump. AWS_ACCESS_KEY_ID: The S3 bucket access key ID. AWS_SECRET_ACCESS_KEY: The key to the associated S3 bucket. RESTIC_REPOSITORY: The target for the restic repository. RESTIC_PASSWORD: The password for the restic repository.