Support database rollbacks in case of bad migrations

[avrittrohwer]

• General
• Backend
• Public
• Partners

Background / Issue

The tofu infra code should trigger a DB backup before it makes any changes to the API container. Because the API container runs prisma migrations when it starts, it would minimize the data loss in the event that we had a bad migration that messed up the database, requiring us to restore to a snapshot.

[avrittrohwer]

Some notes:

The bloom DB specifies backup_retention_period, which means it supports restoring to a specific point in time: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIT.html. AWS triggers daily snapshots, as well as uploading transaction logs to s3 every five minutes. So, in the case where the API applies a bad migration, we worst case can restore to the DB state 5 minutes before that apply.

RDS restore works by creating a new DB instance. So there is still some work to support this cleanly in the terraform code