In no particular order, these are the things that I consider when devising a rollback strategy:
- If you are able to, fix forward instead of rolling back. When you adopt a fix forward mindset you are making changes based on the specifics of the unexpected scenario you find yourself in. With a rollback strategy, you have to guess or anticipate which problems you might have and craft a rollback that is general enough to address all of them. This is hard to do and not fool proof. Fix forward allows you to make the most minimal change possible to address the issue.
- Figure out a way to use temporary or ephemeral databases for deploy testing. This is safe and usually not that hard to do if your org is cool with containerization or fleeting cloud resources in CICD.
- If you have to roll back, consider writing your own instead of relying on automated rollbacks. The automated rollbacks are effective but can be brutish. Create a table or column? Automatic rollback is going to drop it along with any data that may have been written between the time the changeset was executed and the time of the rollback. Crafting rollbacks specific to the change you are making with the context you have enables you to do things like renaming tables and columns instead of dropping them to retain data.
- You’re already thinking about this one, but test your rollbacks as early as possible and include peer review in your checklist for pull/merge requests. It’s easier to fix these up front than when your rollback fails in prod!