Hey @OmarNoah! Answers & recommendations below…
- How do you normally deal with this in real life?
If a database ends up in the state you describe, manual intervention is necessary. You’ll need to clean up the partially deployed or rolled back change. You might also need to update the DATABASECHANGELOG table that tracks changeset execution to remove the entry for the changeset that didn’t rollback cleanly.
- Is there any way to validate that a rollback actually works before applying the change in production?
There are a couple of ways to do this and what’s right for you depends heavily on what’s available in your pipeline. A lot of users will test update and rollback in a pre-prod database that is frequently refreshed from production. Other users will use containerized databases in their workflows to test the update and rollback they are introducing. We recommend that update and rollback not be tested in prod because…well…nothing should really be tested in prod. If you have replicated databases in prod you could conceivably work out a blue/green deployment to test everything if you are able to remove a database from the replication cluster or temporarily pause replication to protect your data through isolation.
- Have you found ways to automatically test
update + rollbacktogether safely as part of a pipeline?
There is the update-testing-rollback command that performs an update, performs the rollback for that update, and then performs the update again to make sure that the rollback sufficiently undoes the changes. I’ve also seen users build a CI workflow with individual commands to accomplish a more thorough version of this: taking a pre-update snapshot; run the update; run the rollback; compare the current state of the database to the snapshot taken in step 1. If there are no differences detected, you can be reasonably comfortable that the rollback did what it was supposed to.
Take a look at the Liquibase Rollback Workflow docs for more information. Good luck!