Success After Failure
Due to inherent variations in processes and human behavior, change implementations will fail for one reason or another. In the event that a change can not be implemented in the time allotted, there must be a rollback plan to restore the CI to a usable state to avoid impacting users as much as possible.
When change builders plan a change they must also plan how to rollback, or remove the change, should there be a problem. The idea is to make them work through and document the steps needed to return the CI to the state prior to the current change. The plan may range from copying files in one path to another to performing full system backups to printing a copy of the current configuration file prior to editing it.
Next, the rollback plan should be tested along with the actual change in question. In Visible Ops, we identify that a mandatory CAB question must be:
“Is there a rollback plan and has it been tested?”
If the answer is no, then the change should not be approved. Moving past Change Management, Release Management will take individual changes and make sure that they work appropriate in the context of any delta or package releases that they are managing.
Once it is identified that a change can not be installed per plan, it should rolled back to minimize impacts to the business. Even though the change failed, a successful rollback can help protect service levels from being breached and other risks from escalating. A good rollback plan that works can be the difference between “back to the drawing board” and a major Incident.