A note: If you have made a non-Pipeline deployment that you would like to roll back, please see this article.
Understanding rollbacks
“Rollbacks,” as they are generally understood (a reversal of a Production deployment), are not consequence-free, and choosing the correct problem-solving method for your Pipeline should reflect this.
Best practice, where possible, is always to fix forward. If you find you regularly need to roll back a Production release, taking a closer look at your process is advisable. You can reach out to your team at Gearset for support with this, or check out one of our webinars for some practical starting points.
For example, if a feature branch has been reviewed, approved, passed all quality gates, and released into Production, but only then highlighted as having issues severe enough to demand a rollback, it is likely your testing or approval processes need review.
An important thing to keep in mind is that some things can’t be deleted over the Metadata API (such as Record Types) and indeed shouldn’t be deleted over the API - there could be serious implications for your data if Record Types were deleted.
Therefore, it's always better to think about how your process can prevent the need for a rollback before planning to carry one out.
When to roll back a Production deployment
Before rolling back, ask:
Is it affecting Production users?
Rolling back immediately is the best option (see Scenario 2).
If it is not, does the change need rolling back, or is it something you can correct by fixing forward?
In most cases, fixing forward is the better path (see Scenario 1).
Important note: Ensure that the auto-delete feature branches checkbox is unchecked during a rollback.
Scenario 1: Rolling back a feature not yet deployed to Production
A feature branch may require more components or to have some removed, pending validation, testing, or feedback.
In this example, a feature branch has been deployed through the Pipeline from DEV to SIT to UAT.
The feature has been deployed to the SIT and UAT orgs. A pull request is automatically opened against Production.
During UAT, your team might identify that the feature:
Needs to be updated by adding or removing components.
Is not working as it should and needs to be removed from SIT and UAT.
You've then got two choices. To either attempt to fix the feature in progress, or to delete the feature in order to start again from the beginning.
To make changes to an in-progress feature
Please see this article.
To completely abandon the feature and remove its components from all environments
Please see the following steps:
In your VCS system directly, there will be a history of PRs already merged. For the feature you're looking to rollback, locate the PR that was merged into your first environment in the Pipeline.
Revert that PR. This should open a new "Revert-PR" against the first environment in your Pipeline.
Documentation for reverting a PR on Github, Gitlab, Bitbucket and Azure DevOps.
At the time of writing, reverting a PR isn't natively supported in AWS CodeCommit. So you may need to follow this process. But replacing "Main" with the first environment branch in your Pipeline.
You should now see this newly created PR appear in your Pipeline, against the first environment. Propagate this PR accordingly through all environments where your original feature was already committed, until your feature has been rolled out of everywhere. You will eventually catch-up to where your original features PR made it to.
Your original feature's PR is likely still open at the environment it has not yet been propagated to. Delete that PR. If you've got the "auto-delete feature branches" option enabled. This will trigger your original feature branch to be deleted also.
The result of this will be that your original feature branch is deleted, and the changes contained within have been removed from all git branches and org environments.
Scenario 2: Rolling back a feature in your Pipeline which has been deployed to Production
Have you already merged a feature branch all the way through your Pipeline and deployed it to Production, and now you need to correct Production?
Before deciding how to resolve an issue with a Production deployment ask yourself these questions:
Is it affecting Production users?
If it is, then rolling back immediately is the best option.
If it is not, does the change need rolling back, or is it something we can correct by fixing forward?
Rolling back Production immediately
If you find yourself in a situation where you need to urgently remove the components deployed to Production from the org you can follow these steps. Because this involves a direct deployment to Production it should only be used in exceptional circumstances.
This is a two-stage process, the actual removal of the unwanted components from the org and the branch changes required to get things in sync. For a shorter path please see the section "Using a hotfix branch to roll back".
Open the CI job dashboard and locate the Production CI job.
Open the history.
Open the most recent deployment details, and click “Roll back”.
Select the components (taking care with some of the MDAPI nuances as listed here, not every component can be deleted using a deployment) you would like to remove from Production.
Proceed through problem analysis and deploy your changes to Production.
From the deployment summary, choose the option to “Deploy to new target”.
Create a hotfix branch from main and commit your package.
Open a pull request from your hotfix branch to the main branch.
After approval, merge your hotfix branch to the main branch. This will bring the main branch and Production back in sync. If the target of the hotfix is already in sync with the main branch, then you will see zero components to validate. This isn’t anything to worry about, and you can merge the branch with no ill effect.
Gearset will back-propagate this hotfix through your Pipeline. Merge it to your environments to make sure they are also up to date with Production.
Review what went wrong to close any gaps in your process to prevent the need for a Production rollback in the future. You can conduct a Root Cause Analysis (RCA) to identify why this was necessary and track the amount of these over time as it will affect your DORA metrics and performance for your team.
Rolling back a specific feature
If there is a specific feature that you need to remove from the main branch/Production you can do so by following these steps. To use this method you will need to make sure that the original feature branch still exists.
Start by navigating to your deployment history page. Locate the user story/ticket number in your deployment history.
Roll back the original commit to the feature branch. This will essentially 'revert' the changes and add an extra commit to your branch.
After the fix has been committed, open a pull request to the main branch.
After all the quality gates and review processes have passed successfully, promote the fix to Production.
Gearset will back-propagate this hotfix through your Pipeline. Merge it to your environments to make sure they are also up to date with Production.
Review which features were affected by your rollback and update any associated tickets.
Review what went wrong to close any gaps in your process to prevent the need for an emergency Production fix in the future. You can conduct a Root Cause Analysis (RCA) to identify why this was necessary and track the amount of these over time as it will affect your DORA metrics and performance for your team
Using a hotfix branch to roll back
If you find yourself in a situation where you want to fix something that has just been deployed to Production very quickly, but without directly altering the Production org to do so, then you should use a hotfix branch.
Develop your fix in a Sandbox, and then open a hotfix branch.
Commit your fix to the hotfix branch using Gearset.
After the fix has been committed, open a pull request to the main branch.
After all the quality gates and review processes have passed successfully, promote the fix to Production.
Gearset will back-propagate this hotfix through your Pipeline. Merge it to your environments to make sure they are also up to date with Production.
Review which features were affected by your rollback and update any associated tickets.
Review what went wrong to close any gaps in your process to prevent the need for an emergency Production fix in the future. You can conduct a Root Cause Analysis (RCA) to identify why this was necessary and track the amount of these over time as it will affect your DORA metrics and performance for your team
How to prevent the need for a rollback
After resolving the immediate need to roll back or fix forward a change in your Pipeline we recommend asking the following questions:
How could we have found this issue earlier?
How can we close this gap in our process going forwards?
We have some tips to help you get started with this immediately:
Use the recommended settings for your Pipeline.
Create Unit Testing and Monitoring jobs for all your orgs, use these results to run daily org scans and identify the following before they ever enter the Pipeline:
Apex code coverage dropping below a set threshold.
Apex code with failing tests.
Code present in your orgs violating your standard or custom rule sets.
Set Gearset as an unavoidable status check on your source control provider, requiring a successful validation before changes can be promoted. Check out these articles for some inspiration: