Getting started
You'll need either a Teams Automation Platform licence or an Enterprise Automation Platform license to use this feature.
This feature should be used by teams interested in tracking their DevOps performance through our in-app dashboard.
What is change failure management?
To accurately track and report on your DevOps performance, Gearset needs to understand when a failure was introduced to production and when it was resolved. Change failure management is a feature that allows you to record this critical information directly in Gearset. This data is essential for calculating your change failure rate and mean time to restore.
What is a "failure"?
A failure refers to any deployment to production that results in end-user impairment, outage, degraded service, or requires an urgent hotfix, revert or rollback.
This means the deployment to production was successful (not to be confused with validation or deployment failures in Salesforce), but resulted in an incident or issue that critically affected the business and/or its end-users.
Change failure rate is the percentage of deployments causing a failure in production. Once such an issue is identified, your team should address it through a forward fix/hotfix or a rollback. Mean time to restore is the time it took to recover from that failure, measured from the moment the failure was introduced until the issue was successfully resolved.
In the context of the DORA metrics, change failure rate and mean time to restore are known as stability or resilience metrics.
Why should you measure it?
These stability metrics are crucial for any DevOps team to track. The lower your change failure rate, the greater the likelihood that your team is producing better quality work and more stable deployments.
However, failures are never entirely avoidable. Your time to recover from failures show how robust your systems for fixing or rolling back changes are built, how closely you monitor production, and whether you’re prioritising recovery enough or too little when there’s a failure.
How do I start?
The specific approach will depend on how your team addresses failures. The different options are described here.
Approach 1: Hotfixes to Production
This approach is for teams who use hotfixes to address urgent issues in production. This means you create a new feature branch, commit the necessary changes and promote it directly to production.
For Gearset to help you accurately report these metrics, you must tell us:
Which promotion caused a failure?
Which feature branch was the intended fix?
How do I record a failure?
Before following the below steps, you should work with your team to understand whether the issue was introduced by an individual feature promoted to production or by a feature that was part of a release.
The issue was introduced by an individual feature promoted to production
Select the production environment
Select the “Recent promotion history” tab.
If the promotion is recent and shows up on this page, skip to step 3.
If the promotion is older, select the "View all promotion history" button and then proceed with step 3.
Click the three ellipses icon next to the promotion that contains the failure and select the "Mark as a bug" option:
A screen will appear where you can:
Select the severity of the failure (e.g. P1, P2).
Enter any supporting notes about the failure (optional).
Press “Mark as bug” to save the detail.
The issue was introduced by a feature that was part of a release
If you need to mark a specific feature within a release as the source of the failure, follow these steps:
Select the production environment
Select the “Recent promotion history” tab
Select the "View all promotion history" button
Find the promotion of the release
On the right, press “Release features”
Click the three ellipses icon next to the feature that caused the failure and select the "Mark feature as a bug" option:
Once this is complete, Gearset will mark the corresponding deployment as resulting in a failure and track it in your DevOps performance dashboard.
How do I link a fix?
Before following the below steps, we strongly recommend recording the failure first as described in the previous section.
Once you've done this, you can:
Select your developer sandbox
Select the “Bug fix feature” button
A dialogue will appear with the following options:
Use the “Select a bug” drop down to search for the failure you are looking to fix.
A branch name will be populated automatically (you can change this if needed).
Select the “Compare now” button and continue with the process
By following these steps, your bug will automatically be recorded as resolved once the bug fix branch has been successfully promoted to production.
Manual Linking
If you already created and promoted a fix without using the "Bug fix feature" workflow above, you can retrospectively link the fix:
Select the production environment
Select the “Recent promotion history” tab
If the promotion is recent and shows up on this page, skip to step 3.
If the promotion is older, select the "View all promotion history" button and then proceed with step 3.
Locate the PR where the failure was recorded (it will be marked as "Bug")
Click the three ellipses icon next to the promotion that contains the failure and select "Update bug":
Select the “Manually link a fix” button
Select the PR that fixed the failure and press "Resolve"
Approach 2: Production Rollbacks
If you are a team who rolls back a production deployment to remove critical issues and bring the org back to a healthy state, then Gearset will do the following:
Once you have started a rollback process for a production deployment, we will mark the deployment as a failure. You will see the corresponding PR(s) with a "Deployment Bug" label in your promotion history:
Once the rollback has been successfully completed, we will consider the rollback deployment as the fix and automatically resolve the bug. This will show up in the bug's audit trail.
Managing bugs
Audit Trail
Every bug has a full audit trail to allow you to see all events that have occurred since its creation. To view the audit trail:
Select the the production environment
Select the “Recent promotion history” tab.
If the promotion is recent and shows up on this page, skip to step 3.
If the promotion is older, select the "View all promotion history" button and then proceed with step 3.
Locate the PR where the failure was recorded (it will be marked as "PR Bug" or "Deployment Bug")
Click the three ellipses icon next to the promotion that contains the failure and select "Update bug"
Select the “Updates” tab:
This screen shows a chronological list of everything that has happened to the failure record and by whom. Any change you make to a failure record is recorded here for future review. If you accidentally make a change and need to delete it, you can do by selecting the three ellipses icon on the right of each change:
Abandoning/Reopening Bugs
A bug can be abandoned if it was marked in error, or reopened if it was resolved but the issue resurfaced. Locate the PR and open the bug details screen.
From the details screen you can:
Add a note and/or change the severity
Abandon the bug if it is currently unresolved (you must provide a reason)
Reopen the bug if it was previously resolved or abandoned
Manually resolve the bug (see the steps above).