How to realign/fix an out of sync Pipeline

IMPORTANT: Before attempting to fix an out of sync Pipeline, they key to long term success is understanding the cause of how and why it got out of sync. That way you can ensure that it doesn't happen again.

Problem:

When first setting up a Gearset Pipeline, the advice is to ensure everything is an in sync as possible. Then from the moment it's configured, all future development changes should be propagated through the Pipeline. That'll ensure that it remains in sync and usable for the foreseeable future.

However, sometimes it's possible that your team has slowly drifted from the process and done things outside of the Pipeline workflow (E.g. Circumventing the Pipeline by developing directly in a long-lived environment like UAT). As a result of this drift, your connected git repository will become out of sync, and you may see unexpected behaviour that you're looking to rectify.

There are two recommended processes to follow, depending on the severity of your teams drift from the prescribed workflow, this document will walk you through both of them.

Scenario 1: A small number of metadata components are out of sync (No new environment branches needed)

Click here for details

If it's only a small subset of metadata components, then you can feasibly refresh your existing Pipeline/branches without needing to start fresh. The real key to success here is how in sync your org environments are before you begin, as well as lack of work-in-progress features. So ideally, you'll follow these next steps after you've both released everything in progress and refreshed all orgs.

When ready, follow these steps:

Ensure that your metadata filter is correctly configured and saved.
1. Navigate to the Compare & deploy page and open the comparison filter option.
2. Select Manage custom filters... and select all the metadata types you'd like your CI job to consider. This selection will be the new all encompassing filter. This document provides more guidance on what to include/exclude.
3. In the bottom left, select the dropdown arrow and select Save as new filter.
4. Give your new filter a recognisable name and save it.
Repopulate Main from Production, using the new filter.
1. From the same Compare & deploy page, run a comparison between Production (as the source) and your Main branch (as the target). Use your new filter selection.
2. Select all components, bringing them into the package.
3. Proceed to the next page. You may see a number of problem analysers triggering for various reasons. Pay attention to these suggestions, but for the most part you can deselect/ignore them. We're looking to seed your repository with all the metadata, bringing it in sync with your org.
4. Commit all these components into your Main branch.
Apply that newly saved filter to each environment in your pipeline.
1. From the continuous integration dashboard, select any of the CI jobs that make up part of your pipeline. Once expanded, select Edit settings...
2. Navigate to the Metadata filter tab and within the dropdown box, you should now be able to find and select the filter you just saved.
3. With the new filter selected, save the job.
4. Repeat this process for the rest of the CI jobs that make up your pipeline.
Re-sync each branch environment in your pipeline.
1. Navigate to your final Pipeline environment on the Deployment pipelines page and select this sync feature (the one with description: "Create sync PRs from main to all your upstream environments" when you hover over it)
2. Create sync pull requests for each and every static environment. Note that if a static environment already has an open sync PR, it won't appear on the list.
  
  Also note that depending on your git provider, if a static environment is already in sync with main, you may see an error message similar to:
  Failed to open sync PR: There are no commit differences between the environment <XXX> and <XXX>. The environment <XXX> is already up to date. This just means that there is nothing to sync, so no sync PR necessary for that environment.
3. A pull request will be opened between Main and the environments you're doing it on(Excluding those that don't need to be synced). The promotion of this PR will sync all these newly added metadata components from your Main branch into your environment branch. This in turn will trigger a run of the CI job. As your org is in sync with Production, we wouldn't expect this run to actually deploy anything to the org. But please be aware that it will, if the environments aren't in sync.

Scenario 2: A significant number of metadata components are out of sync (New environment branches necessary)

Click here for details

IMPORTANT: This method requires new environment branches and CI jobs, which will replace the existing ones used by the Pipeline. Any history within those branches or CI jobs won't be present in the new ones we create. Due to this, it's recommended to keep the originals available but disabled until you're sure you don't need the histories anymore.

If you've struggled to follow scenario 1, or there are a significant number of metadata components that aren't in sync, then we'd suggest starting with new CI jobs and environment branches. This'll ensure perfect synchronicity and a clear baseline for future use of the Pipeline. Just like with scenario 1, follow these next steps after you've both released everything in progress and refreshed all orgs. If you're unable to, you'll need to account for work-in-progress before following this process.

When ready to begin, follow these steps:

Configure and save your new filter.
1. Navigate to the Compare & deploy page and open the comparison filter option.
2. Select Manage custom filters... and select all the metadata types you'd like your CI job to consider. This selection will be the new all encompassing filter. This document provides more guidance on what to include/exclude.
3. In the bottom left, select the dropdown arrow and select Save as new filter.
4. Give your new filter a recognisable name and save it.
Repopulate Main from Production, using the new filter.
1. From the same Compare & deploy page, run a comparison between Production (as the source) and your Main branch (as the target). Use your new filter selection.
2. Select all components, bringing them into the package.
3. Proceed to the next page. You may see a number of problem analysers triggering for various reasons. Pay attention to these suggestions, but for the most part you can deselect/ignore these suggestions. We're looking to seed your repository with all the metadata, bringing it in sync with your org.
4. Commit all these components into your Main branch.
Disable existing CI jobs and remove them from the pipeline.
1. Navigate to the Deployment pipelines page and select any of the static jobs that make up part of your pipeline.
2. Toggle the CI job from Job enabled to Job disabled.
3. Repeat this process for each static environment in your pipeline.
4. Then on the Deployment pipelines page select Edit environments (top right corner), and delete each job from your pipeline (by selecting Delete environment button). Please note that this action is only removing each CI job from the pipeline — it's not actually deleting the CI job. This is intentional as we don't yet want to delete any jobs as we'll need them for the next step.
Duplicate existing CI jobs and create new environment branches.
1. Navigate to the Continuous integration dashboard where you'll still see each of the now disabled CI jobs.
2. Select one of the CI jobs, expanding it to reveal more options, and select Duplicate job. This option will allow you to create a new CI job but nearly all the settings from the original job will be pre-configured to save you time.
3. For this new job, you'll need to once again select the source and target. For the source, select your Git Provider, original repository and initially select your Main branch. Right underneath it you'll see the option + Create new branch from Main — select that and name the new branch in accordance with the target environment for this CI job. If we're duplicating the UAT job, we'd likely name the new branch according to that. (E.g. UAT-Branch). The target in this example would be the UAT org.
4. Next navigate to the Metadata filter tab and apply the newly saved metadata filter to the CI job.
5. Be sure to check all the configuration tabs to make sure it matches what you would expect, and save the job.
6. Repeat this process for each of the original CI jobs in your pipeline.
Add these CI jobs back into your pipeline and connect them.
1. Navigate to the Deployment pipelines page and select + Add environment, then Add static environment.
2. Select Choose from an existing CI job and select one of the CI jobs that you've just created. Click Save to add it to your pipeline.
3. Repeat this process for each CI job in your pipeline.
4. Finally, connect each CI job within the pipeline, as they were originally. This restores your pipeline to how it used to be, but now the filters and repository reflect what they should.

Post configuration considerations for scenario 2

I still have in-flight work open in a PR against one of the old environment branches. What do I do about these?

This is only applicable if you weren't able to release in progress work as per our recommendation. As you now have whole new environment branches, you'll need to bring these PRs and their associated feature branches over to the newly updated pipeline. The original feature branch for each of these PRs should still be active and usable within your repository, so you simply need to open a new PR from each original feature branch to the first environment in your pipeline. This process brings the feature back to the beginning of the new pipeline, which you can them promote through until it resumes its original location. A warning around this would be to check the state of the feature branches in question. Depending on how your Pipeline got out of sync, it may be that your feature branch is polluted with unexpected changes already, in which case you'll likely want to start fresh with a brand new commit from a dev environment into a feature branch.

I had configured automatic Jira status updates for each stage of my pipeline, do I need to reconfigure these?

Yes, you will need to reconfigure these. Please follow our documentation on this:
Updating a Jira ticket's status in pipelines

At what point am I safe to delete the old disabled CI jobs?

Each of these CI jobs contains a history of all deployments that it did. Once you are certain that you no longer require this history, you're safe to delete these CI jobs from the Continuous integration dashboard.

At what point am I safe to delete the old environment branches?

Similar to the CI jobs, each of these branches will contain a history of any previous commits. As well as this, if there is any in-flight work that still needs to be transitioned over to the newly created environment branches, then the branch still needs to be kept. Once you're certain you don't need to transition any further in-flight work from them, or need their history, then you're safe to delete them directly within your Git Provider.

Need any further clarification or advice?

If you have any further questions feel free to reach out to us via the in-app chat.

Keep your environment branches in-sync

Adding previously missing metadata types into Gearset pipelines

Edit or fix a user story in progress in pipelines

Resolving validation errors - Once an approval process has been activated, you cannot add or remove steps. Please create a new approval process from this one for modification. You may only delete an approval process that has no data records associated..."

How to Deploy CPQ with Gearset Pipelines