How does Gearset build a comparison?
In order to build a comparison, Gearset does a lot of work in the background.
Gearset will complete a listMetadata call, and using this information it will create the package.xml(/s) as needed.
Gearset will then use these packages as well as the Metadata API to retrieve the required metadata.
What factors go into deciding if a component is part of a managed package?
Gearset uses some of the information given in this listMetadata call in order to decide whether or not to include a component in the comparison grid.
Specifically, Gearset uses the manageableState of the component. If the component shows as installed then it will only be included if that namespace is included in the Include managed packages section in the metadata filter.
Therefore, if this manageableState is not installed then it will always be included in the comparison, as long as it aligns with any other filtering.
How can I test this?
You are also able to run these listMetadata calls easily and for free using an external tool called Workbench.
To do this first login to the org on Workbench, and then go to info > Metadata Types & Components.
From the list, find the metadata type that you are wanting to test and expand the folder. In this example I've used layouts.
Within the list you will be able to see the manageableState of the component. In this example we can see two layouts, one is the layout that came with the package, and the other is a custom layout that I had created on the managed package. I would expect the custom layout to appear in the comparison regardless of whether Include managed packages was set to "All" or "None".
How to exclude any custom components built on top of a managed package from being in the comparison?
As in the example above, I had created a custom layout inside of a managed package object. Since the manageableState is unmanaged it is included within the comparison.
If I need to exclude this unmanaged component from the comparison, I can make use of Regex in the metadata filter.
This example will look for any components with the prefix FSL__ and not include them in the comparison.
More details on Regex filtering can be found in this article: Using regex in custom metadata filters.