Gearset provides a public External Testing API as part of the Automation API for receiving test data from any third-party testing product. For the testing data to be displayed in Gearset, an integration must be created for transforming the data from the third-party product and sending it to Gearset via the API. The data from the External Testing API is shown as End to end test runs.
End to end testing data in Gearset
The End to end test runs of the latest CI job run can be accessed from the Pipelines or CI Jobs pages.
The End to end tests status column in CI jobs:
The End to end tests status label on an environment in Pipelines:
The End to end tests in the Testing section of a Pull Request in a Pipeline:
In Gearset, the End to end test runs are stored against the CI job runs.
Each CI job run can contain the details for any number of End to end test runs. The status label shown in the Pipelines and CI Jobs sections is an aggregate over all the End to end test runs with data in Gearset. Clicking on the status label opens the individual End to end test run details.
The End to end test results page lists all the test data available for a CI job run:
Gating - blocking deployments with End to End test runs
Gating - blocking deployments with End to End test runs
When a pull request is deployed and that deployment has an unsuccessful End to end test run associated with it, we block selecting the pull request for promotion.
It's still possible to promote the pull request by clicking "Dismiss and continue" and entering an optional dismissal reason. The End to end test run that has been dismissed is considered as successful when promoting pull requests.
Dismissing a result is done per End to end test run. This means that if the End to end test run was blocking multiple pull requests from promotion, all of them are now unblocked.
E.g. pull requests #27 and #17 in the above screenshots.
What testing data can be sent to Gearset
What testing data can be sent to Gearset
The following data can be sent to and displayed in Gearset via the External Testing API:
Provider Name - A mandatory free text field for identifying the testing provider in the UI.
Provider Run ID - An optional free text external ID to allow identifying a test run in the third-party product.
Results URL - An optional URL to the test results page in the third-party product. If provided, a button to open the link will be in the End to end test results table.
Status Class - A mandatory Gearset status classification used to determine how the status is displayed in the Pipelines and CI Jobs sections. All End to end test runs must be mapped to one of the following values:
Succeeded - all tests have finished and passed (or not run).
Failed - some critical tests have failed (or not run).
Warning - some tests have failed, but these are not critical.
In Progress - the test run is in progress.
Scheduled - the test run is scheduled for execution, but has not yet started.
Not Run - there is a test run that should be run, but it's not scheduled yet.
It's the caller's choice of how to map the test run results, but we recommend sticking to the interpretation mentioned above.
Status - The optional (short) status value of the test run as it is shown in the third-party product. This is the status displayed in the End to end test results. If missing, the Status Class is shown instead.
Status Message - An optional (longer) informative message about the current status of the test run, if available from the third-party product (e.g., if there is an error, the error message). If provided, there will be a tooltip with the message when you hover over Status in the End to end test results.
Start Time - The start time of the test run.
End Time - The end time of the test run.
Getting Started With the External Testing API
Getting Started With the External Testing API
First, generate a Gearset API access token.
Set up an outgoing webhook to send the CI job run ID on completion of a deployment.
Use the External Testing API documentation to implement (or test) the Gearset integration.