Gearset provides a public External Testing API as part of the Automation API for receiving test data from any third-party testing product. For the testing data to be displayed in Gearset, an integration must be created for transforming the data from the third-party product and sending it to Gearset via the API. The data from the External Testing API is shown as End to end test runs.
End to end testing data in Gearset
The End to end test runs of the latest CI job run can be accessed from the Pipelines or CI Jobs pages.
The End to end tests status column in CI jobs:
The End to end tests status label on an environment in Pipelines:
The End to end tests in the Testing section of a Pull Request in a Pipeline:
In Gearset, the End to end test runs are stored against the CI job runs, so the unique RUN_ID is important to tie up to relevant Pull Requests.
Each CI job run can contain the details for any number of End to end test runs. The status label shown in the Pipelines and CI Jobs sections is an aggregate over all the End to end test runs with data in Gearset. Clicking on the status label opens the individual End to end test run details.
The End to end test results page lists all the test data available for a CI job run:
High-level workflow
These are the relevant Gearset/Non-Gearset steps and activities that need to take place to ensure the relevant triggers and information is passed in to the API and then visible against your CI Jobs.
See 'Getting Started With the External Testing API' for more details.
Getting Started With the External Testing API
Getting Started With the External Testing API
Pre requisites:
Generate a Gearset API access token for the Automation API to access our External Testing Public API.
Your provider must be able to store this ID to then call our Public API with the test results.
Steps:
Set up your outgoing webhook in Gearset for CI job X (for the environment where the UI testing needs to take place), pointing at your Orchestration engine/directly to the 3rd party tool, with relevant variables such as RUN_ID which is important to feed back into the Gearset API later.
To aid in triggering the right tests, you can use the variable DEPLOYMENT_DIFFERENCES also, which provides key information on the names/types of metadata changed, and help you work out which UI tests should run.
It's important to note this webhook will trigger for every PR merge/deployment to that CI job
Orchestration engine (or direct 3rd party tool) can then parse the provided information, and determine which tests are valid to run - this should be as tightly scoped as possible and not a full regression test for specific features or small bundles of work, as 'locking up' the sandbox from other work is a real risk with UI testing and unit/integration tests are usually better suited.
Full E2E tests could be completed out of hours via a schedule instead of a Gearset deployment trigger.
Once you've parsed the information and selected the right tests to run, these can be triggered, and from the orchestration engine, set up a call the External Testing Public API endpoints (POST initially), using the Authorization token and CI Job Run ID, including any information on the tests starting, and then update the testing API via PUT requests to refresh the status as the testing progresses.
Once the tests are complete, provide a final update to the testing API including the result and any other information on the completed set of tests.
If the tests are unsuccessful (failed), the PR to the next environment will be blocked, and you can manually resolve and re-trigger as needed, or fix-forward the issues found via new commits to the same feature branch and new PR to the initial environment again.
Once complete and unblocked, or test results overridden with a relevant reason, the PR on the next environment will be ready to proceed (in accordance with validation rules there such as unit tests, PR reviews etc)
Gating - blocking deployments with End to End test runs
Gating - blocking deployments with End to End test runs
When a pull request is deployed and that deployment has an unsuccessful End to end test run associated with it, we block selecting the pull request for promotion.
It's still possible to promote the pull request by clicking "Dismiss and continue" and entering an optional dismissal reason. The End to end test run that has been dismissed is considered as successful when promoting pull requests.
Dismissing a result is done per End to end test run. This means that if the End to end test run was blocking multiple pull requests from promotion, all of them are now unblocked.
E.g. pull requests #27 and #17 in the above screenshots.
What testing data can be sent to Gearset
What testing data can be sent to Gearset
The following data can be sent to and displayed in Gearset via the External Testing API:
Provider Name - A mandatory free text field for identifying the testing provider in the UI.
Provider Run ID - An optional free text external ID to allow identifying a test run in the third-party product.
Results URL - An optional URL to the test results page in the third-party product. If provided, a button to open the link will be in the End to end test results table.
Status Class - A mandatory Gearset status classification used to determine how the status is displayed in the Pipelines and CI Jobs sections. All End to end test runs must be mapped to one of the following values:
Succeeded - all tests have finished and passed (or not run).
Failed - some critical tests have failed (or not run).
Warning - some tests have failed, but these are not critical.
In Progress - the test run is in progress.
Scheduled - the test run is scheduled for execution, but has not yet started.
Not Run - there is a test run that should be run, but it's not scheduled yet.
It's the caller's choice of how to map the test run results, but we recommend sticking to the interpretation mentioned above.
Status - The optional (short) status value of the test run as it is shown in the third-party product. This is the status displayed in the End to end test results. If missing, the Status Class is shown instead.
Status Message - An optional (longer) informative message about the current status of the test run, if available from the third-party product (e.g., if there is an error, the error message). If provided, there will be a tooltip with the message when you hover over Status in the End to end test results.
Start Time - The start time of the test run.
End Time - The end time of the test run.
How to find the CI Run ID
How to find the CI Run ID
To find the CI Run ID manually, open the CI job history
Select the View details of a CI Job Run
The CI Run ID is part of the URL
RUN_ID is also an Outgoing Webhook Variable you can use when communicating with external workflows.