What happened?
From 12:08 to 13:50 CST (18:08 to 19:50 UTC) Gearset experienced brief outages and periods of instability affecting app.gearset.com and its background services. Users were unable to access Gearset for approximately 16 minutes during this time (see chart below).
Following this instability there was a delay in pipeline webhook processing that lasted until 17:45 CST (23:45 UTC). Pull request propagations and merge conflicts were delayed during this time.
Periods of high latency and outages (shown in red) plotted against UTC.
What did we do to resolve this?
We were alerted to the problem immediately by several channels of automated alerts and monitoring. We reacted to the performance degradation by scaling the affected infrastructure so that it could handle a higher demand.
We then investigated the source of the instability and took several remediating actions including: blocking the issue from occurring; collaborating with our platform partner, AWS, to remove a bottleneck; and making a change to our platform so that we could regain stability faster.
We continue to investigate and make changes to the affected services so that the same or similar issues cannot cause instability in the future.
Who do I contact if I have concerns?
If you have any questions or concerns surrounding this incident, feel free to reach out to our responsive support team via the in-app chat, or drop us an email at support@gearset.com.
Historical Incident Log
Feb 12, 2025 12:08pm CST: Our team noticed the initial outage and took action to scale the affected infrastructure.
Feb 12, 2025 1:50pm CST: We successfully resolved the instability and took further precautions to prevent it reoccurring.
Feb 12, 2025 2:20pm CST: We observed an impact to users' pipelines where PRs are not opening to the next environment. We believe this is due to a delay in processing rather than an error. We are working to process our backlog of messages with the expectation that it will resolve the issue.
Feb 12, 2025 5:45pm CST: Incident is fully resolved and all pipelines are back in working order.