What Microsoft, GitHub, Cloudflare, Ubiquiti, Cisco Meraki, Palo Alto Networks, GitLab, OpenAI, Anthropic, and Google post on their public status pages, read every 5 minutes while something is open. Each incident shows what else was happening and what changed in the same products just before.
Monitoring shows failures are still occurring in a tool used by Duo Code Review. The service remains available, but reviews may contain less context than expected. A fix is merged and awaiting deployment; we’ll update once recovery is confirmed.
4 earlier updates
13 Oct 00:29 UTCMonitoringErrors affecting Code Review tools have decreased, and the tools appear to be operating normally. We’re monitoring error rates and investigating the improvement. A fix is merged but not yet in production, so recovery is still being validated.
13 Oct 00:28 UTCMonitoringErrors affecting Code Review tools have decreased, and the tools appear to be operating normally. We’re monitoring error rates and investigating the improvement. A fix is merged but not yet in production, so recovery is still being validated.
12 Oct 13:37 UTCIdentifiedWe've identified the cause of the issue affecting GitLab Duo Workflow operations. Duo Code Review remains available, but reviews may contain less context than expected. Our engineering teams are expediting a fix and will continue monitoring until service is fully restored. For more details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/23158
12 Oct 11:30 UTCIdentifiedWe’re investigating an issue affecting some GitLab Duo Workflow operations. Duo Code Review results may be incomplete or lower quality than expected. Our engineering teams are working on mitigation and will provide further updates. More details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23158
Around it6 other incidents at the same time · 1 earlier incident · 1 change in the 14 days before
Elsewhere at the same time · within 12 hours of its start
GitLab.comJob delays on saas-linux-small-amd64 hosted runnerslast seen 21:02 UTC≥ 6 h 30 min
Started 13 Oct 14:27 UTCUpdated 20 h 31 min ago
CI/CDCI/CD - Hosted runners on Linux
Job start times for CI/CD jobs on GitLab.com small Linux x86-64 hosted runners have returned to normal, and the backlog of pending jobs has been processed. The delays were caused by errors and capacity constraints from our cloud provider when provisioning runner capacity in one region, between approximately 13:17 and 19:15 UTC. Jobs that were delayed are now starting as usual. We are monitoring to make sure performance remains stable and will post an update when this incident is resolved.
2 earlier updates
13 Oct 15:46 UTCIdentifiedWe have identified an ongoing Google Cloud Compute Engine issue affecting runner-capacity provisioning in us-east1. Some GitLab.com CI/CD jobs using small Linux x86-64 hosted runners may remain pending longer than usual before starting. Our engineering teams are working with Google Cloud to restore normal job start times. More details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23176
13 Oct 14:27 UTCInvestigatingWe are investigating increased queue times for CI/CD jobs using GitLab.com small Linux x86-64 hosted runners. Affected jobs may remain pending longer than usual before starting. We have identified errors and delays when provisioning runner capacity and are working to restore normal job start times. More details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23176
Around it6 other incidents at the same time · 1 earlier incident · 2 changes in the 14 days before
GitLab.comErrors and slow performance on merge requests in the specific groups.last seen 10:19 UTC≥ 2 days
Started 10 Oct 04:49 UTCUpdated 4d ago
WebsiteAPIGit OperationsCI/CD
We have applied a mitigation and are seeing sustained recovery in error rates and performance on merge request and pipeline pages for the affected projects. We are continuing to monitor. Details in gitlab.com/gitlab-com/gl-infra/production/-/work_items/23150
2 earlier updates
10 Oct 06:26 UTCMonitoringWe have applied a mitigation and are seeing sustained recovery in error rates and performance on merge request and pipeline pages for the affected projects. We are continuing to monitor. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23150
10 Oct 04:49 UTCInvestigatingWe are investigating reports of errors and slow performance on merge request and pipeline pages for projects in the specific groups. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23150
Around it3 other incidents at the same time · 1 change in the 14 days before
Elsewhere at the same time · within 12 hours of its start
Bars compare how long each incident lasted, up to three days; a lighter bar is a lower bound and a hatched one is unknown. Older incidents are kept for 90 days on the timeline.
Azure post-incident reviews
What went wrong in major incidents and what Microsoft is changing