Incident with Actions

Actions

1. 26 Aug 18:01 UTCResolvedOn August 26, 2026 from 15:02 to 15:45 UTC, Actions jobs failed to start. The following 2 hours until 17:40 UTC, Actions runs were delayed starting by more than 5 minutes as the system caught up with delayed load. This impact was triggered by saturation of writes to the database primary used by the service processing triggers for Actions workflows. The primary was failed over, but the system did not fully recover. The saturation was caused by growing daily peak load combined with an upstream issue in GitHub’s event processing infrastructure, https://www.githubstatus.com/incidents/hcbtzksccj2f, which caused burst amplification of already-high load. Downstream throttles that were later used to recover were set ~10% too high to protect the system. At 15:45 UTC, throttling combined with service restarts recovered the service’s core health. Those throttles were gradually raised between 15:54 and 17:22 to restore full webhook processing for Actions runs. This ramp was deliberately slow to ensure we did not re-overwhelm the system given our original throttling was now known to be incorrectly set. The queue of webhook events was fully burned down at 17:40 UTC. 3.7% of larger-runner jobs, along with some scale-set self-hosted jobs, remained stuck in queued or “waiting for runner” state. We deployed a change to force-revoke jobs in this state, and they transitioned to failed at 18:40 UTC, about 50 minutes after incident mitigation. Releasing these jobs also freed hosted concurrency for larger-runner jobs. Customers using concurrency groups saw longer impact due to a separate issue w…
2. 26 Aug 18:00 UTCUpdateAll inbound queues have recovered and Actions is operating as expected. 3.7% of jobs assigned to larger runners during the early stage of this incident are stuck waiting for runner assignment. Those will be canceled within the hour. Other runners are successfully processing all new jobs.
3. 26 Aug 17:54 UTCMonitoringThe degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
4. 26 Aug 17:32 UTCUpdateWe are continuing to observe recovery and expect actions inbound queues to be back to normal in <30min. Work will continue to flow through the system subject to per-customer concurrency limits.
5. 26 Aug 16:50 UTCUpdateWe are continuing to observe recovery and delayed queues are burning down. Some customers will continue to see increased delays until all throttled work has been completed - we expect this within the next hour.
6. 26 Aug 16:49 UTCUpdatePages is operating normally.
7. 26 Aug 16:14 UTCUpdateWe believe we've identified and addressed the issue and are ramping traffic back up slowly to ensure it doesn't recur. Some customers will continue to see delays as we ramp up.
8. 26 Aug 15:48 UTCUpdateprimary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues
9. 26 Aug 15:23 UTCUpdateWe've identified an issue with a database primary and are failing over to a replica immediately
10. 26 Aug 15:12 UTCUpdatePages is experiencing degraded performance. We are continuing to investigate.
11. 26 Aug 15:11 UTCInvestigatingWe are investigating reports of degraded availability for Actions

No field changes in the recorded history.

Cite this recorded revision to keep the same title, summary, facts and links when the page changes.

A day selects the latest observation by the end of that UTC day; today's selection can change until midnight. History starts when this installation records it.

• Service status now: every provider ChangeIntel reads, with ongoing incidents first.
• Status RSS feed: new and ended incidents as they happen.
• Everything about GitHub: its changes, known issues and incidents on one timeline.