Summary of June 8 outage
We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change. We detected the disruption within one minute, then identified and isolated the cause, and disabled the configuration. Within 49 minutes, 95% of our network was operating as normal.
On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances.
Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors.
09:47 Initial onset of global disruption 09:48 Global disruption identified by Fastly monitoring 09:58 Status post is published 10:27 Fastly Engineering identified the customer configuration 10:36 Impacted services began to recover 11:00 Majority of services recovered 12:35 Incident mitigated 12:44 Status post resolved 17:25 Bug fix deployment began
Once the immediate effects were mitigated, we turned our attention to fixing the bug and communicating with our customers. We created a permanent fix for the bug and began deploying it at 17:25.
