Quick takes on the latest Cloudflare public incident write-up
Cloudflare consistently generates the highest quality public incident writeups of any tech company. Their latest is no exception: Cloudflare incident on November 14, 2024, resulting in lost logs.
I wanted to make some quick observations about how we see some common incident patterns here. All of the quotes are from the original Cloudflare post.
In this case, a misconfiguration in one part of the system caused a cascading overload in another part of the system, which was itself misconfigured.
A very common failure mode in incidents is when the system reaches some limit, where it cannot keep up with the demands put upon it. The blog post uses the term overload, and often you hear the term resource exhaustion. Brendan Gregg uses the term saturation in his USE method for analyzing system performance.
A short temporary misconfiguration lasting just five minutes created a massive overload that took us several hours to fix and recover from.
