jsDelivr May outage postmortem

submited by

Style Pass

2024-05-02 15:00:07

During the night, on May 2, 2024, the jsDelivr CDN domain cdn.jsdelivr.net started serving an expired SSL certificate to clients connecting from certain regions.

The outage lasted for more than 5 hours and affected users mostly in Africa, Asia, and certain countries in Europe and Latin America.

This disparity was due to our routing between our main CDN providers. Users that were hitting our Fastly CDN endpoint were unaffected.

The root cause of the outage was Cloudflare’s switch from DigiCert’s certificate authority to Google Trust Services. While the switch itself was benign, it also changed the domain validation method.

Since we’re a multi-CDN, with the traffic being routed between providers based on our own internal rules, we can’t use Cloudflare DNS hosting and have a special setup where they only act as the CDN, with DNS being hosted elsewhere.

To allow Cloudflare to automatically issue and manage our certificates, we added the proper DNS records to our third-party DNS providers. This system worked great for almost 10 years now.