Root Cause Analysis for Render incident - September 19th

On September 19 2021, between 8:42 UTC and 13:30 UTC, all web services, databases and cron jobs in the Frankfurt region were unavailable due to a platform outage. Private services and background workers experienced intermittent degraded services during this period.

The incident started when a configuration change meant for an internal environment was inadvertently applied to to the Frankfurt production environment. This caused a critical set of networking resources to become unavailable, and the failure state required our engineers to recreate these resources manually.

Reliability remains our top priority and we remain committed to the highest standards of service across all Render regions. We are rigorous about incident analysis, and as a result of this incident, we have both planned and completed mitigations to the Render platform to prevent similar outages in the future. These are detailed in the Mitigations section below.

Since uptime is the top business and technical priority at Render, we are giving all affected customers a credit on their Render bill for September 2021. We've sent out an email with details; if you were affected by the outage but did not receive this email, please reach out to support@render.com.

Leave a Comment