Building a Healthy On-Call Culture
In the past, on-call duty was often associated with doctors, but in recent years, it’s become common for software engineers to be asked to be available for support work on short notice. As software has grown to power so much of the world, the need for high availability and rapid incident response has likewise grown.
Another driver of the popularity of on-call work is the ubiquity of smartphones. The days of bulky, unreliable pagers are gone. Now when automated software monitoring detects a system anomaly, an on-call engineer gets an alert in the form of a phone call, a text, or a loud noise from a mobile app.
First and foremost, on-call duty for SoundCloud engineers is optional. We believe this is important to our engineering culture for reasons I’ll discuss below. Secondly, on-call duty outside of normal office hours is compensated at an hourly rate, with additional hourly payments when responding to pages.
On-call engineers are organized into rotations. Each rotation consists of a group of engineers representing a team or, more typically, several teams. At any time, there’s always one engineer on call for the rotation. That engineer is expected to provide first-level support for the systems belonging to all the teams in the rotation.
