Milestones on Our Journey to Standardize Experimentation at The New York Times
Experimentation is an important part of product development at The New York Times — it helps us measure the impact of our changes, ship features safely, and gain insight into user behavior. Experiments are conducted by many different teams working across a diverse portfolio of products and projects, from Wordle to The Times’s Audio app to backend migrations. In the early days, this meant that teams would often develop their own approach to experimentation to address their specific requirements.
Over the past year, we have been on a journey toward standardization. There are many benefits to centralizing experimentation — having a single source of truth, observability into how all experiments impact important metrics, and easier comprehension of results from other teams (not to mention the time saved for individuals who would otherwise maintain their own pipelines!). On the flip side, teams had originally developed their own methodologies and processes for a reason — they were tailored for their unique needs.
At The New York Times, most product experiments use our internal experimentation platform, ABRA, which is short for A/B Reporting and Allocation architecture. Here’s a look at an early version. A lot has changed since those days, and we want to share some of the things we’ve learned on our standardization journey. While we use other types of experimentation, such as contextual multi-armed bandits, this article is focused specifically around the ways we’ve improved our A/B testing processes.
