How Optimizely (Almost) Got Me Fired [REPOST]

This is a repost of an article originally written and published by Peter Borden, and which was published on the blog of the now-defunct social analytics company, SumAll. The original article was published here - http://blog.sumall.com/journal/optimizely-got-me-fired.html - which is now a dead link.

Why am I posting this? I came across this article back when Peter wrote it in 2014, and I found it to be an essential part of the discourse in the A/B testing industry. A/B testing has steadily risen in prominence over the last decade, with more and more business users utilizing testing software and running experiments within their organization. And so I wanted to re-publish the original article, since the original link is now dead, to help preserve its educational value for anyone interested in learning about A/B testing.

This particular article from Peter shines a light on an important topic in the world of testing that can often be overlooked by novice users. Which is the risk of arriving at a false positive due to a lack of understanding about the differences between a 1 vs 2 tailed test, as well as the risk of not letting your test run long enough to give you statistically backed results.

Leave a Comment