A/B testing considered BS
On our launch post we sensationally stated that A/B testing is “bullshit”. We mean that: it's waste of time and promotes bad design. It's essentially a system for marketers to look smart.
Let's take an example website with a decent amount of traffic (10,000 monthly visitors) and a conversion rate of 2% (new signups). Then we set up an A/B test to see whether an update on the front page (like a new background color) would make a positive impact on the conversion rate.
To get statistically significant results we want to collect at least 400 conversions per variation before drawing any conclusions, which seems to be a recommended ballpark number to aim for *.
A quick math 2 * 400 / (10000 * 0.02) reveals that it takes 120 days to run the test before you can confidently draw any conclusions.
The WYSIWYG optimization tools are only designed to test small things like alternate colors, titles, or images. You cannot experiment with the stuff that matters like global design updates, brand renewals, product launches, or any major content updates spanning multiple pages.
