A/A Testing
An A/A test, (also known as an uniformity trial), is an experiment where two populations are exposed to the same experience (in contrast to an A/B test where the groups are exposed to different experiences). The technique is primarily useful to validate the testing infrastructure and cohort assignment for A/B testing.
Running A/A tests is a critical part of establishing trust in an experimentation platform. The idea is so useful because the tests fail many times in practice, which leads to re-evaluating assumptions and identifying bugs.
Running an A/A test is straightforward if you have the capability to run A/B tests. To run an A/A test, you should use the exact same infrastructure and tooling that you use for A/B testing, but not introduce any changes into your product.
Unlike an A/B test that generally runs only once, A/A tests require many successive re-runs. A single test might not catch intermittent or subtle issues. Likewise, an individual failure might not point to any infrastructure issues. Expect an A/A test to fail regularly. If you evaluate tests based on a 95% confidence interval, A/A tests should fail around 5% of the time.
