Correlations and Coin Flips - Cremieux Recueil

“Variance explained” (r²) is a misleading metric. One reason is that its scale is unintuitive. A correlation denotes the gain in Y if you change X by one standard deviation (SD). The problems that follow if you square that value were explained well by Hunter and Schmidt:

The “percentage of variance accounted for”… leads to severe underestimates of the practical and theoretical significance of relationships between variables. This is because r² (and all other indexes of percentage of variance accounted for) are related only in a very nonlinear way to the magnitudes of effect sizes that determine their impact in the real world.

The ballpark is ten miles away, but a friend gives you a ride for the first five miles. You’re halfway there, right? Nope, you’re actually only one quarter of the way there…. This makes no sense in real life, but, if this were a regression, the "r-squared" (which is sometimes called the "coefficient of determination") would indeed be 0.25, and statisticians would say the ride "explains 25% of the variance." There are good mathematical reasons why they say this, but they mean "explains" in the mathematical sense, not in the real-life sense.

Alternative, intuitive metrics to r² have been proposed. A popular one introduced by Rosenthal and Rubin in 1982 is the binomial effect size display, or BESD. The BESD is an excellent way of conveying how something that explains “only” a small share of the variance in something else can have enormous practical implications. Consider this table:

Leave a Comment