Forecasting Accuracy
Reader Dan Turner pointed me toward a highly interesting paper co-authored by Tetlock, about the bin model of forecasting.11 Bias, Information, Noise: The bin Model of Forecasting; Satopää, Salikhov, Tetlock, Mellers; Management Science; 2021. Preprint available on Google Scholar.
This makes it clear that e.g. teaming improves the Brier score by 0.05 over trained individuals, and this comes mainly from noise reduction (0.03), but also to some extent from bias reduction and information increase (0.01 each).
Forecasts are often evaluated using the Brier score. You can find the full details elsewhere on the web, but it’s useful to know that the Brier score effectively runs from 0 (perfect predictions every time) through 0.25 (clueless about everything.) As may be clear already, a lower Brier score is better. Nobody gets a 0, and presumably people are a little better than 0.25, but how well do people perform?
What this bin paper finds out, which is really fascinating, is some of the mechanics behind these improvements. What exactly is it about a person’s forecast that changes when they are put in a team? Let’s find out!
