How to be a Curious Consumer of COVID-19 Modeling: 7 Data Science Lessons from ‘Effectiveness of COVID-19 shelter-in-place orders varied by state’ (Feyman Et Al 2020)

submited by

Style Pass

2024-06-06 23:30:02

Acknowledgments: The author wishes to thank Yevgeniy Feyman for suggesting their paper for this review.1 I am also grateful for feedback and corrections from a number of people, particularly Andres Gannon, Thomas Scherer, Marc Jaffrey, and Andrew Gelman.

How should non-epidemiologists publicly discuss COVID-19 data and models? I asked this question at the start of the pandemic when law professor Richard Epstein claimed COVID-19 would not be a serious global health threat and that little intervention was required (Douglass 2020) . I argued that the high school debate style of the legal tradition did not lend itself to scientific rigor, honesty, or curiosity, and drew some lessons from social science to encourage better empiricism going forward. Four years later, we have tens of thousands of empirical COVID-19 papers from the social sciences, but the state of empiricism has hardly improved. Methodologically weak but seemingly sophisticated techniques are routinely pressed into service to provide answers where the data on hand cannot possibly, mathematically, provide them (Douglass, Scherer, and Gartzke 2020) . So now I return again to the question of how we can promote more careful and curious analysis. Specifically, I draw 7 data science lessons for what to do and not do with COVID-19 modeling illustrated with a simple empirical paper on the causal effects of government lockdowns on individual mobility (Feyman et al. 2020) . It is my hope the reader can take away practical tips for how they can begin to challenge and interrogate seemingly sophisticated modeling- especially modeling that is obfuscated, vague, or claims more than the underlying evidence can support.

Feyman et al. (2020) ask whether ‘shelter-in-place’ (SIP) orders reduced individual mobility at the start of the COVID-19 pandemic, and if so where and why. It is an important question. Whether and to what degree governments can influence public behavior during an emergency has major ramifications for both pandemic response and broader social policy in general (Herby, Jonung, and Hanke 2023; Prati and Mancini 2021; Wilke et al. 2022; Vaccaro et al. 2021) .