Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations
Nature Machine Intelligence (2024 )Cite this article
One of the most promising applications of machine learning in computational physics is to accelerate the solution of partial differential equations (PDEs). The key objective of machine-learning-based PDE solvers is to output a sufficiently accurate solution faster than standard numerical methods, which are used as a baseline comparison. We first perform a systematic review of the ML-for-PDE-solving literature. Out of all of the articles that report using ML to solve a fluid-related PDE and claim to outperform a standard numerical method, we determine that 79% (60/76) make a comparison with a weak baseline. Second, we find evidence that reporting biases are widespread, especially outcome reporting and publication biases. We conclude that ML-for-PDE-solving research is overoptimistic: weak baselines lead to overly positive results, while reporting biases lead to under-reporting of negative results. To a large extent, these issues seem to be caused by factors similar to those of past reproducibility crises: researcher degrees of freedom and a bias towards positive results. We call for bottom-up cultural changes to minimize biased reporting and so top-down structural reforms to reduce perverse incentives for doing so.
The lists of authors and articles generated during the systematic review and the categorizations of every article in the random samples are publicly available at https://doi.org/10.17605/OSF.IO/GQ5B3 (ref. 124).
