P-hacking occurs when scientists engage in various behaviors that increase their chances of reporting statistically significant results. Typical p-hac

Pascal’s Newsletter

submited by

Style Pass

2025-01-24 02:30:04

P-hacking occurs when scientists engage in various behaviors that increase their chances of reporting statistically significant results. Typical p-hacking practices include:

P-hacking is absent from hypothesis-testing theory, but it is ubiquitous in reality. In fact, p-hacking is so prevalent that scientists openly admit that they p-hack:

It is not surprising that p-hacking is so prevalent because scientists have large incentives to p-hack. This is because scientific journals prefer publishing significant results. But p-hacking threatens scientific progress. It leads to excessive rejection of established paradigms and to the unwarranted adoption of new paradigms. One manifestation of uncontrolled p-hacking is the replication crisis in science.

In a paper recently accepted at REStat, Adam McCloskey and I work on the p-hacking problem. Our main contribution is to derive critical values that correct the inflated type 1 error rate (false positive rate) caused by p-hacking. The paper will come out in print in 2026; meanwhile the final version is online at pascalmichaillat.org/12/.