Reflecting on Research Practices
There are growing concerns about the quality, reliability, and repeatability of research results, across the broader computing community and more specifically in cyber security as well. In this Communications Security column, I highlight the need and value of studying research practices themselves—also called meta-research—to address those concerns. Meta-research contributes to improving the research process and supporting researchers in preserving the soundness and validity of their research. I will show how the security research community has already made good progress in engaging in such introspection, especially regarding its methods. Nevertheless, we should continue to expand these efforts to the whole research cycle, to further improve our research practices and ensure we can trust scientific findings to be reliable and reproducible.
Across all scientific disciplines, we increasingly see researchers questioning whether current scientific practices produce trustworthy research findings. For example, the ‘replication crisis’ emphasizes how studies may be difficult or even impossible to reproduce, which threatens the validity of their results. This has even led to statements that “most published research findings are false.”2 The security research community is not immune to such concerns: a recent study by Soneji et al. gave insight into potential shortcomings of the peer review process at top-tier security conferences,5 such as subjective evaluation metrics and perceived randomness in decisions.
