Test Flakiness Across Programming Languages

submited by
Style Pass
2023-03-15 12:00:06

How are programming languages similar, and how do they differ? Most arguments about this are based on lists of language features, but another way to tackle the question is to ask how different languages behave in practice. This paper does that by looking at how flaky tests in C, Go, Java, JavaScript, and Python are similar and how they differ. The authors find that things like concurrency are hard everywhere, but other things (like resource management) vary from language to language. They also found that less than a dozen strategies for fixing flaky tests accounting for 85% of fixes, which suggests that explicitly teaching people bug-fixing and refactoring patterns for tests would be beneficial. Finally, they found that people either fix flaky tests right away or leave them broken for a long time, suggesting either that some tests are useful and some are not, or that some projects have a culture of clean coding and others don't with little middle ground.

Keila Costa, Ronivaldo Ferreira, Gustavo Pinto, Marcelo d'Amorim, and Breno Miranda. Test flakiness across programming languages. IEEE Transactions on Software Engineering, pages 1–14, 2022. doi:10.1109/tse.2022.3208864.

Leave a Comment