How we managed to speed up our CI to save 168+ days of execution time per month
We got a taste for speed after UV gave us back 215 compute hours a month. Our CI workflow gets triggered on each commit, and due to the high volume of contributions, it was triggered 6647 times in March 2024. The full runs, where the whole test suite is being executed, take a long time.
It turned out that the plugin, we used to split the tests into 10 groups, was inefficient. Each pytest job needed to discover all tests, even when the job intended to only execute a subset of them.
Now we have a separate job to discover all tests and split them into 10 groups. The 10 pytest jobs only need to execute a subset of all tests. Not doing full-discovery in each test runner saves us 3 hours on each full run!
Considering the 1378 successful full runs, we would have saved around 4042 hours ~ 168 days of execution time in March 2024 with #114381. Even more, if I had also analyzed the failed/canceled ones.
The more than 168 monthly saved execution days can be used by other jobs and make the CI experience for all developers and our community better. We improved our sustainability by using fewer resources to run our test suite.
