Safely repairing broken builds with ML

submited by

Style Pass

2024-04-24 03:30:06

We strive to create an environment conducive to many different types of research across many different time scales and levels of risk.

Automatically repairing non-building code increases productivity as measured by overall task completion and appears to introduce no detectable negative impact on code safety, provided that high quality training data and responsible monitoring are employed.

Software development is a cyclical process of designing, writing, building, testing, deploying, and debugging. If you’ve ever resolved a build error only to introduce 20 more, you’re familiar with the frustration of hunting down a type mismatch in unfamiliar code, or diagnosing where you went wrong using some new API.

At Google, every time a source code file is saved, a snapshot of the code is saved to version control. Every time a build runs, build logs are saved. This is a data treasure trove! We can determine when a build broke, what error messages were present, when the build succeeded, and what code changed — essentially, exactly how the developer fixed the build.

Today we describe how we trained a DIDACT ML model to predict build fixes. DIDACT, as we previously discussed, is a methodology that uses the software development process as a whole as training data. Using this dedicated build fix ML model, we then surfaced fixes in an IDE and ran a controlled experiment on this feature. This experiment showed statistically-significant gains across several measures of productivity, including a 2% increase in the number of code changes. These gains are not only good for their own sake, but they also remove developer toil through automation, allowing developers more time for creative problem solving. This promotes focus by removing obstacles, thus keeping developers in their flow state longer. Indeed, we now observe that about 34% of users experiencing a build breakage in a given month end up applying such ML-suggested fixes. We also detect no observable increase in safety risks or bugs when ML-generated fixes are applied. ML-powered build repair is now enabled for all Google developers and was recently featured at Google Cloud Next.