Measuring the impact of AI coding tools

Every software engineering team is looking at AI tools for coding. In a Stack Overflow survey, 76% of developers reported they are either using or planning to use AI tools for development. However, with the pace of change in this space, many are already disillusioned with the tools they adopted during the early hype. There is little doubt of AI’s promise to transform software development. But how can engineering teams measure the ROI of these tools in a way that aligns with their priorities? These are just some of the questions we are getting from folks who are considering AI coding tools. So, we want to share a few things we learned from our work with some of the most sophisticated software engineering teams in the world.

Before measuring return on investment (ROI), we must think about what the said investment is supposed to accomplish. If the goal of any software development organization is to efficiently build and operate software that delivers value, then any tooling investment should help achieve this goal. Questions that need to be answered:

Software engineering is, by design, an innovative pursuit. While convenient, thinking of developer workflows as factory operations is faulty. Whether software developers are happy using their tools and want to come back to them repeatedly is also an important factor in determining whether a tool has been successful in a given setup. Therefore, the third important question in determining the value of a tool is:

Leave a Comment