LLMs and Declarative Software Development

submited by

Style Pass

2023-03-26 19:30:05

Large language models (LLMs) and tools derived from them have taken the world by storm. Southpark even dedicated an episode this season to ChatGPT, dubbed “Deep Learning”. I have previously written about the implications of LLMs and the wonders of prompt engineering; and just a few short weeks ago, OpenAI released the latest iteration of their GPT family of models: GPT-4. If you haven’t, I’d highly recommend watching the 24 minute developer demo livestream.

LLMs are getting more and more powerful, and the universe of possibilities around potential applications continues to expand. One area that I’ve been keenly interested in is the application of LLMs to developer tools, and as the potential ultimate developer tool. LLMs are essentially token completion models, and excel in instances where a) it has tons of training data, and b) it’s dealing with a problem space with a finite, formal grammar. It’s not much of a surprise that GPT-4 was able to score within the top 10% of human test takers on Bar exam, as well as exceedingly well on a host of other limited tests. The reason GPT-4 is able to perform so well on standardized is because these are domains were there is lots of repetition, and the bounds of the domain are well defined.

It would seem to follow then that GPT-4 and LLMs like it should be extremely powerful tools for people who write software. Software systems deal with a formal grammar, and there is a host of readily available training data for LLMs to train on – this training data captures the common patterns that humans employ when writing software. In a paper titled On the Naturalness of Software, it was posited that because code is written by humans, it exhibits statistically significant repitions in designs, patterns, and ultimately tokens. These repetitions are patterns that an LLM can learn, reproduce, and build on. Greg Brockman’s live demo of GPT-4 writing a discord bot was just a small example of the power GPT-4 holds as an enabler of developer productivity. As software development becomes more declarative, we need to lean into our tools and learn how to use them in the most efficient ways.