Machine Learning 103: Exploring LLM Code Generation¶

submited by
Style Pass
2023-05-22 11:00:08

This executable blog post is the third in a series related to machine learning and explores code generation from a 16 billion parameter large language model (LLM). After a brief look under the hood at the LLM structure and parameter allocation, we generate a variety of Python functions and make observations related to code quality and security. Similar to OpenAI's ChatGPT, Google's Bard, Meta's LLaMa and GitHub's Copilot, we will see some fantastic capabilities and a few intriguing misses. The results demonstrate that human expertise remains crucial, whether it be in engineering the prompt or in evaluating the generated code for suitability.

The model used here was introduced in the paper titled CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis from late 2022, with code on GitHub, and is supported via the Hugging Face framework. As the paper suggests, in some respects this model's performance is comparable to that of the OpenAI Codex model which powered the GitHub Copilot product. Since this technology is still in its infancy, any misadventures should be interpreted as opportunities for further research and the need for human technical expertise, rather than as criticism.

You can run everything for yourself by loading this .ipynb file into any Jupyter-based notebook system. The goal for the code below is to utilize state-of-the-art models while maximizing simplicity, understandability, and accessibility. This is consistent with the two prior posts in the blog series:

Leave a Comment