LLMs in production: lessons learned - Duarte O.Carmo

A couple of months ago nobody asked me about my work. Something related to computers and AI. Fast forward to today, even my uncle asks me about ChatGPT. The hype is real. Only time will tell if the hype will materialize. But while the world wonders, work goes on.

In the last couple of months, I've helped develop a product that leverages this tech at its core. It was - to say the least - a learning experience. Full of lessons learned, full of little traumas and things I would've done better. In the hopes of helping someone out there, here are some lessons I've learned along the way.

With so much hype surrounding LLMs, it's easy to think they'll solve all problems in Machine Learning. Or at least the ones related to NLP. From what I've seen, this is hardly the case. Let's split Machine Learning tasks into two types. (1) Predictive tasks, such as classifying the sentiment of a tweet, and (2) generative tasks, such as summarizing the content of an article.

GPT is great at generative tasks. Writing an email with context, writing a summary of a web page, and creating an article given some ideas. These are a very specific subset of Machine Learning. Most of the problems we face are predictive problems: what is the sentiment of this tweet? What is the class of this image? It's hard to tell exactly how good these models will become for the predictive use case. But before throwing GPT at whatever you're facing, think about the use case. This leads me to my next point.

Leave a Comment