LLMs shouldn’t write SQL - by Benn Stancil - benn.substack

[ People can ] work like journalists, collating existing metrics and drawing conclusions by considering them in their totality. Rather than looking for new ways to assess a question, they start by asking, “how do we currently measure that?” [ Or they can ] work like scientists, creating new datasets and aggregating them in novel ways to draw conclusions about specific, nuanced hypotheses.

In business contexts, the first type of analysis typically consists of asking a series of Mad Lib-style questions about known metrics, like revenue retention or daily active users or ad spend, and aggregating and filtering those metrics by different dimensions. “Show me total orders in Massachusetts by month compared to the same number from a year ago,” someone might ask. This question will reveal something, like a spike in new orders in February. And that will prompt more questions of the same style—“now show me total orders in Massachusetts by month and by product category”—until people find whatever they’re looking for. 

These questions are often answered using BI tools. Most BI tools support this with more or less the same architecture. First, there’s some sort of data model in which people define metrics, how they should be calculated, and the ways they can be aggregated, filtered, and combined. And second, there’s an interface where people choose which metrics they want to see and how they want to aggregate, filter, and combine them. The former defines the structure of Mad Lib sentences and the words that people can choose to fill them in; the latter lets people fill them in and gives them tables and charts of their results. 2  

Leave a Comment