What AI engineers can learn from qualitative research methods in HCI

At first glance, qualitative research and the highly technical, benchmark-oriented world of AI/ML seem to have nothing to do with one another. But in reality, software developers building on AI models could learn a lot from qualitative research methods.

Hamel Husain, a consultant in AI/ML engineering, has recently been holding open office hours on LLMOps. What caught my eye was this advice, which he gave to many developers building on LLMs:

“[L]ook at your logs/traces — start with 30 or so. Start categorizing the errors and issues you see. Keep looking at logs and traces until you feel like you aren’t learning anything new. In the end, you will know where your biggest issues are. You prioritize those! You will also get a sense of what is most important to measure (and how).”

Taking a pile of qualitative data, categorizing and clustering it, and looking at data until you “aren’t learning anything new”?

Leave a Comment