You Should Probably Pay Attention to Tokenizers

submited by

Style Pass

2024-10-23 11:00:08

Last week I was helping a friend of mine to get one of his new apps off the ground. I can’t speak much about it at the moment, other than like most apps nowadays it has some AI sprinkled over it. Ok, maybe a bit maybe more just a bit – depends on the way you look at it, I suppose.

There is a Retrieval-augmented generation (RAG) hiding somewhere in most of the AI apps. RAG is still all the RAGe – it even has its own Wikipedia page now! I’m not sure if anyone is tracking how fast a term reaches the point where it gets its own Wiki page but RAG must be somewhere near the top of the charts.

I find it quite intriguing that most of the successful AI apps are basically clever semantic search apps. Google search got [somewhat] unbundled at last which kind of makes me think whether their not unleashing all the LLM tech way earlier was behind all of this. But I digress.

The app my friend has been building for the past couple of weeks deals with a lot of e-commerce data: descriptions of different items, invoices, reviews, etc. The problem he was facing was that the RAG wasn’t working particularly well for some queries, while it worked very well for others.