AGI requires better retrieval, not just better LLMs
TLDR; this blog post highlights the importance of retrieval for building AI products that feel like AGI, and goes over some of the most common pitfalls of current search techniques. It presents new retrieval benchmarks to help developers pinpoint the most common failure modes for their use cases.
Artificial General Intelligence is about developing AI that can learn new information on the fly, just like a human being you might hire and train. Despite Large Language Models scoring in the top 10% across nearly every STEM subject, we still haven’t reached AGI. Why? Because we need better contextualized models that can connect to knowledge bases and seamlessly retrieve information as effortlessly as human memory.
Yet, despite this critical bottleneck in retrieval, the vast majority of AI systems in production rely on basic semantic search to provide context. A single retrieval call into a vector database powers most Retrieval-Augmented Generation systems today. If you’ve tried using models like these, you know exactly how limited they are in truly understanding your data.
Not only is this pipeline unable to do anything other than a simple needle in haystack, but the worst part is that very few developers have evaluations to formally verify where and when this pipeline just does not work.
