The Rise, Fall, and Future of Vector Databases: How to Pick the One That Lasts
Special thanks to Doug Turnbull, Daniel Svonava, Atita Arora, Aarne Talman, Saurabh Rai, Andre Zayarni, Leo Boytsov, Pat Lasserre and Bob van Luijt for reading and commenting the drafts of this post
Jo Kristian Bergum recently wrote a massively influential X post: “The rise and fall of the vector database infrastructure category”.
It is all good to chart what happened or happens on the industry level, but what a regular user should do? How to choose a vector database in this vastly fragmented space? This runs on the assumption, that for your product or project idea you’ve decided and are certain, that you need to use vector embeddings to represent semantic (or other) relations between objects (products, texts, images, videos, emails and so on).
As you know, in my own way I participated in creating the emerging trend on vector databases by interviewing and cataloging the players present on the market 3 years ago. Behind the scenes I even advised a VC firm to pick a vector database between (back then) two main contenders. I have talked about Vector Search and Vector Databases in many forums, including Berlin Buzzwords (in 2021 and 2022), Haystack, London IR Meetup, Open NLP Meetup. I have also launched the Vector Podcast in 2021 as a way to explore this new trend. Of course since then the podcast has expanded to more topics, like LLMs, search, data science, neural search frameworks. And, most recently, I have discussed vector search and vector databases in our recent course on LLMs and Generative AI.
