Tech Stories by Dmitry Kan
Jo Kristian Bergum recently wrote a massively influential X post: “The rise and fall of the vector database infrastructure category”.
Tech Stories by Dmitry Kan is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
It is all good to chart what happened or happens on the industry level, but what a regular user should do? How to choose a vector database in this vastly fragmented space? This runs on the assumption, that for your product or project idea you’ve decided and are certain, that you need to use vector embeddings to represent semantic (or other) relations between objects (products, texts, images, videos, emails and so on).
As you know, in my own way I participated in creating the emerging trend on vector databases by interviewing and cataloging the players present on the market 3 years ago. Behind the scenes I even advised a VC firm to pick a vector database between (back then) two main contenders. I have talked about Vector Search and Vector Databases in many forums, including Berlin Buzzwords (in 2021 and 2022), Haystack, London IR Meetup, Open NLP Meetup. I have also launched the Vector Podcast in 2021 as a way to explore this new trend. Of course since then the podcast has expanded to more topics, like LLMs, search, data science, neural search frameworks. And, most recently, I have discussed vector search and vector databases in our recent course on LLMs and Generative AI.
I personally did not feel like it was a gold rush, because a database to me is just a tool: it either covers the facets of your use case, is customizable, scalable and performant, or not. I do realize that companies behind these technologies have raised money to attract talent and build a better version of themselves over time. But it is still a tool, and I sometimes found in my client practice and own projects some of the vector databases simply not fulfiling the basic needs of the application at hand.
