Vector Databases: A Traditional Database Developer's Perspective
As a traditional database developer with machine learning platform experience from my time at Shopee, I’ve recently been exploring vector databases, particularly Pinecone. Rather than providing a comprehensive technical evaluation, I want to share my thoughts on why vector databases are gaining significant attention and substantial valuations in the funding market.
At its core, a vector database primarily solves similarity search problems. While traditional search engines like Elasticsearch (in its earlier versions) focused on word-based full-text search with basic tokenization, vector databases take a fundamentally different approach.
Consider searching for “Microsoft Cloud” in a traditional search engine. It might find documents containing “Microsoft” or “Cloud” individually, but it would likely miss relevant content about “Azure” - Microsoft’s cloud platform. This limitation stems from the basic word-matching approach of traditional search engines.
One common misconception I’ve noticed is that vector databases must use Large Language Models (LLMs) for generating embeddings. This misconception has been partly fueled by the recent RAG (Retrieval-Augmented Generation) boom and companies like OpenAI potentially steering users toward their expensive embedding services.
