Building an image search engine on Postgres
One of the more interesting e-commerce features I’ve seen recently is the ability to search for products that look like an image I have on my phone. For example, I can take a picture of a pair of shoes or other product then search a product catalog to find similar items. Getting started with a feature like this can be a fairly straight forward project the right tools. And if we can frame a problem as a vector search problem, then we can use Postgres to solve it!
In this blog we’ll build a basic image search engine using Postgres. We’ll use a pre-trained model to generate embeddings for images and text, then store those embeddings in Postgres. The pgvector extension will enable us to conduct similarity searches on these embeddings using both images and raw-text as queries.
In 2021, OpenAI published a paper and model weights for CLIP (Contrastive Language-Image Pre-Training), a model trained to predict the most relevant text snipped given an image. With some clever implementation, this model can also be used as the backbone for a search engine that accepts images and text as the input queries. We can transform images into vectors (embeddings), store the image’s embeddings in Postgres, use extensions to conduct similarity searches on these vectors, and use this to build an image search engine on top of Postgres. There are many open source variants of CLIP models available on Hugging Face but we will use OpenAI’s clip-vit-base-patch32 mode for this demonstration.
