Sentence transformers have revolutionized text processing in machine learning and AI by converting raw text into numerical vectors, enabling applications like vector similarity searches and semantic search. With pg_vectorize, an open source extension for Postgres, you have a seamless link to do vector search using over 120 open-source Sentence Transformers from Hugging Face, directly from Postgres.
pg_vectorize is completely open source, and you can run it locally or in your own environment by following the project’s quick-start guide. It is also available by default on the Tembo VectorDB Stack in Tembo Cloud and can be added to any Tembo Postgres instance.
Machine learning and artificial intelligence models ultimately perform mathematical operations. So if you have raw text and want to build a machine learning or AI application, you need to find a way to transform text to numbers. There has been detailed research in this field, and the resulting transformer models encode the meaning behind the raw text data into an array of floats. Once you have the array of floats (also called vectors or embeddings), now you can perform efficient mathematical operations such as vector similarity search and other forms of machine learning. SentenceTransformers is also the name of a popular open-source Python library that provides a simple interface to transformer models, and it is used in this project.
While OpenAI’s embedding API endpoint is commonly used, there are over 120 open-source sentence transformers available on Hugging Face. The right model to use depends on your data and your use case. For example, the all-MiniLM-L12-v2 model is a general purpose transformer useful for a wide variety of use cases such as training a supervised classification model or performing similarity search. Conversely, the multi-qa-MiniLM-L6-dot-v1 model was built specifically for semantic search.