This repository offers a straightforward and cost-effective method to compare the retrieval quality of various embedding models using a collection of

Search code, repositories, users, issues, pull requests...

submited by

Style Pass

2024-11-06 03:30:03

This repository offers a straightforward and cost-effective method to compare the retrieval quality of various embedding models using a collection of unpaired queries and documents. For each embedding model, the process involves leveraging a language model to score the top $k$ documents retrieved and then calculating the average score across these documents and queries. This allows users to perform a head-to-head comparison by analyzing the average scores.

We repeat this process for each embedding model and output the overall average score. Users can perform a head-to-head comparison by analyzing the average scores.

We need to invoke APIs for embedding models and generative language models. To configure the API keys using environment variables, please store them in .env file located in the root directory of your project.

For each query, use a set of embedding models to get its top $k$ candidate documents. Embeddings are saved in ./data/{task_name}/embedding.db. Generated candidates are saved under the folder ./data/{task_name}/meta_data in the format of {query_id: [document_ids]}.