ShishirPatil/gorilla
Gorilla enables LLMs to use tools by invoking APIs. Given a natural language query, Gorilla comes up with the semantically- and syntactically- correct API to invoke. With Gorilla, we are the first to demonstrate how to use LLMs to invoke 1,600+ (and growing) API calls accurately while reducing hallucination. We also release APIBench, the largest collection of APIs, curated and easy to be trained on! Join us, as we try to expand the largest API store and teach LLMs how to write them! Hop on our Discord, or open a PR, or email us if you would like to have your API incorporated as well.
Evaluation: We have included prompts and responces for the APIBench with and without retrievers along with the Abstract Syntax Tree (AST) matching evaluation script at evaluation.
For our dataset collections, all the 1640 API documentation is in data/api. We also include the APIBench dataset created by self-instruct in data/apibench. For evaluation, we convert this into a LLM-friendly chat format, and the questions are in eval/eval-data/questions, and the corresponding responces are in eval/eval-data/responses. We have also included the evaluation scripts are in eval/eval-scripts. This would be entirely sufficient to train Gorilla yourself, and reproduce our results. Please see evaluation for the details on how to use our evaluation pipeline.
