The project was to extend our DevQualityEval LLM LLM code generation benchmark with a new language: Ruby. We successfully used LLMs to transpile exist

Using LLMs to transpile code (from Go & Java to Ruby)

submited by

Style Pass

2024-08-30 13:30:03

The project was to extend our DevQualityEval LLM LLM code generation benchmark with a new language: Ruby. We successfully used LLMs to transpile existing Java and Go code (tasks and test cases) to Ruby. The workflow of this project and our experiences are described below.

Learning a new language is hard. To successfully extend DevQualityEval’s existing Go and Java repositories with Ruby with manual transpilation, we would have needed to know the basics, including:

We prompted LLMs to transpile the test repositories for each task in the DevQualityEval, allowing us to skip the learning process and providing an automated way for the transpilation.

We used a combination of 2 models for transpiling Go and Java code to Ruby: Claude 3.5 Sonnet and GPT-4o. We decided to use these two LLMs because our results show that they are among the most powerful in the tested languages (Java and Go). So we figured they had the potential to work well with Ruby too.

We prompted our chosen models to transpile the existing Java and Go test repositories in DevQualityEval to Ruby. DevQualityEval contains the following repositories for the evaluation tasks: