Search code, repositories, users, issues, pull requests...
llm.rust is my attempt to get more into Rust, after the inspiring work made by Karpathy with llm.c. The code is pure simple Rust, no need for CUDA, and it runs well on CPUs - so no need of GPUs. The idea is to have a simple toolkit, written in Rust, that can run on any given GPT-like model's weights, so that we can continue the training steps and we can also fine tune the model given an input corpus.
If you want to run the code, you can test it against Karpathy's small Shakespeare text tinyshakespeare. You can download the text using the python code:
This will download the input corpus in a data folder. The text is convert the input text to input training and validation tokens (tiny_shakespeare_train.bin and tiny_shakespeare_val.bin respectively). The text is tokenised with GPT-2 tokenizer.
This will run the training, along with validation tests and some inference. This is an example of output you should get from the code:
