Search code, repositories, users, issues, pull requests...
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
When I add https://huggingface.co/meta-llama/Meta-Llama-3-70B/blob/main/original/tokenizer.model I get the same error as on convert.py
@m18coppola the instruct models use two different EOS tokens: the standard one (<|end_of_text|>), and a second one that signals the end of the assistant turn (<|eot_id|>). Generation must stop when either one is encountered.
I'm not sure how to replicate this behaviour yet. The best solution would be to use a list of eos/stop tokens, but I don't know how to do it, any suggestions on where to look?
Another idea would be to use <|eot_id|> (the assistant finalization token) as the only EOS when converting an instruct model, and <|end_of_text|> when converting a pre-trained model.
