Implementing A Byte Pair Encoding (BPE) Tokenizer From Scratch
submited by
Style Pass
From the lengths above, we can see that a 42-character sentence was encoded into 20 token IDs, effectively cutting the input length roughly in half compared to a character-byte-based encoding
As we can see, most token IDs represent 2-character subwords; that’s because the training data text is very short with not that many repetitive words, and because we used a relatively small vocabulary size
Read more sebastianras...
