Why is ChatGPT so bad at math?
If you’ve ever tried to use ChatGPT as a calculator, you’ve almost certainly noticed its dyscalculia: The chatbot is bad at math. And it’s not unique among AI in this regard.
Anthropic’s Claude can’t solve basic word problems. Gemini fails to understand quadratic equations. And Meta’s Llama struggles with straightforward addition.
Tokenization has something to do with it. The process of dividing data up into chunks (e.g., breaking the word “fantastic” into the syllables “fan,” “tas,” and “tic”), tokenization helps AI densely encode information. But because tokenizers — the AI models that do the tokenizing — don’t really know what numbers are, they frequently end up destroying the relationships between digits. For example, a tokenizer might treat the number “380” as one token but represent “381” as a pair of digits (“38” and “1”).
AI systems are statistical machines. Trained on a lot of examples, they learn the patterns in those examples to make predictions (like that the phrase “to whom” in an email often precedes the phrase “it may concern”). For instance, given the multiplication problem 5,7897 x 1,2832, ChatGPT — having seen a lot of multiplication problems — will likely infer the product of a number ending in “7” and a number ending in “2” will end in “4.” But it’ll struggle with the middle part. ChatGPT gave me the answer 742,021,104; the correct one is 742,934,304.
