Language Model Reasoning in Base64
Let’s say that reasoning is the process of manipulating known information using a logical process to arrive at valid conclusions. Large Language Models (LLMs) like Claude 3.5 Sonnet or GPT-4o have demonstrated strong reasoning abilities in literary, mathematical and various other domains where humans have performed well, primarily due to their innate abilities to reason. But whether LLMs reason in ways similar to humans’ is not well understood. This post explores some experiments that test for similarities between human and LLM mathematical reasoning, in how they generalize across languages.
Figure 1. An illustration of how certain types of reasoning is independent of language. A problem posed in English (orange) or French (purple) evokes the same reasoning process, leading to the same answer in different languages.
Humans’ abilities to solve math problems is independent of language – if you can (or cannot) solve a problem stated in English, and if you understand, say French, you will (or will not) be able to solve the same problem presented to you in French (see Figure 1). In other words, the set of problems solvable by a human is the same across various languages. While I only have my human experience as evidence to support this claim, I believe this to be true: in solving a problem (e.g., fifty two times two), we naturally distill the problem into symbolic abstractions (numbers, operators), and solve the problem by manipulating these symbols (add the five, carry the one) using a logical process (addition algorithm). This process of mathemetical reasoning is independent of the language in which the problem was posed, and I would like to find if this is the case for LLMs, like GPT-4o, as well.
