AI Adventures: A Programmer’s Journey
IMHO LLMs primarily integrate strings into complex patterns. This process happens on many levels where texts (or rather some internal representation of texts) computed earlier are later fit into higher level patterns. A good example of that is Chain of Thought - which has the higher level pattern of thinking step by step and lower level patterns of the individual steps. This is Linearized subgraph matching described in Faith and Fate: Limits of Transformers on Compositionality, but I would rather call it Multilevel pattern matching (the equivalence here is like that from Kleene’s Theorem). A good explanation of that paper you can find at the answer.ai blog. This might sound like the stochastic parrot criticism of LLMs and some argue that LLMs do pattern matching instead of reasoning. I think reasoning can emerge from pattern matching just like universal computation arises from a short set of rules and empty tape.
Here, although the probability distribution is unexpectedly altered, the resulting answer remains valid. But in GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models authors provide examples where it leads to an answer they consider wrong. Yet the examples they provide don’t seem entirely fair:
