Computational irreducibility, inference time compute, and why we need to learn programs
Dynamic inference time compute is a hard requirement for AGI. Reasoning tokens are here to stay and enable fundamentally different forms of thinking.
When do we need dynamic inference time compute? Computational irreducibility, cellular automata and algorithmic complexity give us a foundation on which to explore this question.
What does this mean for solving general intelligence? I believe this encourages us to think about learning programs, not just functions. I’ll explain what that means, why it's important, and explore this a little within the context of ARC.
You can view much of machine learning as a problem of function approximation. In the case of an LLM, we are trying to predict the next token (or their probability distribution) from some sequence of preceding tokens.
We do not know what is, so we try to learn it. The assumption is that is a typical mathematical function, and we learn through a process of optimization through gradient descent.
