Towards Acting AI - by Sergey Alexashenko - How the Hell
There was a magical moment in the history of Large Language Models - the release of GPT-3. The technology went from babbling complete nonsense to being able to talk fairly reasonably and solve a wide variety of problems. It was magical. Especially cool was the fact that all it took (not to minimize the herculean task, talking strictly about fundamental technology here) was taking an existing model and SUPERSIZING it 1.
That got a lot of people (including yours truly) very excited. People started thinking, well, if we went from nonsense to sense by SUPERSIZING… What will happen if we… MEGASIZED the SUPERSIZED model? Now GPT-4 is out, and we have an answer to that question. GPT-4 has basically achieved human-level intelligence at a wide variety of tasks from medicine to law to math (and more!).
There are some people who think that if we ULTRASIZE the model, we will somehow get to superhuman intelligence. I think that that’s false and that we are basically at or near the limits of LLM scaling. Consider what a transformer model “wants” when it’s training. It wants to predict the next word, to say what humans would have said. Humans say… all kinds of things, a lot of them very much below the fabled “human-level intelligence”.
