Extending GPT-3’s context window infinitely by storing context in GPT-3 itself or in secondary layers
OpenAI’s GPT-3 is one the best (and most underrated) things that happened to mankind in 2020. It proved that it is possible for an AI to surpass the zero-shot and few-shot learning abilities of most humans on a huge range of general tasks, both logical and creative, and pass the duck test for human intelligence (aka Turing test) with flying colors.
The implications are profound, and provi de a fresh look both about what it means to be intelligent, and the notion of agency. Indeed, what are we if not a machine predicting and merely obeying our next most likely action/thought based on the context window of our life and everything we’ve been exposed to, creating our own model of the world and acting upon it?
GPT-3 has also built its own model of the world, basically of how things (symbols, words and abstract concepts) relate to each other, in a comprehensive and accurate way no symbolic AI researchers/enthusiasts (myself humbly included) could have hoped for, or remotely anticipated.
We have this amazing model, which proves human knowledge can be encoded in a huge neural network, but today can only use it provided the context and expected response fit within 2048 tokens (expressed via language, only around 1,000 words). And most importantly, we cannot easily and efficiently teach it any new information the way we can teach a human intelligence (which is really the Holy Grail for AGI). Fine tuning and retraining the whole model open useful possibilities, but are far from being the kind of one-shot feedback loops humans excel with.
