Computing inside an AI | Will Whitney
The first direction is about technical capabilities. How big a model can we train? How well can it answer SAT questions? How efficiently can we serve it?
The second direction is about interaction design. How do we communicate with a model? How can we use it for useful work? What metaphor do we use to reason about it?
The first direction is widely followed and hugely invested in, and for good reason: progress on technical capabilities underlies every possible application. But the second is just as crucial to the field, and it has tremendous lingering unknowns. We are now only a couple of years into the large model age. What are the odds we’ve already figured out the best ways to use them?
I propose a new mode of interaction, where models play the role of computer (e.g. phone) applications: providing a graphical interface, interpreting user inputs, and updating their state. In this mode, instead of being an “agent” that uses a computer on behalf of the human, AI can provide a richer and more powerful computing environment for us to use.
At the core of an interaction is a metaphor which guides a user’s expectations about a system. The early days of computing took metaphors like “desktops” and “typewriters” and “spreadsheets” and “letters” and turned them into digital equivalents, which let a user reason about their behavior. You can leave something on your desktop and come back to it; you need an address to send a letter. As we developed a cultural knowledge of these devices, the need for these particular metaphors disappeared, and with them the skeumorphic interface designs that reinforced them. Like a trash can or a pencil, a computer is now a metaphor for itself.
