Many, perhaps most, of my friends are giddy about ChatGPT. My Inbox is full of emails from Tyler Cowen touting its glories. More concretely, Alex Tabarrok recently assured me that existing AIs could already beat the performance of most of our economics students in our beloved subject. Though I was on the road almost all of last month, I vowed to put these claims to the test upon my return. […]
You could object: “Who cares if ChatGPT gets a D on your midterm?” I care. At minimum, this shows, contra Alex Tabarrok, that the AI is not yet as good at economics as most of our students. For me personally, this also reinforces my suspicion that AI enthusiasts cherry-pick the best AI output.
I was surprised that this was Caplan’s reaction to a computer program receiving a D on one of his exams. For a long time, it was considered extremely hard to get computers to comprehend even basic sentences. Ten years ago, the state of the art in language modeling was pitiful. You usually couldn’t get an AI to spit out a coherent answer to a question on an economics exam, much less a correct answer.
Given that we went from AI that could barely understand English at all, to AI that can get a D on one of Caplan’s exams within ten years, one would think that this development represented amazing progress, even if the tech is still immature.