Donald Knuth Asked ChatGPT 20 Questions. What Did We Learn?
It seems like everyone’s playing with ChatGPT — including mathematician and long-time programming expert Donald Knuth. Inspired by a conversation with Stephen Wolfram, Knuth conducted “my own little experiment” on April 7 — and recently posted the results online.
Knuth tested what appeared to be ChatGPT-3.5 with a suite of 20 prompts — including some trick questions, like “Who wrote Beethoven’s 10th Symphony?” and “Will the NASDAQ rise on Saturday?”
“Of course I didn’t really want to know any of these answers,” Knuth wrote, adding that instead he’d “wanted to see the form of the answers…”
Knuth’s conclusion? They were “extremely impressive responses, sometimes astonishingly so…” Specifically, Knuth praised “the quality of the wordsmithing. It’s way better than 99% of copy that people actually write.”
But Knuth did also note “surprising lapses… as typical of any large system” — which kicked off a vigorous online discussion. Soon other technologists were testing the same questions on more advanced chatbots — and the whole episode started up a discussion about how performance should ultimately be measured.
