I played chess against ChatGPT-4 and lost!

Last December, I played a few chess games against ChatGPT. These always ended the same way: ChatGPT would play an accurate opening, until it forgot where its pieces were and started playing illegal moves, with full confidence of course. The truth is, GPT-3 does not know how to play chess. Playing a game against it reveals its true nature as a stochastic parrot that merely produces a believable-sounding answer from its training set. I wrote in December:

ChatGPT cannot play chess at a human level (yet). It is clearly aware of the game and able to accurately play mainline openings. But the moment the game moves out of theory, ChatGPT can no longer keep up. This shows that the language model doesn’t (yet) have any understanding of chess fundamentals, but merely repeats moves and phrases that commonly occur in a documented chess game.

ChatGPT’s confidence, combined with its “bending” of the rules of Chess, became something of a meme on the Chess side of the internet, with Reddit posts that hit the front page and YouTube videos receiving millions of views. We laughed at it. Mocked it. Used its deficiencies to justify the superiority of humans over machines.

Leave a Comment