Playing the London System against OpenAI's ChatGPT
I was inspired by a post on HN to play around with OpenAI’s ChatGPT. Over the past two weeks, I’ve seen many hilarious Reddit posts on ChatGPT interactions but this one really takes the cake.
Given the recent Stockfish FOSS win, I thought I’d see if I could play a game of chess against ChatGPT. Since ChatGPT was trained to be as conversational as possible, it would be interesting if I could get ChatGPT to make moves and explain them after practice.
Hmm, ok… if it wouldn’t immediately accept my challenge, then maybe I should find another way to “play” against it.
The plan was as such:
- Setup an analysis board on chess.com
- Pick an opening / make a move
- Copy the PGN and ask ChatGPT for the next move
- Repeat until the game is over.
Now, the question is whether ChatGPT could come to a decision on what move to play. For a chess engine like Stockfish, you would be certain that the moves suggested have been analyzed to maximize win probability. However, since ChatGPT is trained to process and relay information conversationally, we know that it does not “understand” chess the same way Stockfish does. As such, I wanted to find out if ChatGPT could at least play and explain why it made certain moves.
First, I wanted to test if ChatGPT was able to pick the best (engine) move.
No surprises there. Maybe it knows the most popular human move?
Okay… maybe something else.
Bingo! ChatGPT was able to pick a move. Not only that, but it gave an explanation as to why it was able to play the move. Awesome! Let’s begin.
I decided we should play the London system. It was an opening that’s decently explored, straightforward and novice friendly. I wanted to see how ChatGPT could play in a simple game without gambits and weird lines.
The game went on fairly standard. I was impressed by how the early game went, as the moves made by ChatGPT were consistent with the moves players usually make in a London system game. Up until castling, things were seemingly normal. For a conversational bot, ChatGPT was exceeding my expectations.
The first mistake it made was when it hung its knight on h5.
ChatGPT traded its dark-squared bishop for Ne5, and I took Nh5 with my Queen. Okay… ChatGPT isn’t Stockfish right? The next move was to push the g6 pawn to prevent mate in one since my light-squared bishop covered the h6 square.
Naturally, I pushed my Queen ahead for victory. This was when ChatGPT started to break down.
Clearly, ChatGPT forgot about the fact that they had not only castled but also traded away their dark-squared bishop on e5. Since we reached the end of our game, I wanted to review the match to find out why it had hung its knight.
It was then that I realized ChatGPT is no Stockfish. Its explanation made no sense considering the fact that the g3 bishop was in no way defending the h5 knight- they’re on opposing sides!
Maybe the next step is to make ChatGPT “play” against Noctie.
Raph
The final PGN:
1. d4 d5 2. Bf4 Nf6 3. Nf3 e6 4. e3 c6 5. Bd3 Bd6 6. Bg3 O-O 7. Nbd2 Nh5 8. Ne5
Bxe5 9. Bxe5 Nd7 10. Qxh5 g6 11. Qh6 *