Science University Research Symposium (SURS)
Evaluating LLM Reasoning Through Chess Puzzles
Publication Date
11-23-2026
College
College of Sciences & Mathematics
Department
Math and Computer Science, Department of
SURS Faculty Advisor
Dr. Christina Davis
Presentation Type
Oral Presentation
Abstract
Large language models (LLMs) have shown strong reasoning and problem-solving abilities, but it is still unclear how well these abilities hold up in tasks that require multiple steps of reasoning. Chess puzzles offer a way to test this, since models need to understand the board, consider different moves, and find the correct solution.
This project builds a chess evaluation framework using LangGraph and Python to test how different LLMs perform on chess puzzles. Models can use tools to access information, such as board positions and legal moves, as they work through each puzzle. This project aims to use chess puzzles to compare models' reasoning abilities, how they use tools, and where they struggle.
Recommended Citation
Patel, Mayur, "Evaluating LLM Reasoning Through Chess Puzzles" (2026). Science University Research Symposium (SURS). 346.
https://repository.belmont.edu/surs/346
