For more than a decade, advances in artificial intelligence have made computers capable of consistently defeating humans in chess. But it's hard for us to learn from AI that doesn’t complete tasks the way a human would.

A new neural network chess engine trades raw power for a more human-like playing style than existing AIs, allowing it to be used as an effective learning tool and teaching aid.

With Maia Chess, U of T researchers and their collaborators open the door to better human-AI interaction in chess and other domains. Maia can accurately predict the way humans of different skill levels play chess and can even specify the mistakes a player should work on to improve their game.

Ashton Anderson, assistant professor in the Department of Computer Science, and PhD student Reid McIlroy-Young collaborated on the project with Jon Kleinberg, professor at Cornell University, and Siddhartha Sen, principal researcher at Microsoft Research.

The new chess engine emerged from “Aligning Superhuman AI With Human Behavior: Chess as a Model System,” presented last year at the Association for Computing Machinery SIGKDD Conference on Knowledge Discovery and Data Mining.

Provided with a problem to solve, self-trained AIs can take a very different route to a solution than a human might. On top of that, a human can also have a hard time learning how the AI completed its task. To bridge the gap in understanding, the researchers pursued the goal of modeling the individual steps humans take to solve a task, rather than focusing on overall human performance.

With Maia, the researchers posed the question, “Instead of aiming AI at the task of playing chess well, what if we aimed the AI at the human playing chess?”

“If we algorithmically captured human style, human ability, and crucially, human errors, maybe we would have a chess AI that was much easier to learn from and play with,” Anderson explains, adding that this approach could be expanded to other domains of AI research.

The researchers trained nine versions of Maia, corresponding to nine different chess skill levels. At each level, the deep learning framework was trained on 12 million online human games.

By training on games played by humans instead of training itself to win every time, Maia can more closely match human play, move by move.

Other attempts to develop chess engines that match human play have been somewhat effective, but Maia’s performance sets the bar higher.

Versions of two popular chess engines, Stockfish and Leela Chess Zero, match human moves with less accuracy and don’t accurately capture human play at specific skill levels. Maia is built on the open-source AlphaZero/Leela Chess framework and is trained on real human games, rather than games played against itself. It achieves higher accuracy than other engines, correctly predicting human moves more than 50 per cent of the time.

Beyond predicting well-informed moves, Maia is also adept at predicting human mistakes, even egregious ones. This can be especially helpful for players looking to improve, as the engine can distinguish between random mistakes and predictable ones.

“Current chess AIs don’t have any conception of what mistakes people typically make at a particular ability level. They will tell you all the mistakes you made — all the situations in which you failed to play with machine-like precision — but they can’t separate out what you should work on,” says Anderson. “Maia can identify the repeated mistakes you make that are typical of your level, and that you could work on to improve. No other chess AI has that ability.”

The researchers are currently developing a personalized version of Maia that can play like a particular person.

“This will make our training tools even more powerful: you could have your own personalized AI that plays like you do, and it could point out the mistakes you make that it predicts you will make—in other words, mistakes you make so often that it correctly guesses you will do it again,” explains Anderson.

Looking to the future, the team looks to conduct a “chess Turing test” to see if human players can tell the difference between a human opponent and Maia.

Ultimately, the researchers hope that Maia demonstrates the value in considering the human element when designing AI systems.

“We want to show that AI systems can be easier to work with and learn from if they are built with human interaction, collaboration, and improvement in mind,” Anderson says.

Chess players can face off against three versions of Maia on the free online chess server Lichess: Maia 1100, Maia 1500, and Maia 1900.

Ashton Anderson was supported in part by an NSERC grant, a Microsoft Research Award, and a CFI grant. Jon Kleinberg was supported in part by a Simons Investigator Award, a Vannevar Bush Faculty Fellowship, a MURI grant, and a MacArthur Foundation grant.

Header image: Rafael Rex Felisilda / Unsplash