Pay attention the next time a robot server glides effortlessly between tables to deliver your meal. You may be witnessing the result of a breakthrough by a research team that includes University of Toronto collaborators.
Behind seamless AI behaviour is a messy reality: teaching machines to learn is notoriously difficult. Researchers have long faced a choice. They could train systems quickly, risking unstable behaviour, or train them carefully at the cost of time and complexity.
Now, U of T researchers Claas Völcker and Igor Gilitschenski, part of a nine-person team, are showing that this trade-off may not be necessary. Their algorithm — Relative Entropy Pathwise Policy Optimization (REPPO) — enables faster, more reliable AI system training while reducing computational demands.
The work, recently presented at the International Conference on Learning Representations (ICLR), could make it easier to build the intelligent systems reshaping everyday life — from robots to autonomous vehicles.
“At first, it can sound like we’re just optimizing something that already works,” says Gilitschenski, an assistant professor of computer science who leads the Toronto Intelligent Systems Lab. “But those improvements in reliability, cost and speed are what take robots from working in the lab to working in the real world.”
"Environments used to evaluate REPPO, including locomotion, humanoid mobility, robotic manipulation and Atari game-playing tasks." Images courtesy of Relative Entropy Pathwise Policy Optimization authors (Voelcker et al., 2025)
Why teaching AI is so difficult
Reinforcement learning (RL) — the area Völcker and Gilitschenski work in — trains AI through trial and error. Systems experiment, receive rewards or penalties and improve over time.
Völcker, who recently earned his PhD in computer science from U of T, led REPPO’s development. He says RL systems typically struggle in two ways, depending on the training environment.
In fast simulation environments — where thousands of scenarios run in parallel — researchers rely on policy gradient methods. While computationally efficient, this simple algorithm class can be unstable, with performance stalling or collapsing during long training runs. It also requires extensive fine-tuning of internal settings called hyperparameters for each new task.
In slower, more complex environments, researchers use a second model, called a critic or value function. It evaluates actions and estimates alternatives, but is difficult to train reliably, particularly with limited data.
“Working in these slow simulation environments, I got frustrated because experiments took so long,” says Völcker. “At the same time, high-speed simulation using graphics processing units (GPUs) emerged. These simulators are much faster but still reasonably accurate. I started wondering what would happen if we applied methods designed for slow learning to fast simulation environments — and it worked surprisingly well.”
A different approach
That insight led to REPPO, which combines speed and stability. It reduces computational cost, requiring less memory and fewer training examples.
Critically, REPPO remains fully “on-policy,” learning directly from current experience rather than stored data like replay buffers. It also improves value function training, making evaluations more accurate. As a result, by constraining updates at each step, REPPO avoids the large shifts that often make RL difficult to control.
“REPPO requires much less time to train than the industry-standard agent, Proximal Policy Optimization (PPO),” says Gilitschenski. “This is what makes it exciting. We can achieve the same outcome with significantly less data.”
Because PPO is widely used, many systems can be improved simply by replacing it with REPPO. Völcker calls it a plug-and-replace method.
Where REPPO makes a difference
One of the most time-consuming challenges in AI development is tuning — adjusting hyperparameters to make training work.
In testing, REPPO performed consistently across 31 tasks using the same settings, something rarely achieved with standard methods. That consistency allows roboticists to start from a reliable baseline, rather than retuning systems for each new application.
REPPO has shown strong results across humanoid locomotion, robotic manipulation, video game playing and multi-agent systems.
“REPPO is particularly strong in continuous control,” says Gilitschenski. “That’s when an AI is constantly adjusting things like direction, speed or movement.”
Rethinking how AI is trained
REPPO also challenges assumptions about how RL systems are built.
Despite limitations in efficiency and reliability, PPO has long been the default across robotics, gaming and other applications because it’s relatively simple and effective. Rather than incrementally improving PPO, REPPO suggests some core design choices should be reconsidered.
“People have been trying to tweak PPO without questioning the fundamental assumptions behind that setup,” says Völcker. “The RL community, which focuses on value function training, is now much better at training a model that accurately predicts how useful an action will be compared to a few years ago, and we didn’t bother to check whether this would also improve things in environments where simulation is easy.”
In early testing, REPPO matched or exceeded PPO’s performance more reliably. Roughly 80 per cent of training runs reached and maintained high performance compared to about 40 per cent for PPO. It also used significantly less memory — between 100 and 1,000 times less in some settings.
That reduction in resource demand could lower the barrier to experimentation, enabling more researchers to test and iterate on new systems.
REPPO in the real world
If training AI becomes speedier and more reliable, the impact extends beyond the lab.
Systems that once took long to train could be developed more rapidly and deployed with greater confidence. In robotics, more consistent training could also reduce the risk of unexpected behaviour.
For Gilitschenski and Völcker, that’s where the real value lies: making AI systems more practical to build and use.
“If we’ve done our job as roboticists, the end user shouldn’t see our work,” says Gilitschenski. “What they will get is autonomous robots that are seamlessly integrated into everyday life.”
As Völcker sums it up, “It’s not something the end user sees directly, but it’s very important to the field as a whole for it to function better … The whole [REPPO] solution has this ‘obviousness’ to it that gives you a strong feeling that you built something that makes sense.”