PhD students Naruki Yoshikawa and Marta Skreta have developed a framework that converts natural language inputs into a domain-specific language that chemistry robots can understand and follow. (Photo: Matt Hintsa)
With the emergence of self-driving laboratories, a team of University of Toronto computer scientists is focused on improving chemistry robotics processes, enabling smoother interaction between human chemists and their robotic counterparts.
Led by PhD students Marta Skreta and Naruki Yoshikawa, they propose a framework known as CLAIRify, which translates natural language into a domain-specific structured task plan for robots to execute chemistry experiments. Their approach leverages an automated iterative verification technique that uses errors as helpful prompts and considers laboratory resource constraints.
While LLMs like GPT-3 show potential in generating structured plans, the researchers note that there are verification and data-scarcity challenges for domain-specific languages (DSLs) such as those in chemistry and physics. These DSLs have less data available on the internet which makes it difficult for LLMs to generalize effectively.
“The goal of CLAIRify is to make the interaction between a human and a robot easier,” explains Skreta, a third-year PhD student in the Department of Computer Science.
“We wanted to create an interface that goes from natural language — how a chemist might explain an experiment — to something that a robot can understand and then perform an action using that,” she adds.
CLAIRify ensures a task plan is syntactically valid in the target DSL by a repetitive process to detect errors until a valid program is generated, which can then be executed by a robot.
To demonstrate CLAIRify’s functionality, the researchers combined a natural language input instruction — such as mixing vinegar or baking soda with a red cabbage solution — and a description of a chemical programming language, XDL, into a prompt. This prompt was then processed by the structured language generator, GPT-3.
System overview of CLAIRify. (Illustration: Skreta et al.)
“If we ask the LLM to translate a natural language description to XDL, it might give us something that looks like XDL, but we don't know for sure, so we pass it through a verifier. This verifier is a program that checks whether everything in that XDL is allowed — the actions, the tags, the parameters, whether it’s missing brackets,” Skreta explains.
“If it can compile it to correct XDL, then it’s good, then we know that it’s syntactically correct, and then we can pass it to the robot. If it’s not, however, if it catches errors, we provide these errors back to the generator and say, ‘This is what you did before, these are the mistakes that you made, please fix them,’” Skreta adds.
Once the generator output passes through the verifier with no errors, it is syntactically valid structured language. This correct program can then be translated into robot actions after passing through a task and motion planning framework.
“We can use CLAIRify as the interface between a human and a robot and that makes it a lot easier and lowers the barrier of entry for chemists. It’s also significant because we can use natural language model as a planner — the output of the CLAIRify model is a plan that we can pass to the robot for execution,” Skreta notes.
While CLAIRify was tested on robots with simple experiments involving vinegar and baking soda, it is proof of concept that can be extended to more complex chemistry experiment plans, Skreta says.
“The big picture is that we want to be able to use it in robots for experiments that chemists care about,” she adds.
Through this work, Skreta says, the team found they can use a verifier-like style to correct errors that GPT makes. From there, the generator can fix its mistakes, based on the feedback provided by the verifier.
“Combining GPT-3 with other modules is very powerful. So that’s the approach that we took here, where GPT-3 on its own isn’t perfect, but it’s a really useful translation tool.”
Looking ahead, Skreta says future improvements to CLAIRify will be replacing GPT-3 with GPT-4, noting the former was the best model available during the project. The team is also working on experimenting with generating plans for other DSLs.
She explains that while robotics in chemistry already exists, the aim here is moving toward a generalist robot to reduce the need for extensive hard coding on robots, and instead enabling easier communication to carry out experiments.
Skreta says the team is also using CLAIRify in chemistry labs to produce results that are valuable in both robotics and chemistry.
“We can use it for ‘real’ chemistry, where there’s an experiment that chemists really care about, and they want to optimize. So now they can communicate with the robot more easily, instead of spending months figuring out how to code it on the robot. We’re trying to reduce that gap.”
CLAIRify was developed in collaboration with Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, and professors Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. This research is supported by U of T’s Acceleration Consortium, a global community of academia, industry, and government dedicated to accelerating the discovery of new materials and molecules needed for a sustainable future.
U of T’s ongoing research work in AI and self-driving labs will be showcased at the Acceleration Consortium’s upcoming Accelerate Conference in Toronto from August 22 to 25. For more information visit: https://www.accelerate23.ca/