Third-year PhD student Sejal Bhalla developed speech analysis software, alongside a team of computer scientists and clinicians, that uses AI to detect changes in vocal characteristics among patients diagnosed with chronic obstructive pulmonary disease. (Photo: Matt Hintsa)
Researchers at the University of Toronto have developed a new way to monitor patients living with a common lung condition by leveraging artificial intelligence and smartwatch technology.
Chronic obstructive pulmonary disease (COPD) causes restricted airflow and breathing problems. It impacts millions of people and is the third leading cause of death worldwide.
Looking to explore potential solutions to manage this debilitating disease, third-year computer science PhD student Sejal Bhalla led a recent study alongside a team of computer scientists and clinicians. Together, they developed a system called PulmoListener. This speech analysis software can assess the lung condition of patients diagnosed with COPD using audio collected from a smartwatch, which could ease the burden of existing monitoring techniques.
“PulmoListener continuously captures snippets of a person’s voice through a regular smartwatch available on the market,” explains Bhalla. “It detects small changes in the voice to help spot potential worsening of lung condition very early on.”
This research was recently published in the journal Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT). Bhalla’s co-authors from the Department of Computer Science include PhD student Salaar Liaqat, and co-supervisors Assistant Professor Alex Mariakakis and Professor Eyal de Lara. They are joined by two faculty members from U of T’s Department of Medicine in the Temerty Faculty of Medicine: Associate Professor Robert Wu (University Health Network) and Professor Andrea Gershon (Sunnybrook Health Sciences Centre).
Speech is a known indicator of respiratory health. When air is pushed through the vocal folds to produce speech, it lends useful information about an individual’s respiratory pathways, Bhalla notes.
“When the symptoms of a COPD patient worsen, their vocal characteristics subtly differ from how they usually sound. Because these differences are subtle, they’re not decipherable to the human ear. But it turns out that they can be picked up by artificial intelligence,” she adds.
The gold standard for assessing lung function is spirometry, which entails forcefully blowing air into specialized equipment. However, one of the biggest limitations of this test is that it is highly dependent on how well a patient can perform this manoeuvre, says Bhalla. If a patient is not putting their full force into exhaling the air out of their lungs, the results of this spirometry test would not be as reliable, she explains.
PulmoListener shifts the responsibility of lung assessment from the patient to a smartwatch and custom software that resides on it.
“The only effort that a user ever puts in is to wear the watch every day,” says Bhalla. “It not only reduces the user burden, but also enables continuous monitoring of the lung condition, which is challenging to achieve using specialized equipment and doing strenuous tasks like forceful exhalation when your lungs are already distressed.”
PulmoListener constitutes an end-to-end speech processing pipeline that isolates useful speech segments from real-world audio, converts them into meaningful representations, and feeds them into a deep learning model to predict COPD symptom severity. (Figure: Sejal Bhalla)
At the end of each day, PulmoListener identifies speech belonging to the patient wearing the smartwatch and discards the rest of the audio. The team considered privacy concerns in designing PulmoListener, Bhalla notes. When speech is recorded from patients, it is converted into a representation that does not hold a substantial amount of information about what they say, but rather how they sound. These “concise meaningful representations” can be used to infer the daily severity level of COPD symptoms through a deep learning algorithm, Bhalla explains.
“At its core, PulmoListener is a combination of multiple AI algorithms that are designed to tackle different speech analysis tasks,” she adds.
The researchers conducted a six-month study of PulmoListener on eight patients already diagnosed with COPD by pairing the software with a Samsung Galaxy smartwatch. Among the study’s key findings, the researchers noted a change in COPD symptoms brings a change in the vocal characteristics of a patient. Additionally, these changes can be modelled using neural networks, enabling PulmoListener to correctly identify worsening lung condition 80 percent of the time, Bhalla says.
“I think one of the most interesting findings of this work has been that speech begins to change even before a symptom shows up. So, it acts as a warning for deteriorating lung condition. We found that PulmoListener can detect and anticipate these changes in the lung condition up to four days in advance,” she says.
According to the researchers, these findings demonstrate the feasibility of leveraging natural speech for monitoring COPD in real-world settings, offering a promising solution for disease management.
With further research and development, this method could potentially offer timely interventions that could reduce hospitalizations that result from the exacerbation of symptoms, assist patients in tracking their daily lung condition and empower them to make active choices about their daily routines, including how to avoid symptom triggers.
“Off-the-shelf smart devices offer seamless integration into our daily lives while accurately assessing health and well-being,” Bhalla notes. “Combined with the potential of AI, these devices can transform into early warning systems, which could aid clinicians in delivering timely assistance to these patients.” She adds that with more work, PulmoListener could open the door to improved COPD management, especially among frail patients who might have difficulty accessing a clinic or hospital regularly.
While the researchers have successfully employed this technology among COPD patients during their study, Bhalla believes this work “opens up very exciting possibilities for the broader application of speech analysis in health care” with the potential to provide valuable insights into respiratory infections like pneumonia and other chronic diseases like asthma or heart failure.