Brain decoder turns a person’s brain activity into words

The researchers trained their semantic decoder on dozens of hours of brain activity data collected in an fMRI scanner from members of the lab. Nolan Zunk / University of Texas at Austin

Brain-computer interfaces have been developed that can translate brain signals into words. Such devices have the potential to aid communication for people who are unable to speak. But with existing interfaces, users need to have invasive brain surgery. A system that could use non-invasive recordings of brain signals could be much more widely adopted.

An NIH-funded research team, led by Dr. Alexander Huth at the University of Texas at Austin, has been developing a system to decode language using functional magnetic resonance imaging (fMRI) signals. The results of their work appeared in Nature Neuroscience on May 1, 2023.

The team first recorded fMRI signals from three people as they listened to 16 hours of spoken stories. Using these data, the researchers created a decoder for each person that matched fMRI signals to the meanings of particular words and phrases.

The scientists then had the same people listen to new stories and looked at how well the decoders reproduced the text of the story. The decoder reproduced some words and phrases accurately. More often, it produced text that didn’t exactly match the original but still captured the gist of what was being said. For example, when a participant heard the phrase “I don’t have my driver’s license yet,” the decoder interpreted their brain activity as “she has not even started to learn to drive yet.” The decoder predictions were much more accurate than would be expected by chance, based on several different measures.

The team recorded fMRI signals from three different brain regions known to be active during language processing. They found that, by most measures, the decoder performed much better than chance even when using the signal from only one of these regions. This suggests that activity in each region separately represents individual words and phrases. Decoding activity from either the parietal-temporal-occipital association region or the prefrontal region could reproduce most of the word sequences that were decoded from the whole brain.

The decoder was not limited to predicting speech that a person heard. Participants were also asked to imagine telling one of five brief stories. The researchers could identify which story was being imagined based on the decoder output. Next, participants watched short, silent films while in the fMRI scanner. The decoders translated their brain activity into general text descriptions of the events in the films.

“For a noninvasive method, this is a real leap forward compared to what’s been done before, which is typically single words or short sentences,” Huth says. “We’re getting the model to decode continuous language for extended periods of time with complicated ideas.”

The researchers also attempted to address privacy concerns raised by this technology. They showed that a decoder trained on data from one person cannot decode data from a different person. So, a decoder could only work for the person on whom it was trained. Such training would need the person’s active cooperation. The team also found that the decoder could be consciously resisted. When participants focused on other tasks while listening to a story—either naming and imagining animals or telling a different story—the decoder could not reproduce the story being heard.

The system currently relies on fMRI, which is not portable, and so cannot be used outside of the lab. But the researchers believe this system could be adapted to use more portable measurements of brain activity. This would allow its use in real-world situations.

—by Brian Doctrow, Ph.D.