The audio sounds like it is being played underwater. Still, it’s a first step toward developing more expressive devices to help people who are unable to talk. Scientists have trained a computer to analyze a person’s brain activity while listening to music and then reproduce the tune based solely on those neurons. The study, released on Tuesday, created a recognizable, albeit muffled, version of Pink Floyd’s 1979 song “Another Brick in the Wall (Part 1).”
Previously, researchers have discovered how to use brain activity to synthesize music with comparable characteristics to the tune being listened to. Now, “you can actually listen to the brain and restore the music that person heard,” said Gerwin Schalk, a neuroscientist who runs a research center in Shanghai and gathered data for this study. The researchers also discovered a region in the brain’s temporal lobe that responded when volunteers heard the 16th notes of the song’s guitar riff. They argued that this specific area may be implicated in our perception of rhythm.
The findings provide a first step toward developing more expressive devices to assist people who are unable to talk. Over the last few years, scientists have made significant advances in extracting words from the electrical signals produced by patients with muscular paralysis’ brains when they attempt to communicate. However, much of the information given through speech is derived from what linguists refer to as “prosodic” features, such as tone — “the things that make us a lively speaker and not a robot,” Dr. Schalk explained.
Scientists think that by better understanding how the brain metabolizes music, they will be able to develop new “speech prosthetics” for those with neurological illnesses that impede vocal production. The goal is for these gadgets to convey not only what someone is attempting to say, but also some of the musicality, rhythm, and emotion of natural speech. The researchers collected data for the study from the brains of 29 epilepsy patients at Albany Medical Center in New York State between 2009 and 2015.
As part of their epilepsy treatment, the patients received a net of nail-like electrodes implanted in their brains. This provided a rare opportunity for neuroscientists to record their brain activity while listening to music. The researchers chose the Pink Floyd song because it appealed to the elder patients. “If they said, ‘I can’t listen to this garbage,'” the data would have been disastrous, according to Dr. Schalk. Furthermore, the song contains 41 seconds of lyrics and two and a half minutes of melancholy instrumentals, which was valuable for determining how the brain perceives words versus melody.
Robert Knight, a neuroscientist at the University of California, Berkeley, and the team’s leader, requested one of his postdoctoral fellows, Ludovic Bellier, to try to utilize the data set to recreate the song “because he was in a band,” Dr. Knight explained. The lab had previously performed comparable work recreating words.
