For the first time, a single brain implant can now read a paralyzed person’s attempts to speak and gesture at the same time, turning them into text and the gestures of a digital avatar. According to the research paper published in Nature, the device uses artificial intelligence models to read the brain and transform its impulses into speech and body language.
Current brain-computer interfaces (BCI) can decode only one thing at a time. You can either reliably translate speech or you can reliably translate gestures, but not both. They stumble because the brain areas involved in talking about something while moving your hands as you speak partly overlap.
Scientists from the University of California, San Francisco (UCSF), have now showed that their new system handles speech and gestures together thanks to artificial intelligence. Right now, it can handle only small vocabularies and two participants for simultaneous decoding. It’s also imperfectly accurate. But it’s a big step forward, because all humans communicate not only with language but also with how our bodies move. Hand gestures, like facial expressions, are an integral part of the experience of interacting with other people.
But this is only the first step toward a brighter future. “Our longer-term goal is to move toward what we think of as a ‘whole-body BCI’: a single brain interface that can restore multiple forms of communication and movement,” Samantha Brosler, one of the lead authors of the research paper, tells Fast Company. “In principle, the decoded speech and movement signals could be used to control a virtual avatar, other assistive technologies, or a physical robotic surrogate.”
Brosler, a researcher in the University of California, Berkeley–UCSF graduate program in bioengineering, tells me that, ultimately, they want to give people with paralysis more flexibility in how they communicate and interact with the world, “rather than limiting them to controlling one function at a time.”
Since the 2000s, scientists have been developing BCIs capable of triggering certain robotic arm movements, but Brosler says they want to move past those predefined gestures and develop systems that continuously control movements across the whole body in a natural way.
It will mean making the decoding of brain signals faster and more accurate, and studying whether real-time sensory feedback can make the controlling of avatars or robots feel more natural and intuitive. In her view, the study is an early base on which the flexible, multifunctional control that might make this possible can eventually be built.
Two signals, one cortex
When paralysis interrupts the path between brain and body, the brain still issues commands. This is what allows scientists to build BCI implants capable of capturing the electrical signals that represent those commands. The implant is a high-density electrocorticography array designed to catch those commands. Imagine a small flexible sheet studded with a regular grid of tiny metal dots, laid directly on the surface of the brain over the areas that control movement and speech. Hand and arm attempts show up in one region of the grid, head and eye movements in another, and speech and mouth movements in the lower part.
But speech and gestures areas overlap in the brain, which makes things confusing for computers. The researchers found that some electrodes were active for speech only, some for gestures only, and some for both. That’s where artificial engine training came in, to try to make sense of the confusion.
The team tested models trained only on speech or movement, then models trained simultaneously on the two. They found out that models trained on both held up across both contexts and outperformed models trained on one type of data. Software could tell many of the 43 to 45 movements tested apart at above-chance accuracy, with left-hand and leg movements as the exceptions.
Brosler says one encouraging finding is that the models decoded speech-and-gesture combinations they hadn’t encountered during training. In their tests, they made participants attempt 10 phrases, like “hello” and “nice to meet you,” and 10 gestures, like waving a hand and clapping, at the same time. They found that one participant’s accuracy on these pairings was a bit higher than decoding gestures or speech alone. That the system could flag and decode these speech-hand gesture combinations on its own, with no previous specific training, is good news. Collecting training examples of every combination, Brosler tells me, would quickly become impractical.
“An important next step will be determining how much training data is needed to support increasingly complex movements alongside speech, without having to explicitly train on every combination of actions,” she says.
Once they got the reading and interpreting signals phase right, they used the results to move an avatar built in Unreal Engine, a 3D creation tool, while decoded speech appeared as text on the screen. It worked great, even though they didn’t get a 100% translation from thought to avatar with all participants.
The user experience
So no, it’s not all completely accurate and perfect, but it’s a key step forward.
There are still problems that need fixing. The study demonstrated simultaneous decoding during short conversations rather than extended ones. “Minimizing fatigue was important to us,” Brosler says, so each participant picked strategies they could perform consistently. One imagined the gestures, and another silently attempted speech. The team asked one participant whether doing both at once was more tiring than doing either alone, and he said it wasn’t. “That’s encouraging as we work toward supporting longer conversations,” she adds.
Because the system decodes discrete phrases and gestures, Brosler tells me, there is some delay between an attempt and the avatar’s response. “Ideally, we’d like the system to respond quickly enough that speech and gestures feel naturally coordinated during conversation,” she says. The exact latency needed is something the team wants to explore with participants. The lab is now working toward continuous decoding of individual joint movements rather than predefined gestures, to reduce latency and give participants more precise control.
From the metaverse to our universe
Beyond helping people with paralysis, routing this decoding into a mixed-reality avatar or a robot that projects someone’s presence across distances is the natural next step. In principle, Brosler explains, the same decoded speech and movement commands could control an avatar in a mixed-reality environment or even a physical robotic system. “The decoder itself doesn’t necessarily need to change depending on how its outputs are used,” she says.
Science fiction has long imagined brains inside entirely artificial bodies, and that’s where things get really interesting. Brosler sees this work as an early step in that direction, though “we’re still a long way from seamlessly controlling an artificial body,” which is their ultimate destination.
Getting there, she says, means continuous control of movements across the body, better decoding speed and accuracy, and exploring how real-time sensory feedback can make that control more natural and intuitive.
But at least we know that we are on the right track to solve lots of problems for lots of people. And open the path to a future full of extraordinary possibilities for everyone else.