A single cortical implant decoded speech and gestures simultaneously in people with paralysis: combined-data training succeeded where separately trained decoders had failed
A single cortical implant decoded speech and gestures simultaneously in people with paralysis: combined-data training succeeded where separately trained decoders had failed
Edward Chang's neurosurgery team at the University of California, San Francisco implanted a single 253-electrode array on the cortical surface in three people with severe paralysis (two following brainstem stroke, one with amyotrophic lateral sclerosis, a disease that progressively disables muscle control). The device simultaneously recorded attempted speech and attempted gestures, and the decoded signals drove a personalized virtual avatar of each participant in real time. The results were published on September 14 in Nature Neuroscience.
Previously, brain-computer interfaces decoded speech or body movements only in isolation: in earlier work by other groups, a simultaneous attempt to speak degraded cursor-movement decoding, and the reverse held as well. The reason lies in the cortex itself: the regions controlling hand movement and speech partially overlap, and when both execute at once, the neural activity patterns in those regions change. A model trained only on isolated attempts could not account for this shift.
Chang's team tested this directly: decoders trained separately on speech and gestures, when faced with simultaneous attempts, more often classified the signal as rest, with missed-attempt rates reaching 35% in one participant. The composition of the training data proved decisive: models trained on both isolated and concurrent examples maintained high accuracy in both modes, preserving the ability to distinguish speech from gesture by leaning more heavily on electrodes specific to a single modality. Each decoder was additionally taught to treat the other modality's activity as rest: the speech model learned to remain silent during gestures, and the gesture model during speech, which nearly eliminated false activations when both ran in parallel. On novel phrase-gesture combinations, one participant's model performed as well as on previously seen pairings: it learned a general rule that transferred to unseen combinations.
The decoded signals drove a full-body avatar: gestures animated the body, and speech appeared as text. In a simulated conversation, one participant achieved 100% accuracy for both speech and gestures, while the other reached 75–85%; the third participant withdrew from the trial, so the demonstration relies on data from the remaining two. This work continues a line from the same laboratory: in 2023, the same type of implant controlled an avatar through speech and facial movements for one patient, but those facial movements were part of speech articulation itself. The new result adds an independent body-movement channel and tests it concurrently with speech.
Conversation is far more than the words spoken. It is a multilayered, dynamic process that engages the entire motor cortexChang says. Daniel Rubin, a neurologist at Massachusetts General Hospital who was not involved in the study, called the work "an important step toward developing truly natural systems for restoring function in people with paralysis."
The vocabulary remains limited: 10 phrases and 10 gestures. Frozen decoders without retraining maintained above-chance accuracy across days and months. The principle extends beyond the current vocabulary: it is a step toward neural interfaces that simultaneously read multiple coordinated channels of human intention.