Talmo Pereira: NeuroAI should close the loop between behavior, brain, model, and experiment

Talmo Pereira showed how muscle activity can be used to test models of animal behavior

On 26 August, Salk Institute researcher Talmo Pereira spoke at a Foresight Institute seminar about models that connect neural control, the body, and the physics of the environment. The models are evaluated against video recordings of movement and measurements of muscle activity.

A camera records where a mouse places its paw and how it turns its head. However, different movement patterns can produce the same trajectory. In the lecture, Pereira shows a dead fish in flowing water. The current bends the fish's body and changes its direction. The movement results from the interaction between the body and its environment.

A useful model must therefore represent both the body and the physics of the environment. In the MIMIC-MJX preprint, which lists Pereira among its authors, a neural network sends commands to a biomechanical replica of an animal in a physics simulator. The network is trained on recordings of a real animal to reproduce its movements.

In the experiment shown, a mouse moves a joystick while a virtual model reproduces the mouse's movements. A controller trained only to complete the task can also move the joystick, but it may use a different strategy. For a model first trained on the movements of a living mouse, the predicted muscle activity is compared with electromyography recordings of electrical muscle activity from the real experiment. This comparison tests which commands produced the observed trajectory.

Pereira calls this research program virtual neuroethology: simulations of animal behavior that are tested experimentally. Researchers change the conditions in the model, use the result to predict the outcome of the next experiment, and refine the model with new measurements.