Princeton team reconstructed images shown to a person from functional MRI data in about 15 seconds

Princeton team reconstructed images shown to a person from functional MRI data in about 15 seconds

On July 23, a Princeton team posted a preprint describing a system that reconstructs the general content of an image during an MRI session. In an experiment with one participant, it produced a result about 15 seconds after the image was shown.

When a person looks at a lighthouse, a skier, or a dog, the visual cortex produces a corresponding pattern of activity. Functional MRI detects this response indirectly. The scanner measures changes in blood oxygenation that follow neuronal activity. The signal takes several seconds to build, so the system must wait for it before estimating what the image contained.

In the team's preprint, the program waits about 7.9 seconds, aligns the scans, and estimates the response to a single image. MindEye2 then matches this response to image features, and a generative model uses those features to render the scene. In the fast configuration, identifying the original image among a set of candidates took about ten seconds, reconstruction took about fourteen seconds, and performing both operations together took about fifteen seconds.

Before the experiment, the participant spent about an hour viewing images in a 3-tesla scanner. This allowed the decoder to learn the relationship between that person's visual activity and the images. During the second session, the system worked mainly with new natural scenes. It preserved the subject and major details, including a person in the snow, animals on grass, and the colors and positions of objects. The result was a new image with the same composition and overall scene content.

The previous MindEye2 processed recordings after the session and could use data acquired at later timepoints. In the new work, scan alignment, response estimation, and generation occur as scanning proceeds. The program uses only data that have already been acquired. The researcher can see the result while the participant is still in the scanner.

The authors propose using this feedback to guide the experiment. A researcher could show an image, obtain a reconstruction of its representation in the brain, and select the next stimulus based on that result. A fifteen-second delay changes reconstruction from a report produced after the experiment into a signal that can guide its next step.