EL-PERTURB preprint: random data splits overestimate the performance of models that predict cellular responses to gene silencing
EL-PERTURB preprint: random data splits overestimate the performance of models that predict cellular responses to gene silencing
On 19 October 2026, the authors of the EL-PERTURB preprint compared 20 model variants. They compared a conventional random split with a test in which entire groups of similar cellular responses were assigned exclusively to either training or testing.
A “virtual cell” is a program that attempts to predict how silencing a gene will change the activity of other genes. Such predictions could help researchers choose which gene to test in their next laboratory experiment. The authors used CRISPRi, a method that suppresses a selected gene. Perturb-seq then measures RNA in individual cells to reveal the resulting changes in the activity of other genes.
Conventional evaluation randomly assigns genes to training and test sets. Genes with similar measured cellular responses can appear in both sets, giving the model a familiar example to draw on when predicting the response to a test gene. In EL-PERTURB, the authors first grouped the measured responses, then assigned each entire group to either training or testing. For a subset of “unseen” genes, they also removed data from the other cell cultures used in the study. This allowed them to compare predictions for familiar responses with predictions for groups of cellular responses held out from training.
In K562 cells, using a dataset of genes essential for cell survival, splitting the data by these groups reduced the models’ overall scores by approximately 10%, with several declining by more than 40%. In a genome-wide screen in the same K562 cells, in which genes across the genome were silenced, model predictions for “unseen” genes performed only slightly better than a simple baseline: the average response profile across training examples.
The authors attribute this difference to the datasets themselves. Silencing essential genes more often produces similar responses across cell cultures, whereas a genome-wide screen captures much more varied responses. Data from other cell cultures helped models transfer a familiar type of response; this relationship was weaker for new groups of responses.
The authors of a 2025 study had already compared five models pretrained on large biological datasets and two deep neural networks with simple linear models. A July discussion of “Needles in a Haystack” examined another pitfall: a conventional average error metric can suggest good performance even when a model misses a specific response. EL-PERTURB tests an additional condition: whether performance estimates change when the test set includes a group of responses absent from training.
“Their predictions rely on analogy rather than mechanism,” is how the EL-PERTURB authors express their interpretation.
This test distinguishes two abilities: identifying a gene that produces a familiar response and predicting a response from a group held out from training. For a model used to select targets for the next experiment, this distinction determines what its score actually measures.