EL-PERTURB Tests Virtual Cells on Unfamiliar Responses
EL-PERTURB finds that random data splits overestimate virtual-cell model performance when similar cellular responses appear in both training and test sets.
EL-PERTURB shows how data splits affect virtual-cell model scores
In a genome-wide study of the K562 cell line, models predicting responses to silencing genes excluded from training performed only slightly better than a simple baseline: the average response profile in the training data.
A “virtual cell” here means a program that predicts how silencing one gene will change the activity of others. Such predictions could help researchers choose their next laboratory experiment. Testing them requires measured cellular responses: CRISPRi suppresses a selected gene, and Perturb-seq measures RNA in individual cells to detect changes in other genes’ activity.
The EL-PERTURB preprint compared 20 model variants. A conventional random split assigns genes to training and test sets. Silencing different genes can produce similar responses, so a model may encounter a familiar response type during testing. The authors first grouped measured responses by similarity, then assigned each whole group to training or testing. For some test genes, they also excluded data from other cell cultures.
On a dataset of genes essential for K562 cell survival, splitting by response group reduced overall scores by about 10%, with declines exceeding 40% for several models. The genome-wide result described above came from a different dataset; these results describe different comparisons.
The authors attribute the difference to the datasets: silencing essential genes more often produces similar responses across cell cultures, while a genome-wide study captures more varied responses. Data from other cultures helped models transfer predictions for a familiar response type; this relationship was weaker for new response groups.
A model score therefore needs to be read alongside the data-splitting method. Identifying a gene that produces a familiar response and predicting a response from a group excluded from training require separate tests. When choosing the next experiment, researchers need to know which ability a score measures. These results concern the models and datasets tested: they are insufficient to conclude that all virtual-cell models are useless for selecting experiments or rely only on analogy.
The link below leads to the public Eternal Search page for an organization named Experiment. The supplied material does not establish a connection between it and EL-PERTURB.
Open the related Eternal Search page
Sources
[1] biorxiv.org
[2] pmc.ncbi.nlm.nih.gov
[3] t.me
This item was selected because it changes how researchers should interpret model scores before choosing a laboratory experiment. The linked Eternal Search page is included as requested, but the available material confirms only that it is a public page for an organization named Experiment; it provides no factual basis for connecting that organization to EL-PERTURB.