In a preliminary study, the AI model State outperformed a simple method at predicting differences in cellular responses to protein signals in two of three cell types
In a preliminary study, the AI model State outperformed a simple method at predicting differences in cellular responses to protein signals in two of three cell types
On November 6, a group at Wright State University published an analysis of technical artifacts in cell experiments. The authors reassessed published State predictions by separating the response shared across treatments from the differences between them.
State takes a baseline gene expression profile and a description of an intervention, then predicts how gene expression will change. Biologists can use these predictions to choose their next experiments. Model accuracy is assessed against laboratory measurements, so technical aspects of the experiment also affect its evaluation.
The authors of the new study began by asking which components of the response to a given protein vary between people. They examined the Parse dataset, which contains measurements from nearly 10 million immune cells from 12 donors. In the original experiment, cells were treated separately with 90 cytokines, proteins that act as signals. The Allen Institute had already used these data to identify cell types and analyze their responses. Before comparing donors, the authors needed to establish how much variation the experiment itself introduced.
Cells from each donor were distributed across a plate with 96 compartments called wells. Each protein was always assigned to the same position, while six wells containing a saline control were grouped along the edge. This meant that each protein was associated with the same well position and handling conditions across donors, allowing a technical signal to resemble a reproducible biological response. Even two wells containing the same solution differed in the expression of hundreds of genes. Under the criteria used in the original study, a typical pair differed in the expression of 690 genes; restricting the comparison to changes of at least twofold left 12 genes.
Alongside differences between wells, the authors found a shared background signal: proteins that were not expected to affect these cells produced similar shifts in gene expression relative to the saline control. The shared component of the response included both this background and a biological response common to different active proteins. The model score depended on the similarity between predicted and measured changes. In one test, a prediction that was identical for every protein achieved about 60% of the score of a simple method that combined the average response to a particular protein with an adjustment for the characteristics of the cells being studied. Under the original evaluation, State scored below even this identical prediction in all three cell types tested.
The authors subtracted the average response to presumed inactive proteins from the measurements. From each model’s predictions, they subtracted that model’s own predicted average for the same proteins. The revised score measured how accurately a model predicted the difference between the response to a particular protein and the response to inactive proteins. The biological response shared by active proteins was retained.
State outperformed the simple method on the average score in two of the three cell types. In CD14 monocytes, a type of immune cell found in blood, it performed better for 11 of 12 donors, including when stricter criteria were used to select inactive proteins. State predicted the average response to inactive proteins less accurately than the simple method did. Subtracting each model’s own average separated this error from its accuracy in predicting differences between responses.
The authors recommend reporting the accuracy of predicted differences separately from the error in predicting the shared shift. For future experiments, they recommend randomizing treatment positions across donors’ plates, distributing control wells throughout each plate, and including independent replicates. These changes would allow researchers to test whether responses remain reproducible when well positions and handling conditions change.