A preprint derives an accuracy limit for a single protein variant score across different assays

A preprint derives an accuracy limit for a single protein variant score across different assays

Zhongmin Liu's August 26 preprint establishes an accuracy limit for models that assign a single score to protein variants across different biological assays. This limit depends on how closely the assays agree with one another.

A single amino acid substitution in a protein can produce different effects in different assays. One assay may measure how well the protein performs its function, while another measures how much of the protein remains in the cell. A model that assigns a single score must rank the variants in the same order for both settings. The laboratory then selects the highest-ranked variants for synthesis or further testing.

In Liu's study, the limit follows from the agreement between the assays themselves. When two assays rank variants identically, the correlation between their rankings is 1. Across 136 pairs of assays for the same gene from an open repository of large-scale protein variant measurements, the mean correlation was 0.555. Across 18 comparable pairs from another dataset, it was 0.681. These values set upper bounds of 0.745 and 0.825 for the mean correlation achievable by a single prediction.

The author compared 541 pairs of measurements that used the same variant library, platform, and laboratory. Only the protease differed, meaning the enzyme through which the assay measured protein stability. Agreement within these pairs reached 0.897. According to the author's decomposition, differences in biological conditions account for 70–90% of the gap to the upper bound in the analyzed datasets. Measurement error accounts for the remainder. Different conditions measure different properties of the same protein.

Mean correlation and candidate selection evaluate a ranking in different ways. Across 19 proteins, published predictors achieved 63% of the attainable performance measured by correlation. When the task was to select the top 1% of variants for the next experiment, the same predictors reached 18% of the upper bound. This result means that the highest-ranked variants overlap less consistently between two rankings than the rankings do across all variants.

Liu retained the same set of 192 input features, which provide the data the model uses to evaluate a variant, and changed the training target. The model instead estimated the proportion of conditions in which a variant ranked within the top 5%. Across all 18 protein targets, the accuracy of this selection increased from 0.175 to 0.235. The author recommends measuring agreement between assays before comparing models, choosing a metric that matches the laboratory decision, and providing the model with information about the conditions of the specific experiment.