AlphaFold3, Chai-1 and Boltz-2 were tested on 489 coronavirus protein structures determined after their training: they place molecules on the protein more accurately than docking (calculations based on physical equations), while AlphaFold3 and Boltz-2 show no clear advantage over docking in distinguishing true binders from false positives among docking candidates
AlphaFold3, Chai-1 and Boltz-2 were tested on 489 coronavirus protein structures determined after their training: they place molecules on the protein more accurately than docking (calculations based on physical equations), while AlphaFold3 and Boltz-2 show no clear advantage over docking in distinguishing true binders from false positives among docking candidates
On October 12, eLife published a study by a team at the University of California, San Francisco. Mac1 is an enzyme from the coronavirus SARS-CoV-2 that suppresses the cell’s antiviral signal. The models were given its sequence and the molecular formulas of the compounds; each predicted molecular position was compared with the experimental structure.
Docking fits molecules into a protein pocket, a cavity where a drug binds. It uses physical equations to select the best fits from hundreds of millions or billions of candidates. Neural networks directly predict the complex: the protein with a molecule bound in its pocket.
Earlier studies found that neural networks are more accurate for complexes resembling their training examples, a pattern consistent with memorization. In September, the MIRAGE benchmark found a similar dependence in Boltz-2: a simple algorithm outperformed it on proteins poorly represented in public databases. The authors of the eLife study had been searching for Mac1 inhibitors and had used X-ray crystallography to determine about a thousand structures of the protein with bound molecules. For this test, they selected 551 structures determined after the models’ training; 489 could be evaluated.
A prediction counted as correct if the molecule’s atoms deviated from their positions in the crystal by an average of less than 2 angstroms. AlphaFold3 and Chai-1 met this threshold in about 70% of cases, compared with 40% for docking. Boltz-2 performed less well, at 53%, despite having been trained on data through mid-2023, whereas AlphaFold3’s training data extended only through September 2021. To test for memorization, the authors compared AlphaFold3’s prediction error with each molecule’s similarity to training examples and found no significant relationship.
Docking uses a single predefined protein conformation. Neural networks also predict the protein conformation, so they can potentially capture changes that accommodate a molecule, such as a flip in a segment of the protein chain or the opening of a loop. AlphaFold3 predicted 4 of 19 flips and none of 20 loop openings, yet it often placed the molecule correctly in these cases. The authors say this “points to a nonphysical aspect of machine learning methods”: the networks appear to learn typical patterns of contacts.
Only Boltz-2 also estimates binding affinity. Experimental measurements were available for 202 molecules: the correlation between predicted and measured affinity was 0,6 (1 indicates perfect agreement), compared with an absolute correlation of just 0,23 for docking energy.
Neural networks are roughly ten thousand times slower than docking. In a second test, AlphaFold3 and Boltz-2 therefore rescored shortlists that docking had selected from libraries for three proteins: the enzyme AmpC, the σ2 receptor and the dopamine D4 receptor. Some of these molecules do not bind to the protein in laboratory experiments.
Boltz-2 distinguished true binders from false positives better than docking for σ2, equally well for D4 and less well for AmpC; AlphaFold3 performed worse for all three. Docking had selected the shortlists, but the neural networks also failed to outperform it on molecules with intermediate or poor docking scores.
The authors attribute the difference in performance to the tasks themselves. All molecules in the Mac1 dataset bound to the protein and mostly fell into four families, so the task was simply to place a familiar molecule. The docking shortlists were diverse and deliberately unlike known binders. In the authors’ view, neural networks that rely on learned patterns struggle more with such lists. They suggest a possible division of labor: docking searches for new molecules, while neural networks refine the predicted position of a molecule found by docking and, in the case of Boltz-2, rank related molecules.