In a preprint, MetaGNN authors withdrew their claim that connections between reactions improved model accuracy
In a preprint, MetaGNN authors withdrew their claim that connections between reactions improved model accuracy
On September 7, in the MetaGNN preprint, the authors examined a model that estimates metabolic reaction activity from tumor gene expression data. Its near-perfect score came from a task in which the answers had already been calculated from the inputs, while in another dataset the model received zeros. The authors therefore withdrew their claim that connections between reactions improved performance by 0,105 AUROC.
The MetaGNN authors wanted to test whether connections between chemical reactions help estimate their activity in tumors more accurately. The model receives gene expression data and a map of connections between reactions. Comparing model variants was intended to show whether this map adds information about an individual tumor.
For 5 925 reactions, the “active” labels were assigned by applying a threshold to the mean of the same gene expression values used as inputs. Simply ranking these values yields an AUROC of 1,0000, the maximum score on a metric that measures whether a model ranks reactions labeled “active” above the others. This part of the benchmark tested whether the model could reproduce the rule already used to label its inputs.
For another 4 675 reactions, the stored labels could be reproduced exactly using the random number generator RandomState(42) with its fixed seed. The same random set of labels was repeated for every patient. Splitting the data by patient left this set in both the training and test data, allowing the model to memorize it instead of learning differences between tumors.
When reviewing archived colorectal cancer runs, the authors found that the numerical features in the input files for 624 patients were all zero. They had previously used these runs to claim a gain of +0,105 AUROC from connections between reactions. The model received no information about differences between patients, so this comparison did not test the contribution of those connections. The authors withdrew their earlier conclusion.
“Only one configuration in this work was run under conditions we can verify, and it did not outperform the baseline using the original expression data,” the authors write.
After the input data were rebuilt, MetaGNN achieved an AUROC of 0,5800. Simply ranking the original expression values yielded 0,6342, while a baseline that only indicates whether an expression feature is present yielded 0,6085. The authors recommend tracing how labels were generated, comparing the model with simple baselines, testing whether its predictions depend on patient data, and repeating the evaluation on another dataset.