AI-labelled protein pairs used to test how well function predictors handle unknown proteins

The work

Technical · from the record

The study assembled 11 444 metagenome-derived 'orphan' proteins with under 30% sequence identity to UniRef100 and, lacking ground-truth function labels, used a Siamese neural network score together with Foldseek structural alignment to label 6219 of 309 549 structurally alignable protein pairs as likely sharing molecular function. Thirteen function annotation tools, several protein and DNA embedding models, and the ortholog finder SwiftOrtho were then scored on whether they assigned matching annotations to both proteins of a labelled pair. The paper reports ECPred, GOPredSim, Pfam HMMER and GhostKOALA as the top performers on the ΔS and F1max metrics, with all methods scoring closer to the random baselines than to the estimated performance of experimental annotation, and reports that embedding distances alone did no better than random on the ΔS and RecallmaxPPf50 metrics. Method predictions were enriched for sequence-similar pairs, which the authors read as a dependence on homology.

How AI was used

Protein sequences and pre-computed ESMFold structures were taken from the ESM Metagenomic Atlas, filtered by mmseqs2 alignment against UniRef100 and by length and CD-HIT redundancy reduction to an orphan set. Foldseek aligned all orphan structures against each other, and a previously trained Siamese neural network — a pretrained LookingGlass embedding layer, an LSTM layer and an embedding-distance computation — scored functional similarity for every alignable pair; pairs above fixed TM-score and SNN-score cut-offs were labelled siblings, giving a positive-versus-unlabelled test set. The same SNN plus TM procedure was applied to enzymes carrying single experimentally evidenced EC numbers to estimate its precision and recall and an ideal-predictor reference. Thirteen annotation tools predicting GO molecular function terms, EC numbers, Pfam domains or ortholog groups, plus seven sequence embedding models and SwiftOrtho, were run over the orphan proteins; their per-protein outputs were converted into per-pair similarity using information-accretion-weighted Jaccard similarity for GO, plain Jaccard for other vocabularies, and Euclidean or cosine distance for embeddings. Methods were scored by sweeping prediction-score and similarity-score thresholds, with unlabelled pairs under-sampled 100 times, against random-classifier and random-annotator baselines and Wilcoxon and t-test significance testing.