General summary
Predicting how proteins interact is a major key to unlocking new medical treatments, but computational models have long been held back by a surprising problem: a severe lack of high-quality data on proteins that do not interact. To solve this, researchers developed “Flock,” a massive new dataset built using advanced AI agents to read scientific literature and compile the largest collection of non-interacting protein data to date. By providing a rigorous new benchmark free of the hidden data overlaps that artificially inflate test scores, Flock exposes the current limitations of protein models and sets a rigorous new gold standard for the future of drug discovery.
The blind spot in protein modelling
Historically, protein–protein interaction (PPI) prediction models have been trained almost exclusively on positive data or random protein pairs that are assumed to be non-interacting but might actually contain hidden false negatives. Legacy datasets documenting non-interacting proteins are notoriously limited in scale and riddled with mapping errors or incorrect labels.
Furthermore, a pervasive “data leakage” issue plagues model evaluation. Co-folding models are often trained using simple date-based cutoffs without strictly holding out specific protein sequences or structures, meaning models are frequently tested on interfaces they have effectively already seen during training. This leads to artificially inflated performance metrics and a false sense of security regarding a model’s predictive power.
Enter Flock: unprecedented scale and agentic curation
To solve these structural issues, the researchers developed Flock, a negative-enriched PPI dataset of unprecedented scale.
- Flock contains 26,934 positive interaction pairs and a staggering 377,643 negative pairs.
- The negative data was rigorously sourced from Protein Data Bank (PDB) biological assemblies and meticulously mined from scientific literature.
- To mine the literature, the team utilised a state-of-the-art agentic Large Language Model (LLM) workflow, marking the first major negative PPI data update since 2013 and the debut of agentic LLMs for this specific curation task.
The leakage-free benchmark
Crucially, the paper introduces Flock’s Leakage-Free Set. This benchmark strictly controls for sequence similarity, interface similarity, and the training date cutoffs of existing models, ensuring that testing data is genuinely novel.
When the researchers evaluated state-of-the-art models on this rigorous set, the results exposed the reality of current modelling capabilities:
- The co-folding model ESMFold2 experienced a pronounced performance degradation on the Leakage-Free Set compared to a “leaked” set containing previously seen proteins.
- The protein language model ESMC demonstrated much stronger generalisation capabilities, showing a significantly smaller performance gap between the leaked and leakage-free evaluations.
- Both models saw substantial accuracy drops compared to their originally reported benchmarks, proving that previous performance claims were likely overinflated due to poor negative sampling and data leakage.
Access to confident, LLM-curated negative data will force developers to confront data leakage directly, ultimately paving the way for more reliable protein structure models and accelerating therapeutic discovery.