No single AI model wins at vulnerability detection: Study

Software teams hoping one AI model can catch every security flaw may be disappointed. A new comparative study of 11 leading large language models found that none of them reliably outperforms the rest at detecting vulnerabilities across different types of code.

In the International Journal of Applied Cryptography in 2026, the research was carried out by Vasileios Kouliaridis et al., where the researchers at the University of the Aegean and the Joint Research Centre of the European Commission analysed the open-source and proprietary LLMs using four publicly available benchmarks.

The benchmarks included applications for the Android platform, IoT software, and blockchain smart contracts. The researchers also checked whether the retrieval-augmented generation technique would help improve detection rates, which involves feeding the model with external references.

A number of systems performed well in particular areas, but there was not any system that was superior on all four datasets. 

According to the research authors, these findings indicate that contemporary LLMs are inappropriate for scanning vulnerabilities as universal tools, and it would be more rational to select certain systems depending on the software category being evaluated.

Outdated training sets and the phenomenon of AI hallucination are seen by the authors as the most significant limitations.

Industry data referenced alongside the findings suggests newly discovered vulnerabilities have climbed sharply year over year, with exploited vulnerabilities reportedly up 96%, though these figures come from third-party industry tracking rather than the study's own dataset.