New BIML Research Finds Critical Flaws in AI Security Measurement, False Confidence in Benchmark Scores

Better processes and internal model observability are needed for organizations to get ahead of agentic AI security risk

BERRYVILLE, VA, UNITED STATES, May 13, 2026 /EINPresswire.com/ -- The Berryville Institute of Machine Learning (BIML), the leading independent science and research organization focused on machine learning security, today published "No Security Meter for AI," a research paper that examines why none of the existing tools for measuring AI security actually work and what the field needs to build toward. The paper draws on three decades of software security history to diagnose the current state of AI security measurement and lay out a realistic path forward.

The paper was authored by Gary McGraw, Ph.D., Harold Figueroa, Katie McMahon, and Richie Bonett. BIML publishes all of its research openly under a creative commons license, with no commercial stake in the AI systems it evaluates.

"We've been here before. In the late 1990s, pen testing was treated as a security meter, but it’s actually just a badness-ometer that tops out at ‘who knows?’,” co-author McGraw said. “Benchmark scores for AI security are even worse for a bunch of reasons. Admitting there is no magic security meter for AI is step one. The way forward is hard work that we can do together."

The paper argues that AI benchmarks are fundamentally broken. They are often contaminated by their own publication and tend to measure narrow performance rather than actual capability. As the paper puts it, today's security benchmarks are 'Halloween costumes in a security theater.' Recent research from UC Berkeley confirms this: eight of the most prominent AI agent benchmarks can be exploited to achieve near-perfect scores without completing even a single task.

Security has always been hard to measure, even in traditional software. The paper compares the current state of AI security to software security in 1998, when the industry focused on buffer overflows and penetration testing. In 2026, most security people are still talking about prompt injection and AI red teaming. The field is in the same early stage, and has a lot of ground to cover.

The paper also draws a distinction the industry routinely conflates: a model that performs well on security tasks is not necessarily a secure model.

"Being good at security tasks and being secure are not the same thing. Those are completely different problems,” said co-author McMahon. “Until the field starts measuring the right thing, organizations are going to keep believing they are covered when they are not."

The paper outlines three near-term priorities for security and AI practitioners:

--Use benchmarks only for what they are designed to do – comparing models against each other on specific tasks – and never treat scores as security ratings.

--Invest in process, starting by identifying which assurance activities applied to which ML artifacts reliably reduce risk.

--Treat internal model observability as a research priority. Looking inside models rather than just probing them from the outside is a necessary step toward any robust security measurement framework.

The paper also makes a practical case for cleaning up training data. Because a model's behavior is dictated by its training sets, the paper argues that controlling data provenance and integrity is the most concrete step organizations can take for AI security today.

"Security has to be built in, not bolted on, for software and for AI. Right now the industry lacks common measurement frameworks to determine whether AI systems are actually secure,” said Phil Venables, a partner at Ballistic Ventures (a BIML benefactor). “This research is an important step toward the kind of rigorous, process-driven approach that will eventually give organizations real assurance and not just a made up number."

"No Security Meter for AI" is available at no cost under a Creative Commons Attribution-ShareAlike 4.0 license at https://berryvilleiml.com/results/no-security-meter-ai.pdf

About BIML

The Berryville Institute of Machine Learning (BIML) is an independent, donor-funded research organization focused on the intersection of machine learning and security engineering. BIML publishes all research openly, with no commercial stake in the systems it evaluates. Its work includes foundational research on ML risk, architectural analysis of large language models, and the emerging field of machine learning security (MLsec). More information is available at https://berryvilleiml.com.

Media Contact:

Gary McGraw
CEO, Berryville Institute of Machine Learning
+1 703-395-8414
gem@garymcgraw.com
Visit us on social media:
LinkedIn

Legal Disclaimer:

EIN Presswire provides this news content "as is" without warranty of any kind. We do not accept any responsibility or liability for the accuracy, content, images, videos, licenses, completeness, legality, or reliability of the information contained in this article. If you have any complaints or copyright issues related to this article, kindly contact the author above.