LLM Bridge Auditors Diagnose 60% of Vulnerable Source Inputs When Comments Remain
Key Facts
- Six configurations of five open-weight models were evaluated across 16,800 responses.
- 7,200 answers on vulnerable inputs were graded using two blinded LLM graders.
- Without comments, models diagnose 24% of vulnerable source inputs while frequently flagging the secure bridge.
- Original comments, some of which describe the flaw, increase source diagnosis to 60%.
- The paper is arXiv 2611.21777, “LLMs as Validators for Cross-Chain Bridges: Evaluating Source-Level and Transaction-Level Attack Detection on Solana.”
The arXiv paper 2611.21777, “LLMs as Validators for Cross-Chain Bridges: Evaluating Source-Level and Transaction-Level Attack Detection on Solana,” asks whether open-weight language models can diagnose flaws in cross-chain bridge code rather than merely raise alarms. The authors evaluate six configurations of five open-weight models across 16,800 responses and grade 7,200 answers on vulnerable inputs using two blinded LLM graders. The comparison at the center of the paper is simple to state: source inputs with comments versus source inputs without them.
What the study measured
The design separates detection from diagnosis. A model can flag a contract as risky without identifying what is wrong. The paper measures diagnosis on vulnerable source inputs and compares conditions that differ in whether original comments are present. The paper’s title also covers transaction-level attack detection on Solana, but the headline result in the supplied summary concerns source-level diagnosis.
Without comments, the models diagnose 24% of vulnerable source inputs. In that condition, they also frequently flag the secure bridge. With the original comments supplied, source diagnosis rises to 60%. Some of those comments describe the flaw. That detail is essential to reading the number: the comment condition can carry direct hints, so the increase measures the effect of the full condition, not the value of comments as neutral documentation.
Why the gap matters
The difference between 24% and 60% is not a small calibration issue. A validator that flags a secure bridge is producing noise that a human reviewer must clear. A validator that diagnoses 24% of vulnerable source inputs without comments is missing most of the cases in that condition. Neither result supports treating an LLM as a standalone bridge auditor. Both results support asking what context the model receives and what task it is asked to perform.
Cross-chain bridges are the paper’s subject, and the title places the work on Solana. The study’s framing is that the ability of LLMs to diagnose these flaws from source code or transaction records, rather than merely raise alarms, remains insufficiently understood. The comment comparison is built to quantify one part of that gap. It does not settle the broader question, but it makes the input condition part of the result.
How to read the percentages
The 24% and 60% figures are diagnosis rates on vulnerable source inputs. They are not overall accuracy across all 16,800 responses, and they should not be collapsed into a single score for the five models. The evaluation covers six configurations, and the supplied summary does not break out results by model, by configuration, or by transaction-level condition. The 7,200 graded answers are the basis for answer-level grading, but the summary does not give the per-condition denominators behind the two percentages.
Two blinded LLM graders add another layer. Blinding is intended to limit grader bias, but the graders are still models, and the summary does not describe human cross-checks or agreement statistics. That does not invalidate the result. It does mean the 60% is best read as a conditional measurement from this study, not as a general capability estimate for open-weight models.
The secure-bridge flag is part of the result, not a side note. The summary says models frequently flag the secure bridge when comments are removed, but it does not give a rate. That omission matters because a high diagnosis rate and a high false-positive rate can coexist. A tool that finds more flaws by flagging more inputs is not necessarily better, and the paper’s design gives readers a reason to track both sides.
What the finding supports
The paper’s contribution is to make the comment condition visible. If original comments can lift source diagnosis from 24% to 60%, then evaluations that strip comments may understate what models can do, while deployments that supply comments may overstate how much of the diagnosis comes from the code itself. Both readings point to the same reporting rule: publish the input condition alongside the score.
The authors also state that LLM capability to diagnose these flaws from source code or transaction records, rather than merely raise alarms, remains insufficiently understood. The 24% and 60% results are a step toward that understanding, not a final answer. They give security teams a concrete reason to separate flagging from diagnosis, to document whether comments are present, and to treat LLM grader output as one measurement in a longer review chain.
None of this turns the paper into a product review. It is an evaluation of model behavior under specific input conditions. The value is in the comparison: the same vulnerable inputs, two comment conditions, and a clear shift in diagnosis. That is the kind of result that can be replicated or challenged with a different set of models, a different grading setup, or a different comment policy.
The practical takeaway is modest and specific. In this study, comments that sometimes describe the flaw move source-level diagnosis from 24% to 60%. That is a result about a controlled comparison, not a claim that comments make a model a reliable auditor. The next question is whether the same pattern holds across the six configurations, the five models, and the transaction-level inputs. Those details are not in the supplied summary, so they remain open.
Sources & References
- [2611.21777] LLMs as Validators for Cross-Chain Bridges: Evaluating Source-Level and Transaction-Level Attack Detection on Solana. Primary research blog. https://doi.org/10.48550/arXiv.2611.21777
- [2611.21777] LLMs as Validators for Cross-Chain Bridges: Evaluating Source-Level and Transaction-Level Attack Detection on Solana. Primary academic paper. https://arxiv.org/abs/2611.21777
- [2611.21777v1] LLMs as Validators for Cross-Chain Bridges: Evaluating Source-Level and Transaction-Level Attack Detection on Solana. Primary academic paper. https://arxiv.org/abs/2611.21777v1
- Search | arXiv e-print repository. Primary academic paper. https://arxiv.org/search/cs?searchtype=author&query=Rahman,+Y
- LLMs as Validators for Cross-Chain Bridges: Evaluating Source-Level and Transaction-Level Attack Detection on Solana. Primary academic paper. https://arxiv.org/html/2611.21777v1
SecurityXP delivers cybersecurity news, vulnerability analysis, data breach reports, and threat intelligence.
Security Digest
The most important cybersecurity stories, CVE alerts, and threat intelligence — one concise email. Unsubscribe anytime.
Related Articles
Optimizely Signup-Flow Vulnerability Injected Malicious JavaScript Into Sites of 178 Organizations
Optimizely confirmed that the attacker exploited a vulnerability in its Experimentation account signup flow.
Java4Shell: What the CVE-2026-114938 Demonstrations in Jira, Jenkins and Keycloak Actually Show
JFrog and Semgrep analyses show how CVE-2026-114938 turns an OpenJDK Base64 decoder flaw into RCE — and CISA, Canada and Korea have issued urgent patch advisories.
Cisco Investigates Products After eMudhra Root CAs Distrusted Over Private Key Exposure
Cisco is auditing which products trust eight compromised eMudhra intermediate CAs and has published fixes and guidance for IOS XE, ASA, FMC and ISE.
Check Point Gateways Exploited via PHP2Shell and DynamicCLI Socket to Deploy ESXi Wiper
Attackers chained CVE-2026-109618/109619 with CVE-2026-123039 in Gaia DynamicCLI (/tmp/dynamic_socket) to root Check Point gateways, install TSGA, and wipe VMware ESXi hosts.