Skip to main content
Stat grid: the key figures behind this report. Illustration: SecurityXP

MLLMs Trust the Layout: A New Jailbreak Wraps Harmful Queries in Authoritative Documents

· 3 min read · SecurityXP Editorial Desk

Key Facts

  • The paper introduces the Visual Trust Bias Attack (VTBA), a black-box jailbreak that places a harmful image-text request inside an authoritative document layout.
  • VTBA uses an offline bank of eight templates drawn from academic journals, a major newspaper, police records, incident reports and government portals.
  • The attack rewrites the harmful query into a neutral headline-style phrase, inserts it into the template, and uses a harmless guiding prompt to elicit the response.
  • The authors built NatureMMJail, a dataset of 2,500 naturally occurring image-text pairs across five categories, plus a 500-pair Tiny subset for evaluation.
  • VTBA reported average attack success rates of 69.2% against GPT-4.1, 66.3% against GPT-4o, 68.4% against Qwen-VL-Max and 64.7% against Gemini 2.5 Flash.
  • On newer models the paper reports 61.2% for GPT-5.1, 63.6% for Gemini 3 and 60.2% for Grok 4.1, still well above the perturbation and generation baselines.

Multimodal large language models are trained on an internet’s worth of documents: journal articles, newspapers, police forms, government portals. A new paper argues that this exposure creates a blind spot. When a harmful request arrives inside a layout that looks authoritative, the model is more likely to treat it as legitimate.

The paper, “Dressed in Authority: Practical Black-Box Jailbreaking of MLLMs via Visual Trust Bias,” is by Weihang Liu, Qianqian Xu, Yangbangyan Jiang, Zitai Wang, Zhipeng Yu and Qingming Huang. It describes an attack that leaves the harmful image and the harmful intent intact, but changes the visual frame around them.

What the attack does

The Visual Trust Bias Attack, or VTBA, starts with a bank of eight document templates. The authors draw them from academic journals, a major newspaper, police records, an incident investigation form and government portals. Each template has a place for an image and a headline.

For a given harmful image-text pair, the attack selects the template whose authority theme best matches the intent, rewrites the query into a neutral headline-style phrase, and renders the result. A short, harmless guiding prompt then asks the model to interpret the document.

The important part is what the attack does not do. It does not add adversarial noise to the image, and it does not generate an obviously toxic visual. The image stays natural. The harm is carried by the combination of the original content and the authority cues around it.

Why it works

The authors trace the effect to what they call visual trust bias. In attention maps from LLaVA, early decoding tokens focus on institutional logos, structured columns and header typography. Those are the visual markers of credibility that appear throughout pretraining data.

The paper compares the approach to the Milgram obedience experiments: people defer to perceived authority even when the instruction conflicts with their own judgment. The model, the authors argue, has learned a similar prior. An authoritative frame suppresses the refusal that the same content would trigger on its own.

That is a design problem, not a prompt-injection trick. Safety training mostly teaches a model what to refuse. It does not necessarily teach the model that a journal masthead is not evidence that the request is legitimate.

The numbers

The authors built NatureMMJail, a dataset of 2,500 natural image-text pairs across five categories and 32 scenarios, plus a 500-pair Tiny subset. They evaluated VTBA against perturbation-based and generation-based jailbreaks.

On the full set, VTBA reported average attack success rates of 69.2% against GPT-4.1, 66.3% against GPT-4o, 68.4% against Qwen-VL-Max and 64.7% against Gemini 2.5 Flash. Against newer models it reported 61.2% for GPT-5.1, 63.6% for Gemini 3 and 60.2% for Grok 4.1. The baselines were far lower.

The category breakdown matters for defenders. Identity discrimination was the hardest category to jailbreak, with an average success rate of 44.4%. Cybercrime and system intrusion, and dangerous and controlled substances, were more susceptible. The authors suggest those categories are especially responsive to professional-looking academic and journal layouts.

What it means

The paper’s practical warning is that content moderation cannot rely only on the text of a request or on explicit toxic imagery. The same request can be benign in one frame and effective in another. Document-aware classifiers, layout-aware safety training and red-team sets that include authoritative designs are the obvious responses.

There are limits to the result. The evaluation uses GPT-4o as a judge, with a human and model cross-check on a sample. The template bank is small and curated. The attack is black-box but still requires the attacker to choose a template and rewrite the query. And the paper is a pre-print, not a deployed product.

None of that makes the finding less useful. It shows that trust is an attack surface, and that the visual grammar of authority is part of the model’s input, not a neutral wrapper around it.

Tags: MLLM Jailbreak AI Safety Visual Trust Bias arXiv Multimodal Security

Sources & References

SE SecurityXP Editorial Desk
SecurityXP Editorial Desk Vulnerability Research & News Board

Automated and analyst-reviewed threat intelligence briefings tracking active exploitation campaigns, CVE disclosures, and extortion group activity.

Security Digest

The most important cybersecurity stories, CVE alerts, and threat intelligence — one concise email. Unsubscribe anytime.

Related Articles

Process diagram of AI access drift: approval, roles and groups, privilege drift, and unreviewed access, with privilege drift marked as the fault.
AI/ML Security

AI Approval Is Not Access Control: What the 97% Breach Statistic Actually Says

IBM's 2025 Cost of a Data Breach Report found that 97% of organizations reporting an AI-related breach lacked proper AI access controls. Approval is not the same as an access review.

Process diagram of the data breach path: Markdown parse through Data exfiltration, with the fault at Prompt injection.
AI/ML Security

OpenAI Codex Desktop App for macOS Vulnerability Allows Attackers to Inject Indirect Prompt

As AI-assisted development tools continue to gain adoption, vulnerabilities like CVE-2026-14898 underscore the need for secure design practices that account...

Process diagram of the exploitation chain: Model Inference through Misinfo Misuse, with the fault at Output Handling.
AI/ML Security

Threat Modeling Generative AI: What 11,658 Incidents Reveal About Real-World Risk

Instead, improper output handling (42%) and misinformation/misuse (35%) represent the vast majority of actual incidents.

Process diagram of the data breach path: Leaked prompts through Data exfiltration, with the fault at Leaked prompts.
AI/ML Security

AI-Enabled Hacker Caught: Leaked Prompts Expose Resume and IP Address

A fully AI-enabled hacker was caught, revealing his full system prompts, which included his resume and his IP address.