Information overloading: dense image-text prompts jailbreak vision LLMs

What is this?

On July 3, 2026, Haoyu Zhang and Mohan Kankanhalli (National University of Singapore) with Yangyang Guo (Beihang University) published Overloading Large Vision-Language Models for Jailbreaking on arXiv. The paper introduces information overloading, a jailbreak strategy against vision-language models (LVLMs) — the multimodal systems now embedded in personal assistants, document-analysis tools, and browser and OS agents.

Most recent LVLM jailbreaks work by pushing an image out of distribution: they hide harmful intent in unfamiliar visual forms so it does not match the model’s safety-aligned training data. The authors argue that approach is running out of road, because newer models with stronger language backbones and better multimodal fusion are increasingly robust to it — and out-of-distribution tricks rarely transfer between architectures. Their alternative goes the other way: instead of hiding the request, it floods the model with it.

How it works

The core observation is a resource argument, not a disguise. An aligned LVLM produces a refusal (“I’m sorry, but I can’t help with that”) with high confidence when it clearly recognises an unsafe image-text pair. Information overloading attacks that confidence by dramatically increasing the amount and density of content the model must fuse across modalities before it can decide to refuse.

Concretely, the method combines three ingredients that the paper builds up in layers. It starts with a refined text template carrying extensive, verbose instructions rather than a short request. It adds typography-enhanced images — harmful keywords and rule templates rendered as text inside the picture, so the intent only becomes explicit after the model integrates vision and language. Then it arranges these image-typography blocks in recursive, tree-structured layouts that reference each other, exponentially increasing the multimodal complexity the model has to process. No pixel-level adversarial noise is required; the attack is built from ordinary composition, which is part of why it transfers.

The authors back this with an information-flow probe rather than just success rates. Moving from simple prompts to complex ones lowers the model’s refusal rate, and the entropy of the first output token climbs — meaning the model becomes measurably less certain about producing a refusal word once it is overloaded. That is the mechanism the paper claims: dense cross-modal input dilutes the safety signal rather than evading a specific filter. This paper documents the effect and its diagnostic probe; it does not need a reproducible payload to make the point, and none is given here.

Why it matters

The reported numbers are high and, more importantly, portable. Across open-source models (Qwen3-VL, Qwen2-VL, InternVL3.5, InternVL2, Llama 3.2-Vision) the method reaches an average attack success rate of 88.6%, beating the strongest baseline by roughly 33 points. Against commercial models — Gemini-2.5-flash, its lite variant, and GPT-4.1-mini — it averages 84.0%, exceeding the best prior method by nearly 49 points. The transfer result is the one deployers should sit with: prompts optimised on an open-source surrogate (Qwen2-VL) hit 87.9% on Llama 3.2-Vision, where the strongest baseline managed only 39.9%.

Transferability is what turns a lab result into a practical risk. It means an attacker never needs access to a proprietary model’s weights or internals: they can tune a composition against a freely available open-weight model and carry it over to a hosted commercial service. And because the raw material is “visually rich” content — screenshots, webpages, formatted documents, human-designed image-text instructions — the same structure that makes the attack effective is indistinguishable at a glance from the legitimate dense documents these models are sold to read. This is the same theme as low-resolution cognitive overload and poetry- or structure-based text jailbreaks: complexity itself, not a magic string, is the attack surface.

Defenses

Filter on the extracted text, not just the image. Because harmful intent is rendered as typography inside the picture, run OCR / text extraction on image inputs and pass the recovered text through the same safety classifiers you apply to prompts. Intent hidden in an image should not get a free pass around your text-side moderation.

Budget multimodal complexity. The attack depends on unusually dense, recursive image-typography layouts. Treat abnormally high embedded-text density, large numbers of stacked sub-images, or deeply nested visual structure as a risk signal — cap or downrank it, and consider requiring an extra review step before an agent acts on such inputs.

Filter outputs independently of inputs. Input-side detection is exactly what overloading defeats, so defense in depth matters: a separate check on the generated response (does it contain disallowed content?) does not care how complex the input was. Pair input moderation with output moderation.

Watch refusal confidence at runtime. The paper’s own probe suggests a detector: rising uncertainty in the model’s refusal tokens under heavy multimodal input is a measurable anomaly. Monitoring low-confidence, non-refusing responses to dense image-text prompts can flag likely bypasses.

Red-team with complexity, not just OOD. Robustness to out-of-distribution images does not imply robustness to overloading. Add dense, typography-heavy, recursively structured multimodal cases to your evaluation and alignment data.

Constrain what a jailbroken model can do. For agentic deployments, least privilege limits blast radius: if a compromised generation cannot reach sensitive tools, files, or the network, a successful jailbreak produces text rather than action.

Status

Item	Detail
Source	arXiv:2607.02961v1, published July 3, 2026
Authors	Haoyu Zhang, Mohan Kankanhalli (NUS); Yangyang Guo (Beihang University)
Class	Multimodal (image + text) jailbreak; safety-alignment bypass
Open-source ASR	88.6% avg (Qwen3-VL, Qwen2-VL, InternVL3.5/2, Llama 3.2-Vision)
Commercial ASR	84.0% avg (Gemini-2.5-flash / -lite, GPT-4.1-mini)
Transfer	87.9% on Llama 3.2-Vision from a Qwen2-VL surrogate
Nature	Research demonstration; no exploit code reproduced here