Framework · AI Security
Recognition Is Not Resistance
An agent can name the attack and run it anyway
Recognition Is Not Resistance holds that threat awareness and threat resistance are orthogonal capabilities in an AI agent, not two points on one scale. The intuition borrowed from human security is that an operator who spots a social engineering attempt will therefore refuse it — awareness is most of the defense. Agents break that link. Recognition is a description task: the model produces accurate text about what is being attempted. Refusal is a control decision about its own subsequent behavior. Both come out of the same generator, so naming the attack yields a sentence, not a gate. This is why an agent will identify a prompt injection by name, explain the risk it poses, and then execute it — and why a model's self-assessment is the weakest available evidence that it was not compromised, since the faculty producing the assessment is the one under attack. The defensive consequence is that detection belongs outside the agent. Anything the agent must not do has to be enforced by a layer that does not take instructions from the conversation.
The observation
The finding came out of a live agent exposed to a few hundred people who spent six months trying to break it — 4,934 scored attempts, no security training among the attackers, a real agent with real tool access. The experiment was Alex Liverant's: he built the agent, ran the challenge, and hardened it nightly in public. I was one of the participants, and the analysis is the attacker's side of his experiment.
The expected failure was an agent that got fooled. That is not mostly what happened. Again and again the agent saw it clearly. It named the technique, described why the request was dangerous, sometimes explained precisely which of its own protections the request was aimed at — and then did the thing anyway.
One refusal was itself a leak. Asked to hand over its instruction file, the agent declined, and in declining confirmed the file's name, its contents, and that obtaining it would be a total compromise. It understood the stakes well enough to explain them to an attacker.
Why the two capabilities come apart
We inherited the assumption from human security, where noticing and resisting are close to the same act. If a person recognizes a phishing email, they generally do not click it.
In an agent they are different jobs. Recognition is description — the model is very good at producing accurate commentary about the situation it is in. Refusal is a decision about what it does next. Nothing structurally connects the two, because the commentary and the action are drawn from the same process. Correct analysis is one more plausible continuation, not a brake.
That is also why an agent's own account of its security state should carry no weight. In one case the agent classified an attack as blocked and only revised the verdict when a bystander pointed out it had already leaked its rules. It needed an outside observer to learn what it had done.
What hardening did and did not fix
Six months of patching worked, and the data can prove it while holding the attacker population fixed: among attackers active in both the early and late periods, success fell from 37.1% to 9.7%, and 8 of 9 got measurably worse.
What survived is the interesting part. The late compromises are not the early ones with better wording — they moved up the stack. In one, the agent was talked into scanning a local network, enumerating devices, and taking over a television. In another, a stranger sent the single word "Continue" into an ongoing conversation and the agent carried on as though its operator had said it.
Neither is a content-filtering failure. By then the agent was good at recognizing hostile text and still could not establish who it was talking to. Recognition improved; authorization never existed.
How to apply it
Do not accept detection as a control. An agent that can describe an attack class in detail is not protected against it, and a benchmark that measures whether a model identifies an attack is not measuring whether it will resist one.
Put refusal where the conversation cannot reach it. If an action must not happen, it needs to be blocked by something that does not read attacker-controlled text — which is the same conclusion as the Likelihood Collapse, arrived at from the opposite direction.
Treat identity as the unsolved half. Content defenses matured quickly; the question of who is speaking did not. In any multi-user setting, assume the agent cannot tell.
Recognition Is Not Resistance was introduced by Asaf Nakash, Principal Product Manager for AI Security at Microsoft Defender and host of the Context Window podcast, in Attack Patterns Against Live LLM Agents — field research (March 2026).
On the name
First stated in the March 2026 write-up as "threat awareness and threat resistance are orthogonal capabilities," and shortened to Recognition Is Not Resistance when the research was prepared for OWASP AppSec Israel. The stylization "Recognition ≠ Resistance" is used in talk titles and slides; the spelled-out form is canonical in writing.