September 21, 2026 · Edition #33 · by Asaf Nakash
The result I wasn't testing for
Could I get an AI to give a candidate a better score without changing a single qualification?
This week's BragJack research showed browser extensions reaching privileged AI-assistant components. I wanted to try a different route: instructions in a professional profile, aimed at the AI reading it.
I found public profiles addressing AI readers directly. That gave me a pattern to test, not evidence that it worked. I built a fictional candidate, kept the experience and scoring criteria fixed, and used AI to run clean and modified versions through the evaluator. Some versions asked for a perfect score or an endorsement phrase. One asked for a cookie recipe.
I expected the interesting part to be how the evaluator handled those instructions. Across 33 evaluations, including 24 with planted instructions, none produced the requested perfect score, phrase or recipe. Every clean and modified profile still got a recommendation, so that alone couldn't demonstrate influence. A small test in a shared agent setup cannot settle how recruiting systems behave.
Then the audit of the analysis changed the story.
The AI-generated summary said there had been 30 injected evaluations. There were 24. Six clean baselines had been counted as attack attempts. Some scores had also been assigned to the wrong test conditions. The original responses existed; matching them back to their requests exposed the errors.
That's the part I hadn't built the experiment around. When I use AI to help evaluate another AI, the analysis is part of the system under test. It doesn't become reliable just because it is on my side of the experiment.
Another model reading the same table would still need the original records to establish which response belonged to which test. Before asking for an interpretation, I want the counts and comparisons calculated from those records, with the connections preserved.
I started by asking whether I could trust the evaluator. I finished asking what would make me trust the evaluation.
Written by Asaf Nakash, Principal Product Manager for AI Security at Microsoft Defender and host of the Context Window podcast. Originally published in Context Window Edition #33, September 21, 2026.