A single disabled line, ”// Don’t use Azure here,” is enough to pass an eval that only checks whether the word “Azure” appears in generated code. That is the failure mode Waldek Mastykarz, a Microsoft 365 and AI consultant, lays out in a September 19 post on his blog, and it applies to any team grading coding agents by scanning output for expected strings.
Mastykarz’s target is the string contains check: cheap, deterministic, and fast enough to run on every commit without touching an inference budget. The problem is what a pass actually proves. If the grader looks for “Azure” and finds it in a comment, dead code, an unused dependency, or a stray config file the application never loads, the check still returns green. The behavior the team cared about, whether the agent correctly wired up Azure, was never tested.
The reverse failure is just as costly. An agent that uses an SDK without naming the vendor, references a class instead of a brand string, or phrases intent in words the eval writer never anticipated will fail a check that is actually accurate about what it measured, just not about what the team wanted to know. Mastykarz frames this as an inference gap: the grader is truthful about the string, but the team reads a claim about behavior into a result that only ever described text.
His fix is a two-sentence completion exercise applied before any grader ships: “If this grader passes, I now know that ______,” and the matching sentence for a failure. For a bare string match, the honest answers are narrow, something like “the string Azure occurs in the inspected content.” When a team’s actual intended claim (“the agent correctly used Azure”) does not match that sentence, the grader is measuring the wrong thing.
The prescription is not to abandon deterministic checks. Mastykarz argues the opposite: when a property is genuinely deterministic, use the system built to answer it. If the question is whether generated code compiles, run the compiler rather than asking an LLM judge to guess at correctness, since his team has seen solutions score well on judge review and then fail to build. If the question is whether dependencies restore or a schema validates, run the restore or the validator. String matching becomes a problem specifically when it substitutes for a check the team could run directly but skipped because building, deploying, or provisioning credentials for the real system is harder.
LLM judges get their own carve-out: semantic questions, like distinguishing “use Azure for storage” from “do not use Azure for storage,” are exactly where a judge earns its cost. But Mastykarz warns against using a judge to approximate what a deterministic tool already answers authoritatively, calling that redundant rather than rigorous.
The framing he lands on treats agentic evals as integration tests rather than unit tests: separate gates for build, test, run, and deploy, with LLM judges reserved for the semantic layer sitting alongside them. That separation matters for any team shipping an MCP server, an agent skill, or a coding extension where CI currently reports a passing eval suite. A grader that runs on every commit is not the same as a grader whose evidence supports the claim being made from it.
Teams running agent evals in CI should audit their existing graders against Mastykarz’s two-sentence test this quarter: any check whose honest “if it passes” sentence does not match what the dashboard claims needs to be rebuilt or replaced with a direct system check before the next release cycle.
Waldek Mastykarz, “Is your eval lying to you?”, published September 19, 2026 on his blog.
