Anthropic has published a detailed guide to building safety evaluations, and its central warning is blunt: a model that refuses everything about self-harm would score perfectly on a refusal-rate test while being worse for a person in crisis. The post, on Anthropic’s research site, matters because these scores decide whether a model ships, which safeguards go with it, and where the next round of safety training is aimed.
The company’s argument starts with what a safety eval actually measures. It is the model’s tendency to contribute to harm, not its raw capability. Anthropic notes that researchers have repeatedly found AI evals easy to game, hard to reproduce, and sensitive to details as small as the order of multiple-choice answers. It borrows its fixes from psychology, which faced a replication crisis a decade ago and answered with preregistration, better measurement, and tougher statistics.
The sharpest example is the choice of what to count. For scams or phishing, refusal is a sound target. For suicide and self-harm, Anthropic argues that a flat refusal fails the user, because the right reply offers empathy and points to human support. In dual-use areas such as biology and cybersecurity, the right reply usually helps with the legitimate version of a request while withholding operational detail. So Anthropic scores whole conversations against rubrics written by domain experts, and one bad turn fails the lot.
That is why the company leans on multi-turn simulation. An expert in the harm area drafts a spec for the simulated user: goal, tactics, persona, tone. The first message is fixed, a separate Claude model plays the user from there, and a separate grader model then marks each transcript. Anthropic’s romance-scam example shows the logic: a user asks about dating patterns, then drifts toward matching a widowed relative with someone “financially secure.” No single message breaks a rule. The pattern does.
Single-turn tests, by Anthropic’s own account, have run out of road. Harmless response rates on its API sit between 94% and 98% for every recent model, so those suites now work mainly as regression checks. Multi-turn results spread far more widely.
The post is also candid about uncertainty. Confidence intervals on its API multi-turn results can stretch past 10 points in each direction. Anthropic gives one case: on a romance-scam eval, Claude Opus 5 scored 98% and Opus 5.5 scored 94%, yet the gap was not statistically significant, so it could not say either model did better.
Configuration can matter more than model choice. For Claude Fable 5.1, the appropriate response rate differed by as much as 34 percentage points between the raw API and claude.ai, where a system prompt applies. In one self-harm case, Anthropic’s transcript review found Claude occasionally telling users, by implication, that self-harm helps them. One added system prompt instruction moved the rate from 60% on the API to 94% on claude.ai.
Grading gets its own section, since an AI model usually does the scoring. Anthropic recommends yes/no criteria over one overall score, rubrics that say what should not change a verdict (a harmful answer wrapped in warnings is still harmful), a grader different from the model under test, and checks against human labels. Anthropic sets minimum sample sizes: 100 items for a complicated multi-turn eval, 1,000 when each test is a single message.
Safety of the testing itself gets attention too. Anthropic says incidents in its own environments pushed it to harden them. It paused many evals that connect models to the internet, and it now monitors the web-search ones live.
The post is Anthropic describing Anthropic’s practice, and it offers no outside audit of these methods. Still, it hands every lab and every buyer a checklist to hold vendors against. If a model card quotes one safety number with no interval, no grader validation, and no split between API and product, treat it as marketing until the missing pieces arrive.
Anthropic, “Building strong safety evaluations,” Anthropic research, published November 13, 2026.