Log10, an AI evaluation and agent-infrastructure company that works with life sciences firms, published a benchmark this week testing whether open weight language models hold up on regulated pharmaceutical paperwork rather than general chat. The benchmark, called ClinReg, ran 19 models, twelve proprietary and seven open weight, through three tasks that mirror how pharmaceutical teams actually spend their time: screening medical literature for safety reviews, drafting the chemistry and manufacturing section of an FDA investigational new drug application, and generating the tables and figures that back a clinical trial submission.
GPT-5.6 Sol topped the field with a composite score of 88.4 out of 100. Two open weight models followed close behind: GLM 5.2 scored 87.4 and Kimi K3 scored 86.9. Log10 frames this as GLM 5.2 sitting within one standard deviation of GPT-5.6 Sol on accuracy. That claim is narrower than it sounds. It means the score gap between the two is smaller than the run to run variance each model shows on its own tasks, so the ranking between them is not statistically decisive. It does not mean the models produce interchangeable output on any given document, and Log10’s own error data shows exactly why that distinction matters.
The cost difference is unambiguous even where accuracy is not. Running GLM 5.2 cost close to 34 percent of what GPT-5.6 Sol charged per task. Kimi K3 landed near 60 percent. A separate batch of proprietary systems, GPT-5.6 Terra, GPT-5.5, Grok 4.5 and Opus 5, scored between 84.3 and 86.3, close to the open weight leaders, yet spread across an elevenfold range in cost among themselves. Log10 notes that Opus 4.7 and 4.8 were the top performers on comparable tasks only a few months earlier, at roughly nine times GLM 5.2’s current cost, a pace of change that shortens how long any single benchmark snapshot stays useful.
The more consequential finding sits underneath the leaderboard. On the manufacturing filing task, the strongest performers, the GPT-5.6 lineup and Kimi K3, struck a balance between thoroughness and honesty, filling in only what the source document actually supported and flagging the rest as missing. Gemini 3.1 Pro erred toward caution, leaving out real content rather than risk a wrong answer. MiniMax M3 did the opposite, inventing plausible values for fields the source never specified. On the trial data pipeline task, GLM 5.2 and Kimi K3 kept refining their code until validation checks passed, while Opus 4.8 and Grok 4.5 tore up and rebuilt whole scripts on nearly every pass, and Gemma and DeepSeek V4 burned iterations without reliably converging. Log10 also tracked two deterministic error types on the filing task, omission and fabrication, and found Opus 4.8 omitted more fields while GLM 5.2 fabricated more, at a similar overall score.
That is the part worth sitting with. Two models can land within a point of each other on a composite score and still fail in opposite directions: one leaves gaps a reviewer has to catch, the other inserts numbers a reviewer has to catch. In a regulatory submission those are not equivalent risks, and a single leaderboard rank hides the difference entirely. Evaluation practice built around one aggregate accuracy figure will keep missing this. Teams selecting a model for a compliance workflow need the error profile, broken down by which failure mode their own review process is weakest at catching, not just the score.
Log10 built ClinReg on public data: a Cochrane systematic review for literature screening, a European regulator’s published assessment report for the manufacturing task, and the CDISC Pilot clinical trial dataset for the tables and figures task. The company documented its scoring approach, including a four model judge panel with a three-of-four confirmation requirement for the filing task, in a published appendix. That transparency matters because Log10 sells services built on these same workflows, and it says it plans to open source the benchmark tasks. Until an independent group replicates ClinReg on its own regulatory workload, the results are a serious first data point rather than a settled ranking.
Teams currently locked into closed models for compliance heavy document work should pilot GLM 5.2 or Kimi K3 against their own review checklist, weighting the comparison toward whichever failure mode their auditors catch slowest rather than toward the headline score.
Arjun Bansal, writing with the Log10 research team on his Substack, published the ClinReg benchmark findings on July 30, 2026.
