Prime Radiant, the company publishing under the prime-radiant-inc GitHub organization, has released smevals, an open source command line framework for building and grading evaluations of language models. It ships under the MIT license and installs with uv tool install smevals, pip install smevals, or a one off uvx smevals --help. The tool’s top level unit, called an Eval, is defined in its documentation as a way to measure how good a model on its own, or a model paired with an agent harness, performs at one specific capability. That second option is the real news: smevals scores the scaffolding wrapped around a model, not only the model itself.
The vocabulary underneath that idea is layered on purpose. An Eval is a directory holding one or more Tasks, the individual exercises a model must complete, and one or more Configs, each specifying a model along with optional parameters, prompts, and tool access. Evals can be grouped into Suites purely to organize them on disk. Executing a Task against a Config through a Runner, a lightweight CLI program which can call a model directly or drive an agent harness like Codex or Pi, produces an immutable Run.
That inclusion of the harness is a deliberate design choice, not an afterthought. Public benchmarks typically score a base model, but the numbers a business actually experiences come from a model wrapped in prompts, tool calls, retries, and an agent loop built by whoever shipped the product. Swap the wrapper and the same underlying model can look sharper or weaker on an identical test. Smevals treats a model plus its harness as the single unit under test. Two teams can run the same Eval against the same base model with different agent code and land on different, comparable scores, which separates a claim about raw model capability from a claim about a shipped system.
Grading follows the same layered logic. Each Grader is defined in YAML and lists an ordered sequence of Checks; every Check names a Checker, either a built in operation, such as contains, which scans a response for a substring, and xml-valid, which confirms a file parses, or a custom executable that can do something more involved, like rendering an SVG and asking a separate model to judge the image. Checks can be marked required, so one failure halts the rest of the sequence. When a Runner exits with a non-zero status, the Run it produced is marked failed, treated as a harness error, not proof the model itself failed, and left out of grading and reports entirely. The -n flag tops each Task and model pair up to a target count of successful Runs, which accounts for non deterministic output and lets an interrupted session pick back up without redoing finished work.
Five commands cover the workflow: run executes Tasks against one or more models, grade applies a Grader to ungraded Runs, report prints a leaderboard of scores by config and model, serve opens a live web UI that rereads results from disk each time it refreshes, and build exports that UI as a static site. Multiple Graders can sit on top of the same Runs at once, so a project can pair a cheap deterministic grader with a slower model judged one without re-executing anything.
Smevals fits teams building or operating an agent product, a coding assistant, a support bot, anything wrapping a model in tools and a loop, who need to compare full configurations rather than just swap base models. It would replace the pile of one off scripts most eval curious teams already maintain to glue a prompt runner to a grading function. What the README cannot tell a reader is how established the project actually is. The repository carries a published PyPI package, an MIT license, and a passing test badge, but it says nothing about who runs it outside its own haiku grading example, how many teams have adopted it, or how it holds up at a scale beyond that demo.
Today’s issue also covers Ramp’s private production benchmark, built after public leaderboards saturated and their questions leaked into training data, a response to the same measurement crisis from the opposite direction. Teams currently trusting a vendor’s leaderboard score should run smevals, or an equivalent, against their own production harness before locking in a model choice for the next quarter.
Prime Radiant maintains the smevals project and its documentation, published on GitHub under the prime-radiant-inc organization.
