Empero, a small AI lab, has taken a 165-million-parameter language model through every stage of a modern training pipeline for roughly $4,200 of compute, and released four checkpoints under Apache-2.0. The model, Quark, is not meant to beat anything big. According to Empero’s write-up, its job is to prove the stack works before the lab commits real budget to a larger model.
The bill covers about 1,300 H100-hours. Empero says the tally also absorbs an abandoned first design, dead-end experiments, synthetic data jobs, and all of its evaluations, and that lium.io sponsored part of the compute. A full dress rehearsal at that price is cheap insurance. A broken data job or a fragile restart path costs far less to find on a small run than halfway through a big one.
The architecture is built around inference cost. Of Quark’s 16 layers, twelve look only at the most recent 256 tokens, and every fourth layer attends to the entire context. Just those four global layers need a full key and value cache, so at 128K tokens the cache comes to about 0.5 GiB in bf16, a quarter of an all-global design.
Pretraining used a single 8×H100 node and roughly 500B tokens. The node failed seven times, and Empero reports no loss spike at any restart. It resumed the first two runs by hand; later failures recovered automatically within two to four minutes. Since the step count alone fixes the order of the data, a resumed run sees exactly the tokens it would have seen anyway.
On nine zero-shot benchmarks, Quark-150M-128K-Base averages 52.8, the best result among open models between 120M and 170M parameters with comparable token budgets. It leads on six of the nine tasks. All figures come from Empero’s own runs with a shared harness. SmolLM2-135M, trained on 2T tokens, averages 55.3 and wins seven of nine, so Quark’s win holds only against peers that saw a similar amount of data.
The more unusual result comes from reinforcement learning. The Quark-150M-128K-Zero checkpoint was trained with GRPO, skipping supervised fine-tuning entirely, and no output from any other model served as a training target. Code-based checks such as unit tests and exact-answer matching supplied about 95 percent of the rewards, with LLM judges scoring the remainder. Before that stage, the base model emitted tool calls and then made up what they returned: only 3 of 1,152 audited replies ended correctly. In training rollouts on easy terminal tasks, spread over 15 emulated agent harnesses, the solved share climbed from 3 percent to 65 percent.
Empero is unusually frank about what failed. Web search never improved, and one harness format failed outright. Scores on long inputs also dropped during the RL stage. Quote attribution breaks at every length in the long-context checks. The agentic figures are training rollouts rather than a held-out agent benchmark. Handed a well-known bat-and-ball riddle, with its thinking mode switched on, the model confidently answers $2.10. It has not been safety-evaluated, and Empero calls the release a research artifact, not a product.
The release also includes Quark-150M-32K-Raw, the checkpoint from before the final anneal, which Empero says is rarely published and makes a natural base for further pretraining. For other small labs, that matters more than the leaderboard position: a documented, restartable, cheaply reproducible recipe is the scarce item here.
Empero says its next step is scaling this recipe to a bigger model and running the RL stage to full length. Teams budgeting their own from-scratch training should copy the method before the model: rehearse every stage at small scale, and publish the failures next to the scores.
Based on reporting by Empero (empero.org), published November 15, 2026.