Replit has built two internal systems, ViBench and Telescope, to improve its production coding agents without touching a single model weight. Michele Catasta, Replit’s president of AI, described the pair in a post on X on July 6, 2026. The systems target a constraint that most agent builders now share: the underlying model is closed.
When a team runs its agent on a closed frontier model, it cannot fine-tune the weights. Continual learning has to happen somewhere else: in how the agent’s scaffolding, the harness, directs the model’s behavior, or in what information gets fed into the context window at run time. Catasta’s post frames this as the operating reality for most production agents today, not an edge case.
ViBench is Replit’s evaluation system for functional app-building success measured against natural-language specifications. Rather than scoring an agent on whether code compiles or passes unit tests, ViBench checks whether the resulting application actually does what the plain-English spec asked for. That distinction matters for a coding agent product, since a user’s prompt is the spec, and the gap between code that runs and an app that works is where most agent failures live.
Telescope handles the other half of the loop. It automatically clusters production failure traces, meaning real instances where the agent’s output did not satisfy a user, into actionable issue groups. Instead of an engineer manually triaging individual failure logs one by one, Telescope groups similar failures together so the team can prioritize fixes by frequency and impact.
Together, the two systems form a substitute for the retraining loop that open-weight teams can run. A lab with its own weights can retrain directly on failure data. A team building on a closed model has to find the failure pattern first (Telescope), decide whether the fix belongs in the harness, the prompt, or the retrieved context, and then verify the fix against a standing benchmark such as ViBench before shipping.
Catasta’s post does not disclose ViBench’s scoring methodology, how many task categories it covers, or what volume of failure traces Telescope processes. It also does not name a specific fix that came out of a Telescope-identified cluster. Those gaps make it hard to judge how mature either system is relative to eval and observability tools already sold by third parties to agent builders.
The approach is still a useful marker for where agent engineering is heading. As more companies build products on top of closed frontier models rather than their own, differentiation shifts from model quality to the surrounding infrastructure: how well a team can measure functional success and convert failure data into harness changes. Weight access was the old lever for continual improvement. Eval plus failure clustering is becoming the new one.
Teams building coding or app-generation agents on closed models should treat failure-trace clustering as near-term infrastructure to build, not a nice-to-have, since it is the main lever left once retraining is off the table.
Based on a post by Replit’s Michele Catasta on X, July 6, 2026.
