Microsoft has laid out the internal engineering harness it uses to keep AI agents from breaking once they leave a demo and reach live users. Microsoft Foundry, which Microsoft describes as its platform for running production AI agents, already counts more than 80,000 enterprise customers, and Microsoft 365 Copilot serves more than 20 million users on its own. The account comes from Marco Casalaina, Microsoft Core AI’s VP of Products, in an interview published by ByteByteGo. What Casalaina describes is not a model upgrade. It is the plumbing that decides whether an agent that passed every test case keeps working after a thousand unscripted conversations.
Casalaina’s central claim is that the model matters less than what surrounds it. When Anthropic shipped Claude Opus 4.8, the engineers behind GitHub Copilot CLI spent real time retuning their harness and rerunning evaluations before the new model could go live inside the product. Models are not interchangeable the way a database version bump is. Each one behaves differently enough that the system wrapped around it, not just the prompt, has to adapt.
The clearest architectural idea in the harness is turning retrieval into an agent of its own. Classic retrieval-augmented generation embeds a question, searches one index, and hands back the top results, with no path to recovery if that first pass comes up empty. Foundry IQ instead plans a query, tries a source, checks the result against the original question, and tries a different source if needed. When retrieval keeps failing, it hands back a plain “I don’t know” instead of a confident guess the model invents to fill the gap.
The same loop applies to tools, not just documents. Rather than listing every available tool in the system prompt, an agent fetches a tool only when a specific step actually calls for it. Microsoft describes this as a pattern it converged on independently, alongside OpenAI and Anthropic. That convergence suggests tool search is becoming a default architecture for any agent carrying more than a handful of capabilities, not a Microsoft-specific trick.
Agents that take real actions (sending an email, booking a table, editing a shared document) get their own identity inside Entra, Microsoft’s enterprise directory, with role assignments and an audit trail separate from any human user they act for. Guardrails move to the tool boundary as well, screening what a tool receives and sends back instead of only checking the model’s prompt and reply. That matters because a hidden instruction embedded in a retrieved document can hijack an agent that never sees the user’s actual request, the mechanism behind indirect prompt injection.
Generic evaluation metrics such as groundedness and task completion only prove an agent produced a technically valid output. A booking agent that confirms a reservation without first checking real availability passes those metrics and still fails the customer. Microsoft’s fix is rubric-based evaluation: specific yes-or-no checks tied to behaviors the team defines, scored continuously against live traffic. The results feed into the Agent Optimizer, which rewrites prompts, swaps the underlying model, or reprioritizes sources when a rubric starts failing, then promotes whichever candidate scores best.
Publishing this much architectural detail also functions as a competitive argument. Microsoft is betting enterprises will pay for Foundry’s identity, retrieval, and evaluation layers rather than rebuild the same primitives themselves on a bare model API, the same wager OpenAI and Anthropic are making with their own agent platforms. The harness, not the model, is where that competition will actually be won.
Teams shipping agents without a distinct retrieval loop, a first-class agent identity, or rubric-based evaluation tied to an automated improvement step should treat this account as an audit checklist. It is not just a rundown of Foundry’s feature list.
ByteByteGo reported this account of Microsoft’s production agent architecture on July 14, 2026.
