Meta now runs AI-written tests against its riskiest code changes every night, and says the tests helped catch 74 serious privacy bugs before they reached users between January and June 2026. The company calls the technique Catching Just-in-Time Tests, or JiTTests, and laid out the production results on its Engineering at Meta blog on 29 October 2026.
Meta also reports that the rate of serious privacy incidents fell by 20% across those months. It credits the tests with a “key role” in that drop, which is not the same as sole credit, and the post does not say what the rate would have been without them.
Most AI bug-hunters work like a fast human reviewer: they read a change and give an opinion. JiTTests produce something different, a test that actually runs and fails on the new code, which shows a concrete difference in behavior. Meta concedes this takes more computation than a review pass. Its argument is that a failing test catches kinds of bugs a reviewer, human or machine, can read straight past.
The system runs on spare capacity overnight. It picks code changes submitted the previous day that a risk score, trained on past changes, flags as dangerous. Two generators then work on each one. The first, which Meta calls “dodgy diff,” assumes the change is a planted bug and writes tests that tell the old behavior from the new, without caring what the author meant. The second asks a model to work out the change’s purpose from the code, title, and summary, list the ways an implementation could go wrong, and write a test for each imagined bug. Any test that fails against the real change is kept as a “weak catch.”
Weak catches are only suspects. To sort real bugs from false alarms, Meta has a panel of several language models read the test, the failure output, the change itself, and the inferred intent, and vote. Model verdicts shift from run to run and are hard to verify, so Meta adds a rule-based checker named RubFake whose judgments are deterministic. The two scores combine into a ranking, and only the highest-ranked catches go to a person.
That last step is where Meta pushes hardest. It describes the setup as “human-on-the-loop”: engineers set the criteria and look at aggregate outcomes instead of approving each item. In practice, the engineer who gets pinged mostly confirms whether a behavior change was intended, without reading test code.
Engagement numbers suggest the pings are worth opening. Meta says 589 engineers, 64% of those contacted, have engaged with the results, and that 60% of the findings were true positives. Read the other way, roughly four in ten alerts still wasted someone’s attention. Those figures come from Meta’s own engineers on Meta’s own code, and the post offers no outside comparison or independent audit.
The timing matters. Coding agents let teams ship changes faster than reviewers can read them, so review is where the bottleneck forms. A system that turns a suspicion into a failing test, and screens most of the noise before a human sees it, is one answer to that. Meta also says the paper behind the work was presented at FSE 2026, a software engineering research conference.
Any team buying or building AI review now has a reference point: 60% of flagged findings real, on a codebase Meta puts at hundreds of millions of lines. A tool that cannot show its false-positive rate against that number has not yet earned a place in the review queue.
Reported by Engineering at Meta on 29 October 2026.
