Perplexity has published Q2D-Web, a retrieval benchmark and public leaderboard aimed at the embedding models that power agentic search. According to Perplexity’s own technical blog post, the corpus behind it draws on nine months of the company’s production search traffic, filtered for personal information before release. Outside developers can submit a Hugging Face model for evaluation, but the queries, the documents, and the scoring pipeline all stay under Perplexity’s control.
The mechanics matter for anyone deciding whether to trust the result. Q2D-Web pairs 69,721 agent-reformulated queries, phrased across ten languages, with a document pool built by pulling the top 5,000 results per query from Perplexity’s retrieval system and deduplicating near-identical pages. That construction method, not random web sampling, is what generates the “hard negatives”: pages that match the language or the subject of a query while missing the exact date, entity or quantity it actually calls for. A retriever that cannot make that distinction loses points here even if it would look fine on a looser test.
Relevance itself is not settled by one source. Perplexity built three separate judgment sets: which documents an agent actually cited, which documents its internal ranking stack surfaced, and a combined set topped up with judgments from its own DeepSeek-V4-Flash model. The company argues this spreads out labeling bias rather than trusting a single pipeline. It is also an admission that no single signal, including citation, is a clean stand-in for relevance at this scale.
The caution that belongs on every line of this benchmark is straightforward: the corpus is private, the leaderboard is Perplexity’s to run, and any number attached to a Perplexity model was measured by Perplexity on Perplexity’s own data. The company acknowledges as much, noting that its models “may benefit from an in-distribution advantage” even though the specific queries and documents used for evaluation were withheld from training. That disclosure is honest. It does not change who controls the test.
The results themselves show why a single benchmark number is a weak proxy for “best retriever.” On the Combined relevance set, Perplexity’s own pplx-embed-v1-4b leads Recall@1000 at 69.11, edging Nemotron-3-Embed-8B’s 68.58. Switch to Combined nDCG@10, a metric about ranking quality rather than raw coverage, and Nemotron-3-Embed-8B moves ahead at 47.44 against pplx-embed-v1-4b’s 45.84. No model tested topped every judgment set and every metric, which is the strongest evidence in the post that retrieval quality is multidimensional rather than a single leaderboard slot.
Cost economics explain why Perplexity built a shrunk version of the test at all. A full run against the complete document pool costs thousands of GPU-hours even for a mid-sized model, so submissions are first scored on a subsampled corpus built with reciprocal rank fusion, a method the company says preserves model rankings better than pulling documents at random. Only entrants that place in the top ten of their parameter-count bracket on that smaller test earn a run against the full corpus.
For teams choosing an embedding model for retrieval-augmented systems, the practical read is to treat Q2D-Web as one directional signal rather than a verdict, and to weight the Citation-only results more heavily than Combined, since Citation reflects what an agent chose to act on rather than what a labeling model decided was probably relevant. Anyone benchmarking a candidate model for production should still run it against their own query distribution before trusting a rank built on someone else’s search logs.
Perplexity Engineering detailed the benchmark’s construction and results in a September 9, 2026 post on the Perplexity blog.
