Consensus Beats Claude and ChatGPT in Academic Search

Summary

We compared Consensus Pro against two frontier general-purpose models with web search — GPT-5.5 and Opus 4.8 — on the core task of literature review: finding relevant papers. 100 real, randomly-selected, anonymized user queries, judged blind by four independent PhD-level researchers and separately by LLM-as-a-judge across frontier models.

Consensus returned the most relevant results of the three. Measuring search relevance within the top 20 results, Consensus beat GPT 5.5 by 4.5% and Opus 4.8 by 13.2% respectively.

It also did this an order of magnitude faster: a median of 8.7 seconds end to end, against 74.8 for GPT-5.5 and 102.8 for Opus 4.8. Consensus returned better results 10x faster than the frontier models. 

The takeaway is not that frontier models are bad at research. They are extremely good for systems never built for it. It is that brute-force compute is no substitute for a system designed around the task. An agent purpose-built for literature search wins on quality, and wins on speed by a factor of ten.

1. Introduction: Why This Evaluation?

A year ago, the competitive benchmark for academic search was Google Scholar. Today a large and growing share of literature review starts somewhere else entirely: a general-purpose chat model with web access. Researchers, students, and professionals paste a question into ChatGPT or Claude and ask for the relevant papers.

That behavior is rational. These models are extraordinary reasoners and they are already open in another tab. But they are general-purpose systems solving a specialized retrieval problem with the tools they happen to have: a consumer web index and a very large budget of inference-time compute. When a general model needs better research results, its main lever is to try harder: more queries, more iterations, more tokens.

We wanted to know whether that works. Specifically: can a system designed for scientific search beat a frontier model that is willing to spend ten times the compute and time on the same question?

2. Methodology

We compared Consensus Pro — our agentic search mode, which decomposes a query, runs multiple retrievals against our corpus of 220M+ papers (many of which we have full-text access for) and re-ranks — against GPT-5.5 and Opus 4.8, each with web search enabled and each given the same query.

Queries: 100 randomly sampled queries from actual Consensus users, anonymized and scrubbed of all PII. As in our Google Scholar evaluation, these span simple keyword lookups, metadata queries, and full natural-language research questions.

Results: For each query we collected the top 20 results returned by each system.

Relevance judging: Every (query, paper) pair was scored on a 0–4 scale (0 = irrelevant, 4 = perfectly relevant) by four independent PhD-level researchers. We then ran the same judgments through LLM-as-a-judge across multiple frontier models as a second, independent signal. Reported precision is the share of returned results scored 3 or 4 — papers a researcher would consider genuinely relevant to the query.

Latency: We measured end-to-end wall-clock time from query submission to a complete set of results, and report median, mean, and p75.

Bias control: Human labelers had no affiliation with Consensus and were not shown which system produced which papers. They received a set of query–paper pairs, paper metadata, a set of instructions, and nothing more.

Important notes: Scores reflect mainly textual relevance. A perfectly aligned but obscure 1985 paper can still score a 4. Judgments were made on title and abstract, due to full-text access restrictions of some of the systems.

3. Results

3.1 Precision

Consensus Pro leads at both depths. At 10 results it is 1.8% better than GPT-5.5 (1.4 net points) and 11.6% better than Opus 4.8 (8.2 net points). At 20 results the margins grow: 4.5% better than GPT-5.5 (3.0 net points) and 13.2% better than Opus 4.8 (8.2 net points).

The direction of that change matters more than its size. Every system degrades as you go deeper into a results list, that is expected, and it validates the methodology. But the general models degrade faster. GPT-5.5 loses 10.3 points of precision between rank 10 and rank 20; Consensus loses 8.7. A frontier model with a web index can find a handful of good papers on almost any topic. Sustaining relevance across a full page of results, is a harder problem.

When looking at the query-by-query results, the picture becomes even more staggering. Across the 100 query sample, Consensus had the most relevant results across all three platforms on 59 of them. GPT won on 22 queries, Claude on 6, with 11 ties.

3.2 Latency

This is where the gap stops being incremental. At the median, Consensus Pro is 8.6× faster than GPT-5.5 and 11.8× faster than Opus 4.8. Even at p75, our slow queries against their fast ones, we are 3.8× and 5.1× faster.

The distributions differ in shape too. Our median (8.7 s) sits well below our mean (16.3 s): most queries resolve fast, a tail of hard ones take longer. The general models barely spread. GPT-5.5 runs 74.8 s at the median and 81.8 s at p75. They spend the same time on an easy metadata lookup as on a complex synthesis question, because the loop runs until the loop is done, not until the answer is found.

3.3 The Compute Argument

If more inference-time compute reliably bought better retrieval, the frontier models should be winning this evaluation. They are running many more iterations, issuing many more queries, and burning a great deal more time and cost per result than we are. 

The reason is that most of what makes scientific search hard is not reasoning. It is corpus coverage, metadata quality, and ranking. Knowing that a paper exists, knowing what kind of study it is, and knowing which of 4,000 candidate papers to surface first. Those are things you build into a system. You cannot iterate your way to a paper that isn't in your index, and no amount of re-querying fixes a ranker that was never designed for scientific relevance.

When looking at relevance precision compared against latency, the full story of this evaluation becomes extremely clear: Consensus returned more relevant results than the frontier systems in a fraction of the time.

A caveat on cost: we measured time, not tokens or dollars.

4. Conclusion and Discussion

Frontier models are now a real front door to research literature, and they are better at it than most people expected. That is worth saying plainly.

But this evaluation shows that a purpose-built system still wins on the core task, and wins it 10× faster. Consensus Pro returned more relevant papers at every depth we measured, held relevance better across the full results list, and returned in single-digit seconds where the general models took one to two minutes.

The broader point is the same one we made about Google Scholar, arriving from the opposite direction. Search quality is not dictated by legacy and scale, and it is not dictated by raw compute either. It comes from building for the task. We intend to keep widening that gap.