How Search Decides What You See

Years ago, in library school, I was taught information retrieval for maybe five minutes in a course called Information Searching & Sources. TF-IDF got those five minutes and a formula. Vector search (Vector Space Model) got one slide near the end. I remember someone in class asking the lecturer to explain it because they didn’t understand. The response was basically: don’t worry about it, you won’t be examined on it.

I remember that moment because it captured how unimportant the whole thing seemed. And, to be fair, for a long time it was. I went on to spend years as an academic librarian, teaching classes, studying discovery systems and blogging about them, and nothing in the job really required me to understand the retrieval mechanics under the hood.

Librarians are genuinely expert searchers. We write nested Boolean strategies that run to dozens of lines and still hold their logic. We know that the same concept may be described differently across databases, and that in some of them you also need to know the right controlled vocabulary term. We know which field codes exist, which platform drops a truncation operator without saying so, and which one changes its behaviour inside a proximity operator. We know how to rescue a search returning nothing and how to explain to a researcher why theirs returned forty thousand things. There are many ways to be expert in search. This is one of them. The one the profession teaches and practises and this book assumes you have it.

What it does not include is any account of what the machine is doing at the moment it decides.

For most of the period in which our search training took shape, it did not have to. Academic databases behaved alike underneath: words as admission rules, and a ranking assembled from how often a term appears and how rare it is. When every engine works the same way, knowing the engine buys you nothing and knowing the syntax buys you everything. Attention went where it paid, and that was the right call — not a failure of the curriculum but a correct reading of the world it was preparing people for. I argued this at more length in “‘We’re Good at Search’… Just Not the Kind That the AI Era Demands,” which ended by calling for real information-retrieval competence in the profession. This book is my attempt to answer my own provocation.

That old division of labour made sense for a long time. I’m not sure it does anymore.

Two products can both claim to offer “semantic search” while doing quite different things under the hood. One might be BM25 with query expansion. Another might use dense-vector retrieval, learnt sparse retrieval, lexical–dense fusion, or a neural reranker over a lexical shortlist. Those are not implementation details in the trivial sense. They affect how you should enter your search query, which papers even make it into the candidate set and which ones rise high enough for the user to see.

Since pretrained Transformer models started entering text ranking around 2019, the field has also been moving unusually quickly. It still feels a little like the Wild West. There is no single neural architecture that has simply replaced lexical retrieval.¹ 

Instead, we have an expanding pile of overlapping labels from both the research literature and vendors, including neural search, vector search, embedding search, agentic search, semantic search, and the even less helpful vendor favourite, AI-powered search.

The problem is that these terms are often used as if they were interchangeable when they may be describing completely different things: a representation, a matching method, a stage in the retrieval pipeline, or simply a capability someone wants to market. Appendix D, A librarian’s map of search terminology,  tries to sort some of this out.

In other words, the words on the outside of the box no longer tell you much about what is inside it. That matters for libraries. We are used to evaluating discovery products in terms of coverage, interface, support and price. Those questions still matter, of course. But they can now stop short of something rather important: how the system actually decides what comes back.

This does not mean librarians need to become retrieval engineers. But I do think somebody in the library needs to understand enough of the whole stack to ask sensible questions, compare systems, and work out which layer a problem actually belongs to. This is the case I made in “AI Search Is Rebundling Everything Libraries Know About Discovery.” Chapter 12 sets out what that takes in practice.

I have spent the years since 2022 learning this the slow way: introductory retrieval courses taught online, engineering blogs from the companies that build search infrastructure, conference talks, vendor documentation, and cutting edge peer-reviewed retrieval papers on the latest Transformer based retrieval techniques.

It is a steep climb, and not because the individual ideas are necessarily that difficult. The material is scattered across communities that rarely cite one another, much of it is pitched at readers with a mathematics or computer-science background, and almost none of it is written with a librarian’s questions in mind.

And the questions I actually had were usually much more fundamental: How is this different from Boolean or lexical search? How should I enter my search queries? What are the pros and cons of these new approaches? And, ultimately, how does this change which papers my researchers find?

Answering those questions usually meant reading three things that were mostly about something else.

This book is that climb, compressed. It is what I wanted in 2022 and could not find. If it does its job, you will reach a higher vantage point than I did for a fraction of what it cost me to get there.

What this book is, and what it is not#

Because this book is written for librarians, it is not a typical Information Retrieval 101 text. It follows a few deliberate principles.

• Understanding before detailed mechanism. The inverted index is explained in enough detail to show why it is fast and what it structurally can and cannot do, but not enough for you to implement one. You will get a sense of the latency and other trade-offs involved in different indexing and retrieval choices, but building these systems would require further study. Where the fuller machinery matters, I put it in the appendices rather than interrupting the main argument.
• Examples familar to librarians. Wherever possible, I tie a concept or principle to what a named library or academic search system is documented to do, together with the date I checked it. Vendor documentation is thin, uneven and prone to moving, so when the documentation runs out, I say so rather than infer what is happening. This is the part of the book that will age fastest, and it is dated so you can see it ageing.
• Formulas only where they carry an argument. A formula is a kind of compression: useful if you already read formulas fluently, but potentially an obstacle if you do not. A librarian assessing a search system needs to understand what BM25 responds to and what it is blind to, not necessarily how to calculate it by hand. The practical convention I use throughout the book is set out in How to read this, immediately below.

Of course, this is a crash course, not a substitute for a graduate course in information retrieval, and specialists will find simplifications throughout. It is meant to give you a floor to stand on: enough to follow a technical conversation, interrogate a vendor claim, and recognise when one of your own search problems is really a retrieval problem.

Where to go deeper lists what to read for readers instructed to go beyond.

Two different problems sit inside every AI search product:

• Retrieval decides which records exist as far as your question is concerned, and in what order they are placed.
• Generation turns some of those records into prose.

This book is almost entirely about the first.

Four shapes of AI academic search

The diagram sorts current academic tools along two axes. Horizontally: what the tool hands you—a ranked list of records, or a synthesised answer or report. Vertically: how much retrieval it performs—a single pass, or repeated, multi-step searching. That produces four recognisable shapes: quick search, quick answer, iterative search and deep research.²

The important point for this book is that the two axes describe different things. The horizontal axis describes output format. The vertical axis is entirely a retrieval property: how extensively the system retrieves—a single or limited pass, or repeated multi-step searching. Half the design space of AI academic search is therefore retrieval design.

Agency is a separate question: who decides what to search for next? Neither axis records that choice. A tool in the upper half may follow a workflow fixed by an engineer, or may let a model choose each subsequent action from what the previous round returned. That third property—agency—is examined near the end of this book, once there are enough retrieval components for a controller to choose among.

Whatever quadrant a tool occupies, retrieval still comes first. A quick answer is retrieval followed by generation.³ Deep research is many rounds of retrieval followed by generation. Nothing in the right-hand column can discuss a paper that the left-hand machinery never surfaced—the shortlist bottleneck examined later in this book. That is the practical reason to spend a whole book on the less glamorous half. A generated answer can be fluent, correctly formatted and properly cited, and still be wrong about the state of the literature, simply because the search behind it missed something. No amount of prompt engineering or model upgrading reveals what was never retrieved.

So this book does not cover how language models generate text, how to detect a fabricated citation, how to write prompts, or how general-purpose agent architectures and planning algorithms work. It covers what happens before any of that: how text becomes searchable, how a system decides which records are eligible, how it ranks them, and how much of that you can establish from the outside. It does examine one narrow slice of agent behaviour—how an agent selects, repeats and stops retrieval actions—because that slice is retrieval design.

Most librarians understand Boolean searching. Yet three familiar search behaviours raise questions about what the system is actually doing.

Three familiar search results—and three puzzles#

In the first example, a scite paper search combines a meaningful question about the open-access citation advantage with an invented string. Useful papers still appear near the top, and the interface reports millions of results.

In the second, Google Scholar reports about 9.4 million results for open access, yet its interface reaches page 100 and its help documentation says that no more than 1,000 results can be shown for one query.

In the third, an August 2026 Semantic Scholar search exposes a more pointed contradiction. The name Semantic Scholar suggests meaning-aware search and may lead a reader to expect dense retrieval or embedding-based matching, for which a natural-language question should be an appropriate query. Yet Is there an open access citation advantage returned only 13 results—far fewer than the hundreds of relevant papers reasonably expected for this established topic. Removing the conversational framing and searching open access citation advantage returned about 35,300 results, a much more plausible result-set size. Both queries express essentially the same information need. Why did the supposedly more natural query retrieve dramatically less?

Puzzle one: the nonsense term did not empty the result list

Puzzle two: the reported result set is much larger than the viewable list

Puzzle three: a natural-language question retrieved fewer papers than a short keyword query

Natural-language question

Short keyword query

These screenshots establish visible behaviour, not the products’ complete internal architectures. We will return to these examples after building the necessary machinery from the ground up.

Begin with the most explicit case: a search in which every operator is an admission rule.

Strict Boolean search: words as admission rules#

Consider the query:

delulu AND job

A Boolean system requires a record to satisfy both conditions.

Boolean operators create hard decisions:

• AND requires conditions to occur together.
• OR allows alternatives.
• NOT excludes records.

Boolean logic decides whether a record satisfies the query. A record that passes all of the query’s hard conditions is eligible to enter the result set.

This is not a historical curiosity. It is the model behind the advanced-search interfaces librarians use most: PubMed’s Advanced Search with its search-history builder and MeSH terms, Web of Science Advanced Search with its field tags, and the equivalent screens in Scopus, EBSCOhost and Ovid. A search strategy written for a systematic review is a Boolean expression, and it is reported in the published paper precisely because every line of it is an admission rule that can be inspected and rerun.

Boolean logic does not by itself assign relevance scores. Once several records are eligible, Boolean alone supplies no rule for deciding which should appear first.

Boolean decides who gets into the competition. Ranking decides who finishes first.

This distinction matters when query expansion goes wrong.

Suppose an LLM expands delulu into:

delulu OR unrealistic OR irrational OR foolish

If foolish is a poor alternative, records containing it may enter the result set. Ranking may push them down, but Boolean has already admitted them.

A poor synonym expansion changes which records can enter

The reverse error is more serious. If a relevant record fails a compulsory AND condition, ranking cannot rescue it. It never enters the competition.

This is why LLM-generated synonyms require particular care in formal Boolean strategies. A weak suggestion is not merely given a low weight. It changes which documents can be returned.

The admission decision is now clear, but the ordering problem is still open. First we need to see how a search engine finds the eligible records without rereading the entire collection.

How Boolean search uses an inverted index#

Large academic indexes like OpenAlex, Semantic Scholar, Ex Libris CDI span hundreds of millions of records. To answer queries as quickly as possible, Boolean search does not normally reread every record whenever a query arrives.

It instead uses an inverted index: a structure that starts with searchable terms and points to the records containing them. Before examining how Boolean operators use that structure, we need to see how text becomes searchable terms.

Before the index: text becomes tokens#

A computer does not begin with an intuitive understanding of words. It needs a rule for dividing a string of text into units it can process. Tokenisation is that division, and each resulting unit is a token. A token is often a word, but it can also be punctuation, part of a word, a character or another unit chosen by the system.

For a simple English lexical index, the analysis might proceed as follows:

Table 2.1 — Stages of lexical analysis, and what a sample string looks like after each.

There are different ways to do tokenisation. A different analyser might retain about, keep jobs unchanged, treat the exclamation mark as a token or preserve the complete string in a keyword field. Languages without spaces between every word require different boundary rules, while names, apostrophes, hyphens, URLs and identifiers create choices even in English.

Tokenisation is only the first step. Lowercasing, stemming, stop-word removal and synonym or controlled-vocabulary mapping are separate operations. The broader sequence is often called text analysis. Its final output supplies the terms that can enter a lexical index.

At search time, the query is also tokenised and analysed. Lexical matching becomes possible when a resulting query term corresponds to an indexed document term. Query and document analysis need not be identical in every system, but incompatible choices can create missed or unexpected matches.

Text becomes tokens; analysis turns selected tokens into searchable terms.

After tokenisation: stemming and lemmatisation#

Tokenisation decides where one unit ends and another begins. Stemming and lemmatisation are later analysis steps that may transform some of those units before matching. Their practical purpose is to let related grammatical forms share searchable evidence even when the user and the document do not contain exactly the same surface form.

Table 2.2 — Stemming and lemmatisation compared: how each reduces a word, and what to remember about the difference.

These outputs are illustrative rather than guaranteed. Algorithms, dictionaries and languages differ, and a product may use neither method or may apply different analysers to different fields. A title-and-abstract field might reduce jobs to job, while an exact-name, code or identifier field deliberately preserves the original string.

The benefit is usually recall. If both query and document analysis reduce jobs and job to the same term, either form can retrieve the other. The risk is lost precision through over-stemming, when unrelated words collapse to the same term. The reverse problem is under-stemming: related forms remain separate and a potentially useful match is missed. Lemmatisation uses more linguistic information than stemming, but it is not automatically correct, semantic or superior for every collection.

Compatibility matters. If the document index stores job but the query analyser searches only for jobs, the forms may still fail to meet. When forms do converge, the analysed term determines which indexed-term entry is consulted, how occurrences are counted and how many documents contain that term. These choices affect Boolean matching and the collection statistics used by ranked lexical methods introduced later.

Neither technique is the same as user-entered truncation such as librar*, whose expansion depends on database syntax. Nor are they synonym expansion, controlled-vocabulary mapping, semantic query rewriting or dense retrieval. Those operations can connect different expressions or concepts; stemming and lemmatisation primarily normalise grammatical form within a lexical analysis pipeline. For a fuller technical account, see Manning, Raghavan and Schütze, “Stemming and lemmatization”, in Introduction to Information Retrieval.

Whatever analysis choices are made, their final terms become the keys used to build the inverted index.

From analysed tokens to a term-first map#

A collection is naturally organised document first: open a record, then read the words inside it. An inverted index reverses that direction for searching. It starts with an analysed term and points to the documents in which that term occurs.

Imagine four tiny records:

Table 2.3 — A miniature four-record collection, used throughout this chapter.

After the analysis described above, the index maintains a dictionary of searchable terms. Each dictionary term points to a posting list: an ordered list of entries identifying the documents that contain it. A selected part of the dictionary for these records might look like this:

Table 2.4 — The same four records as an inverted index: each term paired with its posting list.

A posting is one entry in such a list. Real postings normally store more than a document identifier. They may also record how often the term occurs, which field contains it and the positions at which it appears. Those additions support later relevance scoring, field weighting, phrase searching, proximity scoring and result highlighting.

The earlier analysis choices determine which dictionary term and posting list a query consults. They can therefore change which records match even before any ranking method is applied.

An inverted index is best understood as a term-to-document map built from analysed text.

Boolean treats postings as sets#

For the query:

delulu AND interview

the engine does not need to reread all four records. It looks up two posting lists and intersects their ordered document identifiers:

delulu   → D1, D4
interview→ D1, D2

intersection → D1

OR combines lists, while NOT removes documents found in an excluded list. At this level, Boolean retrieval is largely set processing over postings.

Why inverted indexes can be so fast#

The first advantage is selective access. The engine goes directly from each analysed query term to its posting list rather than scanning every document. Rare terms are especially selective because their lists are short.

The second advantage is ordered access. Posting lists are normally sorted by document identifier. For a Boolean AND query, this makes it quick to find identifiers that appear in more than one list: the engine can work forwards through the lists instead of repeatedly searching earlier entries. A worked two-list example appears in the advanced appendix.

The inverted index gets us quickly to the eligible set. But if several records pass, the posting lists do not inherently say which one deserves first place. What is missing is a graded relevance score.

BM25: words as weighted clues#

Most academic databases allow a relevance sort. Scopus and Web of Science, for example, let a searcher choose relevance rather than date or citation-based ordering. The label tells us that records will be ordered, but not how.

A common academic-search pattern has two controls. A Boolean expression first defines the eligible set; a separate scoring method then orders the records that survive. Strict Boolean logic can decide who passes, but it cannot distinguish among records that all pass.

The ranking stage need not be plain BM25. Production systems may combine lexical scores with field weights, phrase and proximity boosts, citations, recency, resource type or peer-review status, and may rerank a shortlist later. “Relevance” or “Best Match” therefore names an ordering, not a formula.

BM25 is one standard lexical answer to the ranking problem: it turns matching terms into unequal, graded evidence. Chapter 4 will show another valid architecture—BM25 retrieving and ranking candidates directly, without a strict Boolean AND gate.

An introduction to BM25#

BM25 is a lexical scoring function that ranks documents by combining weighted evidence from matching analysed terms. It is one of the standard workhorses of lexical retrieval, remains a common baseline against which newer retrieval methods are compared, and is the default text-scoring model in Apache Lucene and major Lucene-based search engines such as Elasticsearch, Apache Solr and OpenSearch.⁴

Librarians may be more familiar with the term TF-IDF. Its central intuition is straightforward: a matching word provides evidence that a document is relevant, but a relatively uncommon word usually tells us more than one appearing throughout the collection.

BM25 follows the same practical intuition that matching terms should contribute different amounts of evidence, but it is not simply a revision of the canonical TF-IDF formula and instead comes from the probabilistic relevance tradition.⁵

Still ultimately similar to TF-IDF, it depends on the query and document sharing indexed terms. Unlike strict Boolean, however, it combines those matches into a graded score.

In simplified form, BM25 adds a contribution for each matching query term. That contribution answers four questions about the term. The first two are the familiar TF-IDF intuition; the last two are what BM25 adds to it.

1. Did the term match at all? A term that does not appear in the document contributes nothing.
2. How rare is the term in the collection? A term found in relatively few documents usually contributes more (higher score) than one found almost everywhere. This is inverse document frequency.⁶
3. How often does it occur in the document, with diminishing returns? Repeated occurrences provide more evidence that the document concerns the term, but each additional occurrence adds progressively less evidence. Seeing delulu twice may be more informative than seeing it once. Seeing delulu 100 times does not make a document 100 times more relevant. But how much should it be? In BM25 this term saturation is handled by a parameter conventionally written k₁.⁷
4. Is that frequency unusually high given the document’s length? Five occurrences in a short abstract may be stronger evidence (higher score) than five occurrences in a very long document. This is document-length normalisation and, like term saturation, the strength of this normalisation is controlled by a second parameter, conventionally written b.

Questions 1 and 2 are shared with TF-IDF. Questions 3 and 4 are BM25’s substantive additions, and each is governed by one of those tuning parameters. Typical defaults are k₁ = 1.2 and b = 0.75, the values used in the worked example in the appendix. A librarian inspecting a search engine’s ranking configuration will meet these two letters.

The four questions BM25 asks about each matching term

Try it: the formula is easier to understand when you can interfere with it. The BM25 Evidence Lab lets you change one source of evidence at a time — term rarity, repetition, document length — and watch a six-record ranking respond. It takes about five minutes and needs no prior knowledge of the equation.

This description uses the standard single-field form of BM25 as a simplifying reference; related variants adapt it to structured records and very long documents.⁸

BM25 and lexical ranking in academic search#

BM25 and lexical ranking in academic search

PubMed offers an unusually clear documented BM25 example. Its published multi-stage architecture for Best Match first retrieves and orders matching records with BM25, then uses LambdaMART⁹ to rerank the top 500. PubMed’s current help still describes a weighted term-frequency stage followed by machine-learning reranking. It therefore shows both BM25’s continuing role and why a production ranking pipeline should not be reduced to one formula.

Two product descriptions show how lexical evidence is usually embedded in richer ranking. Scopus describes term frequency and inverse document frequency alongside field, position and proximity signals. Primo’s Central Discovery Index combines field and rare-term weighting, term frequency, field length, phrase and proximity boosts with static signals such as resource type, recency, peer-review status and citations.

One discovery index documents its ranking signals

Only the PubMed example identifies BM25 directly. The Scopus and Primo descriptions establish that lexical evidence is active in current academic search, but not that either user-facing ranking is plain BM25.¹⁰

How BM25 uses the same inverted index#

The earlier Boolean explanation introduced the dictionary and posting lists as a term-to-document map. BM25 can use the same structure. Boolean operators treat postings mainly as sets; BM25 uses term frequency, document frequency and document length to calculate graded scores.

A common two-stage pattern: Boolean eligibility, then BM25 ranking

In this two-stage arrangement, query processing has five conceptual steps:

1. Look up each analysed query term. The dictionary points to the corresponding posting lists.
2. Gather the eligible records. Boolean operations over those posting lists determine which records satisfy the query.
3. Obtain the scoring evidence. Postings can supply within-document term frequency, the dictionary supplies document frequency, and document metadata supplies length.
4. Add the contributions. BM25 combines the evidence for each matching query term into a document score.
5. Choose the output boundary. The engine may completely order the scored candidates or retain only a top-k.

The Boolean admission rule, BM25 scoring rule and top-k output boundary remain separate controls, even when a product executes them in one request. In direct BM25 retrieval, the second step changes: the engine gathers the union of records matching any analysed query term and scores them without first enforcing a strict Boolean AND.

Why the same index remains fast for ranked retrieval#

Selective access still matters. The Boolean query obtains its eligible records from posting lists rather than scanning every document, and BM25 can read the scoring evidence stored with those same postings. Ordered postings also help the engine bring together the score contributions belonging to the same document while moving forwards through the lists.

Production indexes add further engineering to store posting lists compactly and avoid unnecessary work. Compression and blocks reduce storage and memory traffic, while score upper bounds and safe pruning methods such as MaxScore, WAND and Block-Max WAND can avoid scoring candidates that cannot enter the leading results. These techniques change how efficiently the same term-to-document map is stored and traversed; they do not turn BM25 into semantic retrieval. The advanced appendix provides a complete worked scoring example and explains these techniques in more detail. The classic Introduction to Information Retrieval provides another introduction to dictionaries, postings and basic query processing.

As a broad practical rule, lexical retrieval over an inverted index is generally fast and often less computationally demanding than dense retrieval at a similar scale. That is not a universal benchmark result: hardware, index design, query mix and accuracy settings all matter. The different candidate-generation machinery used for dense vectors is explained in the later dense-retrieval section, after vectors and embeddings have been introduced.

Key distinction

Lexical search does not have to mean Boolean search#

Lexical retrieval derives evidence from relationships between analysed query terms and indexed document terms. Boolean retrieval and BM25 both belong to this larger family, but they use the evidence differently.

In many academic databases, a structured Boolean expression defines the eligible set and a separate relevance ranker orders the survivors. BM25 or a richer lexical pipeline may contribute to that order. This common arrangement is not universal: a keyword or natural-language box may instead use minimum-match rules or direct any-match ranking.

Suppose the analysed query is:

apple orange banana

A familiar academic-search interpretation is:

apple AND orange AND banana

Under that model, implied AND admits only records containing every term and a ranking algorithm orders that Boolean set. But seeing a relevance-ranked list does not prove that every query term was compulsory. The same vocabulary, analysis pipeline and inverted index support at least three execution rules:

1. Implied AND, then ranking: all three terms are compulsory. A record missing banana is excluded before the ranker sees it.
2. Minimum match or soft conjunction: a configured number or proportion of the terms must match.
3. Any-match ranked retrieval: one matching term can make a record a candidate, while every additional match can add weighted evidence. BM25 commonly operates this way.¹¹

All three are lexical. In the third, BM25 does not wait for a separate Boolean AND gate: it gathers candidates from the union of the query-term posting lists and ranks them by accumulated evidence.

Same analysed terms. Same inverted index. Different rules for admission and scoring.

Three admission rules over one inverted index

An admission rule, a ranking function and a top-k output boundary therefore solve different problems: who may compete, how candidates are ordered, and how many leaders are retained. A production system can control all three independently.

A bad term: exclusion under AND, distortion under BM25#

Consider this query:

delulu job interview expectations foolish

Under implied AND, foolish is compulsory. Otherwise relevant records lacking it are excluded before ranking, and no later ranker can recover them. Under any-match BM25, those records can still compete—but the bad term can distort their order.

BM25 does not know that foolish is a poor substitute for delulu. If it is rare, inverse document frequency gives it more influence, so irrelevant records containing it may rise. The bad term stops deciding who competes and starts distorting who wins. The Evidence Lab demonstrates the effect.

Rare query terms contribute stronger lexical evidence through IDF

Term-frequency saturation cannot identify a weak query term; it only limits the benefit of repeating that term inside a document. BM25 is lexical evidence made less binary, not semantic understanding.

What might have happened to a missing query word#

When a returned result lacks one of the words typed into the search box, several explanations remain possible:

• Removed during analysis: the system discarded a stop word or punctuation before retrieval.
• Removed or changed during rewriting: an upstream component shortened, expanded or reformulated the query before retrieval began. A few paragraphs below examines this case, which is the only one of the four that changes the query itself.
• Retained but optional: matching the term could improve a score, but was not required.
• Retained but unmatched: the term stayed in the processed query but occurred in no posting list, so it contributed nothing.

These behaviours are not mutually exclusive. A system may analyse the query, require only a subset of the surviving terms, and then rank the qualifying documents using weighted word matches. None of that behaviour, by itself, identifies the complete retrieval architecture.

Early Google: a useful counterexample#

The words sent to retrieval may not be the words you typed#

One explanation in that list deserves more than a passing mention, because it is the only one that changes the query itself. Everything so far has assumed that the words reaching the index are the words the searcher entered. Many systems first interpret what was typed and may then construct a different internal query, better suited to the index behind them.

Four things are now worth keeping apart, all of which this book has already introduced.

• Query analysis covers the routine lexical steps described earlier—tokenisation, normalisation, stemming and lemmatisation, and stop-word handling—which prepare strings for matching and do not by themselves imply AI or semantic understanding.
• Query execution rules decide how the resulting clauses are applied: required, optional, or governed by a minimum-match threshold.
• Query understanding interprets the request—for example by recognising a phrase, entity, date, intent or proposed constraint. It can inform a later decision without changing the retrieval input at all.
• Query transformation, also called query rewriting, changes or creates the retrieval input: it may correct, map, expand, restructure or split what the searcher entered.

The last two are related, but they are not synonyms. Understanding may cause a transformation or a retrieval decision; transformation is the observable change to what a retrieval pass receives. Query transformation is not a kind of retrieval. It hands a changed input to whatever retrieval method sits behind it, and that method may be exactly the Boolean and BM25 machinery already described. Correction, controlled-vocabulary mapping and synonym expansion are longstanding transformations; newer systems may use neural models or large language models. Neither fact identifies the engine receiving the result.

A contemporary interface may accept a natural-language question, identify its dates or proposed filters, and use a parser or language model to construct a Boolean expression. Natural-language input therefore does not imply a non-lexical engine: the transformed query may still go to Boolean admission and BM25 ranking.

This closes the list offered a few paragraphs ago. A typed word can disappear because analysis removed it, execution made it optional, no posting matched it, or transformation rewrote the query. Understanding alone does not remove the word; it matters when its interpretation causes one of those later changes. Chapter 8 returns to these operations with multi-query, routed and feedback-based retrieval available.

Returning to the first two opening puzzles#

Scite: the invented string in the opening screenshot was not compulsory. It may have been removed, retained as optional evidence or retained without matching a posting. The screen cannot distinguish those explanations or identify BM25, dense retrieval or any other architecture.¹⁴

Google Scholar: the reported hit count, relevance ordering and viewable set are different properties. Google Scholar reports millions of matches but documents a display limit of 1,000 results. That is an observable output boundary, but it does not reveal the scoring signals, whether every match received a final score, or whether the displayed records are a global top-k.

Semantic Scholar remains unresolved: the August 2026 result counts tell us what happened, not where meaning entered the pipeline. We do not yet have the distinctions needed to ask whether retrieval itself was lexical, retrieval itself used dense representations, semantic processing occurred only after retrieval, or several stages were combined. The third puzzle therefore has to wait for Part II.

A search system can therefore be non-Boolean without being non-lexical or semantic.

One category error resolved—and two distinctions still ahead#

The first category error is now resolved: lexical does not mean Boolean. Boolean admission, minimum-match thresholds and graded ranking are different ways of using word-based evidence.

Two further distinctions become important as we turn to neural retrieval. First, semantic search does not mean dense retrieval: semantic names a goal, while dense retrieval names one technical route towards it. Second, vector does not automatically mean dense, learnt or semantic. BM25 scores can be represented with sparse, vocabulary-aligned vectors, while contemporary bi-encoders usually produce learnt dense vectors. These distinctions are related, but they are not the same claim.

The remaining problem is vocabulary mismatch: a paper about unrealistic expectations may stay out of reach of a search for delulu. A system can bridge that gap in two places:

• Change the query, keep the index. Correct, expand or map the query, then match terms as before.
• Change the representation. Encode queries and documents so that texts expressing the same idea are placed near one another whether or not they share any words, and retrieve by proximity rather than by matching.

The next chapters examine the second route while keeping the first in view.

From Word2Vec to retrieval embeddings#

Before comparing kinds of vectors, it helps to separate three related terms. A representation is any form a system uses to stand for an item. In this chapter, the representations of interest are usually vectors: ordered lists of numbers. In a simple diagram, two numbers could give an object horizontal and vertical coordinates. Machine-learning vectors usually have hundreds or thousands of dimensions, so we cannot draw them directly, but the same idea remains.

An embedding is a mapping that places an item—such as a word, query or passage—into a vector space. In modern NLP, that mapping is usually learnt from data during model training. By common shorthand, the resulting vector is also called an embedding. Items placed near one another are treated as similar according to whatever patterns the model learnt.

Word2Vec provides a useful first example. It learns one vector for each word from the contexts in which words occur. Words used in similar contexts tend to receive nearby vectors, even when they are different strings. This made it possible to perform useful comparisons with coordinates rather than relying only on exact word matches.

The individual dimensions of a Word2Vec vector are not normally labelled occupation, emotion or age. The learnt pattern is distributed across many dimensions, most of whose values are non-zero. That is why it is called a dense word embedding.

Word2Vec is only a stepping stone to modern dense retrieval. Classic Word2Vec gives a word essentially the same stored vector wherever it appears and represents individual words rather than complete information needs. Modern encoders can instead read a whole query or passage in context and be adapted, through pooling and task training, to produce one vector for that complete text unit.

Context changes the representation#

Consider bank in the river bank and in the bank refused the loan. Classic Word2Vec holds one stored vector for that string, so both uses begin from the same location and nothing in the model separates them. A contextual encoder instead computes a fresh representation for each occurrence from the sequence it appears in. The two instances of bank receive different vectors because the words around them differ.

These context-dependent vectors are often called contextual embeddings. More precisely, the encoder transforms its initial token embeddings into contextual token representations; obtaining one embedding for a complete query or passage requires a later pooling step.

The architecture that made this practical at scale is the Transformer, whose self-attention mechanism allows each token’s representation to draw on the other tokens in the sequence instead of being fixed in advance. ELMo, published in 2018, produced contextual representations using stacked bidirectional recurrent networks. Transformer-based encoders subsequently became dominant because self-attention made sequence processing more parallelisable and scalable for large-scale pretraining.¹⁵ The appendix on the Transformer sets out what self-attention does and why nearly every model named in this book is built from it.

What BERT is, and what it still needs#

BERT is a Transformer encoder pretrained to learn general language patterns, chiefly by predicting masked tokens from both sides of their context.¹⁶ That produces a general-purpose language model, not a search system.

Turning a pretrained encoder into a classifier, named-entity recogniser or retriever requires task training on labelled examples. Generic BERT is a poor retriever because masked-word prediction gives it no reason to place independently encoded questions near relevant passages. Models such as Sentence-BERT change the architecture and training for that purpose.

Domain pretraining is a separate lever. BioBERT continues pretraining on biomedical literature,¹⁷ BiomedBERT starts from biomedical text and vocabulary,¹⁸ and SciBERT does the same across scientific literature.¹⁹ This changes what the model knows, not what it should rank; each still needs retrieval-oriented training.

“Uses BERT” therefore leaves two decisive questions unanswered: was it trained for retrieval, and where does it sit in the pipeline? The same family can interpret a query, produce first-stage vectors or compare a shortlist as a reranker.²⁰

Contextualisation supplies information a model could use. Task training decides what it becomes good at.

Model tokens are not indexed terms#

Classic Word2Vec uses a fixed word vocabulary, so an unseen complete word may map to an unknown token.²¹ Modern encoders usually divide unfamiliar strings into reusable subwords, characters or bytes. This improves coverage but does not imply understanding: a tokeniser can split a string even when the encoder has learnt little about its use. Appendix B gives the detailed worked example.

The word token is also used in lexical indexing, but the units perform different jobs:

Table 5.1 — Lexical index terms and dense model tokens are produced by different processes and are not interchangeable.

For a conventional single-vector dense retriever, the path is:

text
  → model tokens
  → token IDs
  → initial token embeddings
  → contextual token representations
  → pooling
  → one query or passage embedding

The model tokens are therefore inputs to the encoder. They are not the dimensions of the final pooled vector, and they are not automatically searchable terms in an inverted index.

A contextual encoder combines these input units and pools them into one representation for the complete query or passage. The next section explains the training that makes distances between those representations useful for retrieval.

How a contextual encoder becomes a retrieval encoder#

It is tempting to say that dense embeddings represent “meaning”. That is useful shorthand, but it leaves out the most important question: meaning for what task?

A dense retriever commonly starts with a pretrained language model. Producing one vector for a whole text requires a pooling step that combines the contextual token representations—commonly by averaging them, or by taking the representation computed at the sequence-opening position. Pooling on its own is not enough. Pretraining gives the model broad linguistic patterns, but a model that predicts masked or next words does not automatically arrange its pooled vectors well for search, and two texts can sit close together for reasons that have nothing to do with one answering a search for the other. Retrieval training changes the representation so that labelled relevant texts receive higher similarity scores than competing texts.²²

Two influential examples make that transition concrete. Sentence-BERT (SBERT) adapted BERT so that independently encoded sentences could be compared efficiently. Dense Passage Retrieval (DPR) applied the bi-encoder approach directly to retrieval: a question encoder and a passage encoder are trained so that relevant passages score above competing passages. The important step in both cases is not simply using BERT, but training the independently produced representations for the comparison the system needs to make.

Pretraining gives language patterns; retrieval training teaches what should rank together

Suppose the query is:

Am I being delulu about getting this job?

A training dataset might label this passage as relevant:

How to recognise unrealistic expectations during a job search

It might contrast that passage with an easy negative:

A history of agricultural employment

and a hard negative:

Ten signs that you performed well in your job interview

The hard negative shares the broad topic and several important words, but it does not address whether the searcher’s expectations are unrealistic. Training encourages the model to score the labelled positive above both alternatives. Different systems implement this intuition with objectives such as triplet loss or contrastive softmax, and the selection of negative examples materially affects what the model learns.²³²⁴²⁵

Retrieval training pulls labelled positives closer and pushes negatives away

Where did those labels come from? They might be based on human relevance judgements, clicked results, questions paired with answer passages, titles paired with abstracts, citations, automatically generated queries or neighbouring passages from the same document.

Those sources teach different ideas of relevance. Clicks can reward popularity and ranking position. Question–answer pairs favour answer-bearing passages. Citations may teach scholarly relatedness rather than direct relevance. Titles and abstracts may mainly teach topical similarity.

Where the training data came from matters as much as what it taught. Training pairs are drawn from some particular literature, genre and style of question, and the model learns the notion of relevance that holds there. A retriever trained on web question–answer pairs has learnt what a good answer to a general-knowledge question looks like. That is not the same as what a useful paper looks like to a systematic reviewer, what a relevant record looks like to someone tracing a chemical compound, or what a good match looks like in a language or discipline barely present in the training data. The same architecture, trained on different material, becomes a different retriever.

This can be exploited deliberately. SPECTER trains document-level embeddings of papers on the citation graph, so that papers citing one another are pulled together—a definition of relatedness drawn from scholarly practice rather than from web behaviour.²⁶ It can also be a trap: a model that performs well on the benchmark it was tuned for may transfer poorly to a different collection, query population or retrieval task, which is what the discussion of out-of-distribution transfer examines once the whole pipeline is in view.

“Positive” and “negative” are training labels, not universal truths.²⁷

A retrieval embedding therefore preserves the similarities and differences that its starting model, training pairs, negative examples, loss function and optimisation process taught it to preserve. If exact identifiers rarely matter during training, it may blur them. If negatives rarely contain contradictions or specialised senses, it may fail to preserve precisely those distinctions.

Every retrieval embedding space contains a theory of relevance, whether its users know what that theory is or not.

Single-vector dense retrieval: compress first, compare later#

A conventional dense bi-encoder uses a query encoder and a document or passage encoder. They may share weights or use distinct parameter sets, while normally being trained jointly on the retrieval objective. Dense Passage Retrieval is a well-known example of this dual-encoder pattern.²⁸

Each indexed unit is encoded without seeing the query, so its vector can be calculated in advance and stored. At search time, the system encodes the query and compares it with the stored vectors using a score such as dot product or cosine similarity. The comparison may be exhaustive, or a large collection may use an approximate nearest-neighbour index to reduce the work.²⁹

Like the ranked retrieval introduced in Chapter 3, dense retrieval assigns candidates scores and orders them; what changes is where those scores come from.

Key distinction

Semantic search is a goal, not an architecture#

Semantic describes what the search is trying to achieve: matching by meaning rather than only literal words. It is usually contrasted with lexical search, which draws its evidence from matches between analysed query and document terms. BM25, ontology or query expansion, dense retrieval and neural reranking describe mechanisms a system might combine in trying to achieve the semantic goal.

“Semantic search” is not another name for “dense retrieval”. Dense retrieval is one technical approach to semantic search that is very popular as of the mid-2020s. Natural-language input does not imply dense retrieval either: it tells us what the search box accepts, not how a retrieval stage builds its candidates.

Dense retrieval over vector embeddings is the most familiar contemporary implementation of the semantic goal. This is why semantic search, dense retrieval, vector search and embedding search are often used as if they were synonyms. They are not: semantic names an intended capability, dense describes the shape of a representation, vector names a kind of representation, and embedding names a learnt mapping into a vector space or, by shorthand, the vector it produces.

Those mechanisms do not contribute in the same way. BM25 by itself remains lexical, but it may supply candidates that a meaning-aware model later reranks. An ontology can add controlled concepts, synonyms and relationships to a lexical query. A dense retriever can compare learnt representations without requiring word overlap. A neural reranker can make a richer semantic comparison only after an earlier stage has retrieved a shortlist. A product may use one or several of these arrangements. The word semantic does not tell us which one, or where in the pipeline meaning enters.

What dense retrieval gains—and compresses#

The bi-encoder pipeline: encode independently, then compare vectors

The benefit is vocabulary bridging. A suitably trained model may place:

Am I being delulu about getting this job?

near:

How to recognise unrealistic expectations during a job search

even though the important expressions do not overlap lexically.

Dense retrieval can bridge different phrasings without lexical overlap

The trade-off is compression. Each indexed passage must compress its topic, named entities, claims, qualifications and terminology into one point before it knows which query will arrive. Shorter chunks reduce that pressure, but each chunk is still represented by one pooled vector. Independent encoding therefore creates an information bottleneck as well as making precomputation possible.

The compression bottleneck in single-vector dense retrieval

Broad similarity may consequently dominate a small but decisive detail. A model can recover the general meaning while blurring an exact code, a rare name, a negation or the difference between a question and its opposite.

How nearest-neighbour indexing makes dense retrieval practical#

The earlier lexical sections separated Boolean eligibility from direct BM25 candidate retrieval. In both routes, an inverted index can jump from analysed query terms to matching records without scanning the complete collection.

A dense embedding does not provide an equivalent term-to-document pointer. Most vector dimensions are active, the dimensions do not correspond directly to recognisable words, and a relevant document need not share any query term. There is therefore no posting list for delulu that can directly identify all vectors expressing unrealistic expectations.

The exact baseline is a flat or exhaustive vector search: compare the query vector with every stored document or passage vector. The engine can then construct a complete ranking or select only the leaders. This route is exact, but its work grows with the collection size and vector dimensionality.

Large dense-retrieval systems therefore often use an approximate nearest-neighbour index, or ANN, to reduce the number of comparisons. Rather than examining every vector, an ANN structure searches promising regions or navigates through selected neighbours. HNSW, which organises vectors as a navigable multi-layer graph, is one widely used example. Other approaches partition the vector space, compress vectors or combine several strategies. ANN changes how the engine searches and may miss a true nearest neighbour; it does not define the number of results returned.

Candidate boundaries in lexical and dense retrieval#

A candidate boundary decides who can receive a score. Strict Boolean admits only records satisfying its processed conditions; direct BM25 normally requires at least one analysed-term match. Both use posting lists, although either candidate set can still be very large.³⁰

Dense retrieval has no comparable term-match floor. Unless a filter narrows the collection, every stored vector can receive a similarity score. Even an unrelated document will be nearest if everything else is farther away: nearest means best available, not necessarily relevant.

Key distinction

Top-k: complete rankings are possible—but often unnecessary#

Information retrieval uses k for the requested number of leaders. A top-k request asks for the k highest-scoring candidates rather than a complete ordering. Ten highest BM25 scores and ten nearest vectors both have k = 10; only the scoring evidence differs.

Top-k is therefore an output boundary, not a retrieval method. It is separate from the admission rule that forms the candidate set and from the ranking function that scores it.

Nor does top-k mean that only k candidates were examined. An exhaustive method may score every candidate while retaining only the current leaders. Exact BM25 pruning can safely skip candidates that cannot enter the leaders, while approximate nearest-neighbour search may inspect only part of the vector collection and miss a true neighbour. Exactness is another independent choice; Appendix C explains exact lexical pruning.

Visible counts and cut-offs reveal little about that execution. Google Scholar’s 1,000-result display limit does not prove that the interface shows an exhaustively ranked global top 1,000.³¹ In multi-stage systems, different values of k can govern different transitions: Chapter 7 shows a 30-record reranker budget and a five-abstract generation budget in Primo Research Assistant.

A top-k result can be Boolean-filtered, lexical, dense or hybrid, and exact or approximate. k says how many leaders were requested—not who could compete, how relevance was calculated or how much work was done.

Lexical retrieval can jump from a term to its matching documents. Dense retrieval must search for nearby vectors.

The inverted index described earlier and a dense-vector index provide two different routes to a candidate set:

Table 6.1 — How the candidate set is bounded on the lexical route and on the dense-vector route.

This is a comparison between indexing and retrieval routes, not necessarily between two mutually exclusive kinds of product. A product described as a vector database may store dense vectors and provide a nearest-neighbour index, but it may also support metadata filters, lexical fields or an inverted index. Likewise, contemporary search engines can combine lexical and vector retrieval: both Elastic and Weaviate document hybrid search that combines keyword or full-text evidence with vector evidence.

Returning to Puzzle 3: why did the keyword query retrieve more?#

Puzzle 3 began with an expectation created by a name and an interface. Semantic Scholar sounds as though it searches by meaning, perhaps through dense retrieval over embeddings, so a natural-language question seems like the query the system should handle best. Yet in August 2026, Is there an open access citation advantage returned only 13 results—far fewer than the hundreds of relevant papers expected for this established topic—while the compact keyword query open access citation advantage returned about 35,300. Why did the supposedly more natural query retrieve dramatically less?

The strongest architectural clue comes from Semantic Scholar’s documented 2025 main relevance-search pipeline:

Semantic Scholar’s documented main relevance search (2025)
query → Elasticsearch keyword retrieval over titles, abstracts and author names → up to 1,000 candidates → LightGBM reranking

The product name and natural-language-looking search box do not establish dense first-stage retrieval. The living technical account, revised 25 April 2025, documents lexical first-stage retrieval rather than embedding-based first-stage retrieval in this pipeline.³² On a lexical route, conversational words such as is there an may affect query analysis or execution, whereas the shorter query concentrates the vocabulary likely to occur in titles and abstracts. Depending on rules that the result screen does not expose, those extra words could narrow or otherwise alter the candidate set.

This is the strongest documented explanation for why the dense-retrieval intuition is unsafe, but it is not proof of the exact August 2026 cause. The architecture was documented in 2025, the current interface may have changed, and the two result counts alone cannot reveal the query parser or execution rule. Nor does the larger count prove that every returned paper is relevant. It shows only that keyword-style input produced a result-set size more consistent with the known breadth of the topic.

The point is not that Semantic Scholar lacks semantic capabilities. The platform also uses SPECTER embeddings for author disambiguation and recommendations. Their presence elsewhere still does not tell us which mechanism forms the candidates for main relevance search or where semantic capability enters that pipeline. Chapter 7 returns to the LightGBM stage once reranking is in view.

OpenAlex Alice semantic search supplies a deliberately stark contrast in which the documentation does identify dense first-stage retrieval. OpenAlex says Alice shipped in February 2026; its current documentation describes this route:

OpenAlex Alice semantic search (2026)
query → query embedding → comparison against stored work embeddings → nearest works by cosine similarity

OpenAlex embeds work titles and abstracts with GTE Large EN into 1,024-dimensional vectors, embeds the query with the same model, and returns the closest works by cosine similarity. Its 25 April 2026 Alice account describes a custom Elasticsearch vector index containing 413 million embeddings, including title-only embeddings for works without abstracts. The API treats search, search.exact and search.semantic as alternatives and permits only one per request, so this documented Alice route should not be relabelled as a hybrid query.

Table 6.2 — Semantic Scholar and OpenAlex Alice place candidate retrieval on different architectural routes.

Two systems can therefore both be discussed under the broad goal of semantic search while placing semantic capability in completely different parts of the pipeline. One begins with lexical candidates and applies a learnt ranker afterwards; the other retrieves candidates directly by dense-vector similarity.

The search box tells you how the user can express the query. “Semantic search” tells you what kind of matching the system aspires to provide. Dense retrieval, lexical retrieval and reranking tell you how particular stages may actually do the work.

Key distinction

A vector does not have to be dense or semantic#

Dense text embeddings provide the most familiar contemporary example of vectors in search. Now that their basic role is clear, we can add an important qualification: vector, dense and semantic do not mean the same thing.

Imagine a collection vocabulary containing 50,000 terms. Each term can be treated as a dimension. A document about Gen Z job-search anxiety might have non-zero weights for delulu, job, interview and a few other terms. The remaining dimensions would be zero.

That is a sparse vector: extremely long in principle, but with relatively few active dimensions.

BM25 can be represented this way. The query activates dimensions corresponding to its terms; the document contains calculated weights for terms occurring in it; and their score can be expressed as a dot product between sparse representations.³³ In practice, BM25 is usually executed with the kind of inverted index described earlier rather than a dense nearest-neighbour index.

Dense embeddings look different. They may have hundreds or thousands of dimensions, most of them non-zero. Those dimensions do not normally correspond directly to recognisable words such as delulu, job or unrealistic. Meaning and other learnt distinctions are distributed across the representation.

The terms sparse and dense therefore describe the shape of a representation. They do not tell us whether it is neural, learnt, semantic or interpretable.

Sparse and dense vectors compared

Table 6.3 — Three retrieval methods by representation: how their weights arise, and what their dimensions mean.

Density and learning are separate axes of a representation

Terminology varies across fields and systems. This book uses representation as the broad term and vector for the numerical object. Embedding refers to a learnt mapping into a vector space and, by common shorthand, to the resulting vector. BM25 is therefore described as a calculated sparse representation, SPLADE as a learnt sparse representation, and a dense bi-encoder as producing learnt dense embeddings. Other sources may call SPLADE-style outputs sparse embeddings.³⁴

One document, one chunk and one vector are not the same thing#

Dense retrieval also forces us to stop using document as though it meant only one thing.

A source document might be an article, paper, report, webpage or book. The retrieval system may index its abstract, sections, paragraphs, sentences or overlapping token windows. Those are the indexed units.

A 20-page article might therefore produce one indexed abstract, ten section units or 50 overlapping passages. In a conventional dense bi-encoder, each of those indexed units receives one pooled vector.

The same article indexed under different chunking strategies

These are separate choices:

• Source-document granularity: the object a person thinks of as the document.
• Indexed-unit granularity: the text unit the system searches.
• Representation granularity: how each indexed unit is encoded.
• Result aggregation: whether matching passages are displayed separately or combined into one article-level result.

Chunking a paper into 50 passages creates 50 searchable units and, with a conventional bi-encoder, 50 independently searchable vectors. It does not mean that the complete paper has been captured faithfully in one vector, nor that evidence spread across different chunks will automatically be combined.

A paper split into many searchable chunks and vectors

This is not a theoretical distinction, and current products sit at very different points on it. The Wiley AI Gateway returns chunks of the full text that may be relevant to a query, alongside the paper’s metadata—so the indexed unit is a passage, and evidence buried in a methods section is reachable. A discovery layer built over records described by their metadata and abstracts is working with a much coarser unit. Both may describe themselves as searching articles.

The consequence is blunt: a system cannot retrieve evidence from text it never indexed. If a claim appears only in a paper’s results section, and the retrieval system holds only that paper’s abstract, no encoder, reranker or generated summary downstream can recover it. This is the same bottleneck logic that governs shortlists, applied one level earlier—at the point where the collection itself was defined.

For librarians evaluating semantic search, the immediate questions are therefore more practical than architectural: what text was embedded, how large was each unit, and how are several matching passages turned back into an article-level result?³⁵

Rerankers: compare more carefully after retrieval#

A reranker is a second-pass scorer. It does not search the collection; it receives a list of candidates that an earlier stage already retrieved and decides how they should be ordered. Everything it can do is therefore bounded by what that earlier stage handed it.

That constraint is the whole reason rerankers exist in the form they do—and the reason they cannot fix certain problems at all. This section explains why systems accept the constraint, then works through the three arrangements that do the reordering: cross-encoders, late interaction, and large language models.

Why search systems use multiple stages#

Many production search systems are pipelines rather than one scoring formula. Query understanding and transformation may first interpret or rewrite the input, while retrieval control may route it. Once retrieval begins, fast methods search the full collection, while progressively more expensive methods examine progressively fewer records. Applying the most detailed query–document comparison to millions of records for every search would usually be too slow and costly.

Table 7.1 — The stages of a multi-stage pipeline, and why each belongs where it does.

A reranker therefore usually does not search the complete collection. A fast first-stage retriever—perhaps BM25, a dense bi-encoder, ColBERT or a hybrid combination—first produces a shortlist. The reranker then spends more computation deciding how those candidates should be ordered.

This division of labour creates both an efficiency gain and a ceiling. First-stage retrieval normally balances speed with recall; reranking concentrates on precision near the top. Chapter 11 gives those two words their exact meaning, and explains why only the first of them is settled here. Every shortlist is also a bottleneck: a reranker, fusion stage or answer-generating model cannot rescue a relevant document that no earlier route allowed into the candidate set.

Chapter 6 resolved the misleading semantic/dense assumption behind Puzzle 3. We can now see the rest of the documented architecture. Semantic Scholar’s 2025 technical account describes an Elasticsearch keyword search over titles, abstracts and author names, capped at 1,000 matches, followed by a trained LightGBM reranker that emphasises direct title matches, citation counts and recency. The first stage determines which papers the learnt ranker ever gets to reconsider. The second stage changes the order within that boundary. This is machine-learned reranking, but not dense retrieval or an LLM reranker. LightGBM cannot retrieve a paper excluded by the first stage, and its presence does not by itself explain why the August 2026 natural-language query returned 13 results. Appendix E reconstructs the earlier documented 2020 version in depth; the architecture itself belongs in the main chapter.

Retrieve broadly, then rerank a shortlist more carefully

Cross-encoder reranking#

A bi-encoder encodes the query and candidate independently. Document or passage vectors can therefore be calculated in advance, and search reduces to comparing the query vector with stored vectors. That is what makes large-scale dense retrieval practical. The cost is that the query and document do not interact until after each has already been compressed into its representation.

A cross-encoder does the opposite. It places the query and one candidate into the model together, typically as a single input sequence, and allows attention across all their tokens before producing a relevance score. The model can therefore examine whether a particular query term aligns with a particular passage, whether a qualifier changes the claim, or whether two otherwise similar texts differ in negation, entity or intent. Cross-encoder is an arrangement rather than a model: the same pretrained encoder that a bi-encoder trains for independent encoding can be trained to score query–candidate pairs jointly instead. Putting BERT to work this way is how the arrangement became prominent in retrieval (Nogueira and Cho, 2019).

The price is computation. The candidate representation cannot simply be precomputed and reused because its encoding depends on the query. Reranking 100 candidates requires roughly 100 query–candidate model evaluations. This is usually affordable for a shortlist, but not for millions of records. Cross-encoder scores also reflect their training data, text truncation and definition of relevance; joint attention makes finer comparison possible, not infallible.

Bi-encoders precompute; cross-encoders compare query and candidate together

Where ColBERT-style late interaction fits#

ColBERT sits between the pooled bi-encoder and the full cross-encoder. Like a bi-encoder, it encodes the query and document independently, so document token vectors can be prepared before the query arrives. Unlike a conventional bi-encoder, it does not reduce each indexed unit to one vector. Its late-interaction score compares query-token vectors with document-token vectors after encoding.

This preserves more local matching evidence than a single pooled vector without paying the full cost of jointly encoding every query–document pair. ColBERT-style models can be used for first-stage retrieval or to rerank a shortlist, depending on the implementation and scale. They require more storage and computation than single-vector retrieval, but normally less query-time work than a cross-encoder over the same candidates.³⁶

Where late interaction sits between bi-encoders and cross-encoders

Two consequences are worth a librarian’s attention, because they pull in opposite directions.

The storage bill is real. Keeping one vector per token rather than one per passage multiplies the index by roughly the number of tokens in a passage—two orders of magnitude is a fair mental estimate before compression. This is why late interaction is an infrastructure decision rather than a model swap: ColBERTv2 reports cutting the footprint of earlier late-interaction systems by six to ten times through residual compression, and the fact that such work was necessary tells you how large the untreated figure is. A vendor who has deployed late interaction has spent something to do it.

The interpretability is the best available in neural retrieval. Because the score is a sum of per-query-token maxima, it decomposes: for each word in the query, the system can name the document token that matched it most strongly. That is an inspectable alignment, not a post-hoc rationalisation, and it is closer to the evidence a lexical system can show than anything a pooled vector offers. The limit is that the token vectors themselves remain latent—you can see which tokens aligned, not why the model considered them similar—so ColBERT is inspectable at the scoring layer without being transparent end to end. Chapter 12 returns to what that distinction buys. For a visual, in-depth introduction, see Amélie Chatelain’s multi-vector search lecture.

All three arrangements are now on the table, and they answer the question the section on BERT had to leave open. A single pretrained encoder can be deployed in any of them: encode query and candidate independently and compare two pooled vectors, encode them jointly in one sequence and score the pair, or encode them separately as sets of token vectors and compare those. The choice is not a detail of implementation. It fixes what the system can afford—independent encoding scales to a collection, joint encoding does not—and it fixes what evidence survives to the comparison, because pooling discards the local detail that the other two keep.

Naming the model identifies the family it came from. What determines behaviour is which of these arrangements it was placed in, and what it was trained to rank.

LLMs as rerankers#

A general-purpose or specially trained large language model can also act as a reranker. Instead of merely comparing embeddings, it receives the query, candidate text and sometimes explicit relevance instructions. This makes it possible to ask the model to consider criteria such as population, method, date, study design or whether a passage actually answers the question rather than merely sharing its topic.

LLM reranking is commonly framed in three ways:

• Pointwise: judge or score each candidate independently. This scales linearly with the shortlist, but the scores may require calibration and the model does not directly compare candidates.
• Pairwise: compare two candidates and choose which is more relevant. Direct comparison can help with subtle distinctions, but many comparisons may be needed and preferences can be inconsistent.
• Listwise: present several candidates together and ask the model to return an ordering. This gives the model comparative context, but long lists exceed context limits, so systems often use sliding windows or repeated partial rankings. Input order can influence the result.

Pointwise, pairwise and listwise LLM reranking

These are not merely prompting details. They create different costs and failure modes. Research has demonstrated both listwise and pairwise LLM reranking, including zero-shot listwise reranking (Ma et al., 2023) and pairwise ranking prompting (Qin et al., 2024).

The words pointwise, pairwise and listwise also occur in classical learning-to-rank research, where they describe what the training objective learns from. Here they describe how an LLM is asked to judge candidates at reranking time. The shared vocabulary does not make the two arrangements identical; Appendix E separates them.

LLM rerankers can use richer instructions and longer textual context than many conventional rankers, but they are usually slower and more expensive. They may also be sensitive to prompt wording, candidate order, context-window limits, model updates and sampling settings. A generated explanation can make a decision easier to inspect, but a plausible explanation is not evidence that the ranking is correct.

The practical questions are therefore: what produced the shortlist, how many candidates reached the reranker, what text and metadata the reranker saw, which ranking formulation it used, and how the complete pipeline was evaluated.

A sophisticated reranker can reorder only what the retriever allowed it to see.

Reranking in two documented academic pipelines#

Two systems already mentioned in this book document their reranking stage precisely enough to see the pattern, and they use different machinery for it.

PubMed Best Match retrieves and orders records with BM25, then applies LambdaMART to the top 500. Primo Research Assistant retrieves through CDI's largely lexical ranking, then reranks the top 30 with embeddings. These values of k are stage-specific candidate budgets: they set what each reranker can see, not what is relevant.

Learning to rank (LTR) names the broader supervised approach behind LambdaMART. Instead of manually fixing the final influence of every ranking factor, a system is trained on examples to combine signals into an ordering. Those signals can include lexical evidence such as BM25 and field matches alongside document or query properties such as recency, publication type or past usage. They need not be embeddings, and the model learns only from the features and relevance evidence it is given. Appendix E develops the idea without requiring the algorithmic detail.

Notice what the two numbers do. Five hundred and thirty are not statements about relevance; they are budgets. Everything below the line is ordered by the first stage alone, and everything outside the first stage's results is unreachable regardless. When a vendor says its search "uses AI reranking", the useful follow-up is not which model but how many candidates reached it.

Key distinction

Neural information retrieval is broader than dense retrieval#

Neural information retrieval names an umbrella category, not a dense representation or a single pipeline stage. A neural model may retrieve with one dense vector, retain several token vectors for late interaction, produce sparse vocabulary-aligned weights, or rerank candidates retrieved by another method. The cross-encoders and LLM rerankers above are therefore part of neural IR even when the first-stage retriever is lexical.

Most contemporary examples in this book use Transformers, but neural IR is older and broader than that architecture: earlier systems used feed-forward, convolutional and recurrent networks, and a Transformer model is not inherently a retriever.³⁷

Late interaction, described above, is one of two arrangements that sit awkwardly between the simpler lexical–dense categories this book has used. The other runs the other way: it is lexical in shape but learnt. It belongs here, immediately before hybrid retrieval, because it is the alternative to one—a single model that carries both lexical and semantic evidence, rather than two retrievers and a rule for combining them.

Learnt sparse retrieval, represented by systems such as SPLADE, uses neural training to assign weights and predict useful expansions while retaining a sparse, vocabulary-aligned representation suitable for an inverted index.³⁸ It can be understood as a learnt bridge between lexical matching and semantic expansion. The floor described earlier still holds: a purely lexical scoring model must match at least one term. What changes is that the floor now applies to the expanded term set rather than the words as typed, which is how a sparse system running on an inverted index can retrieve a document sharing no typed word with the query without becoming dense. It can also be more interpretable than a pooled dense embedding: its active dimensions are labelled with vocabulary terms, so an analyst can inspect which original or expanded terms received weight. That visibility does not make every expansion correct or explain the model’s complete decision process.³⁹ As a concrete literature-review example, Elicit uses SPLADE: model-suggested related terms enrich the query before they are sent to regular full-text search.

SPLADE learns a sparse, vocabulary-aligned expansion

It runs on the infrastructure a library already has. This is the practical point that most often gets lost. Because the representation stays sparse and vocabulary-aligned, learnt sparse retrieval is served by ordinary Lucene-family inverted-index machinery. There is no separate vector database to license, operate, back up or keep in sync. For a library evaluating how far a semantic capability would reach into its stack, that is a materially different proposition from dense retrieval.

The cost reappears as query latency. Expansion is not free: adding terms to a query lengthens the posting lists the engine must traverse, and expanded terms tend to be common ones with long lists. Reported latencies for SPLADE-style models have run several times BM25’s on the same hardware, and a line of work exists specifically to close that gap through query-side regularisation and pruning.⁴⁰ “Runs on an inverted index” means the infrastructure is familiar, not that the query is as cheap.

The two variants are not equally visible in this sector, and the asymmetry is worth stating rather than leaving as an accident of which example was available. Learnt sparse retrieval has a named academic deployment in Elicit. Late interaction does not appear to have a comparable publicly documented deployment in a major academic search product—it is well represented in the research literature and in general-purpose retrieval infrastructure, less so on a vendor’s architecture page. The storage cost described above is the usual explanation offered. Absence of documentation is not evidence of absence, and this is exactly the kind of claim that dates, so treat it as a description of what is currently inspectable rather than of what exists.

Both SPLADE and ColBERT remain dependent on model vocabulary and training data, and neither escapes out-of-distribution transfer. But both matter more to a library than their research profile suggests, for a reason that has nothing to do with ranking quality: they are the two neural methods whose evidence a librarian can actually read. One exposes weighted vocabulary terms, the other exposes token alignments. Where a pooled dense vector offers a similarity score and nothing else, these offer something a search can be documented and defended with.

Why hybrid retrieval remains attractive#

The pipeline table above quietly introduced something the text has not yet named. Its second stage combines candidates supplied by parallel retrievers—and a system that draws on more than one retrieval method at once is a hybrid. Every method needed to build one has now been described, so this is the point to ask why anyone would.

The cleanest summary is more complicated than keywords versus semantics:

A lexical system may preserve the word and miss the meaning.

A single-vector dense system may recover the meaning and blur the exact word.

Hybrid retrieval protects against systems with different and partly complementary blind spots.

Boolean can provide explicit control. BM25 can preserve analysed lexical term evidence, including rare identifiers. Dense retrieval can bridge different formulations in a compact latent space. Rerankers and less common neural architectures may add further signals, but the main reason for combining lexical and dense retrieval is already clear: each can recover relevant material the other may miss.

Hybrid and multi-stage are not synonyms. Hybrid describes multiple retrieval signals, representations or result lists contributing to the search. Multi-stage, which the preceding section described, means operations applied sequentially to progressively smaller candidate sets. One is about how many kinds of evidence contribute; the other is about the order in which work is done.

A system can be hybrid, multi-stage, both or neither. Lexical and dense retrievers running in parallel, followed by fusion and reranking, form both a hybrid and a multi-stage pipeline. BM25 followed by a cross-encoder reranker is multi-stage without necessarily being hybrid. A single scoring stage that blends lexical and dense evidence can be hybrid without using a reranking cascade.

The arrangement and the combination method both matter. A system may merge ranked lists with reciprocal rank fusion (RRF), combine normalised scores, or combine parallel retrieval with a later reranker. Routing is a different decision: it chooses which path or paths run; if several lists result, a later fusion rule still has to combine them. RRF gives each record a contribution based on its position in every input list, so it can combine, for example, a BM25 ranking and a dense ranking without pretending that their raw scores share a scale. It rewards high placement and repeated appearance, but discards the size of the score gaps inside each list. It is a fusion rule, not a relevance model or reranker. Appendix E gives the formula and a worked example.

Two hybrids that combine differently#

The difference between blending and routing is easiest to see in two current products.

Web of Science Smart Search blends. An NLP parser first builds a structured Boolean query; document search can then run a Boolean keyword path and a semantic-vector path and combine their results. One interface, one question, two kinds of evidence contributing to a single ranking. This is the parallel arrangement described above—and note that its query-parsing stage is not the hybrid part. Parsing changed the query; the blend is what makes it hybrid.

Scopus AI routes. Rather than always running both paths, it can decide per question whether to search lexically, to search over vector embeddings, or to do both. The design space is the same, but the combination rule is a decision rather than a fixed blend—which means the answer to "is this a lexical search or a vector search?" may differ from one query to the next, and neither answer is wrong.

That second pattern is doing something the first is not, and it is worth naming. A blend runs and combines prescribed paths. A route is a retrieval-control decision made at query time. A rule, classifier or model may make that choice; routing alone therefore does not establish agency. Chapter 8 separates the interpretation that may inform a route from the route itself, and Chapter 9 asks who chooses subsequent actions.

“Hybrid” is therefore not one architecture or a guarantee of quality. It is a design space—and, as those sections will add, it still says nothing about who decided to search this way in the first place.

We can now return to the distinction introduced during the lexical sections. Query understanding interprets the request and may leave the retrieval input unchanged. Query transformation changes or creates retrieval inputs. Retrieval control chooses where those inputs go, whether several paths run, and whether another search happens.

These operations are often grouped loosely under query understanding, but they do not occupy one pipeline stage. Interpretation may trigger transformation, a routing decision or both. Most transformations occur before a retrieval pass, while feedback methods construct a later query from the results of an earlier one.

Before phrasing: identify the query object#

Advice about keywords, questions and prompts often treats them as three ways to fill the same box. They are still strings. The more consequential choice is what mechanism the search addresses and what object stands for the information need. Phrasing style is downstream of both.

Table 8.1 — The query object determines which retrieval mechanism the search addresses.

These objects cooperate. Free-text and controlled-vocabulary searches can find seeds; citation traversal can expose missing terminology; accepted records can drive a feedback pass; an API can make the resulting procedure repeatable. This is especially important when recall matters. In one audit of a single review of complex evidence, Greenhalgh and Peacock traced 495 included sources: only 30% came from the protocol-defined database and hand searches, while 51% were found by snowballing. That result is not a universal estimate for every review. It is a concrete demonstration that one well-phrased database query may still cover only one route to the literature.

Seed-based methods also differ from one another. Direct citations follow an explicit edge. Bibliographic coupling finds papers that share references; co-citation finds papers cited together; text similarity compares their language or representations. Sjögårde and Ahlgren's 2024 evaluation found that combined citation approaches generally retrieved more relevant publications than any one citation relation, and that adding PubMed's text-based related-article score improved the combination on some measures. The lesson is mechanism diversity, not that one citation method always wins.

This wider view also explains why real searching is iterative. Bates called the pattern berrypicking: the need and the useful query develop as the searcher encounters documents, authors, citations and terminology. What changes between steps may therefore be the words, the seed, the graph edge, the judged examples or the source being browsed. Calling every step a reformulated query erases the evidence the searcher actually used.

Phrase for the mechanism, not the box#

When a string really is the query, its useful form depends on what will read it. A polished paragraph is not intrinsically more semantic than a short term list, and a keyword string is not intrinsically more reproducible than a visible structured expression.

Table 8.2 — The useful form of a text input depends on its receiver.

Research on automated query construction reinforces the distinction. In systematic-review screening, Wang and colleagues found that converting Boolean strategies into natural-language queries could improve a neural ranker, while fusion with the original Boolean results helped in some settings and not in others. In a separate evaluation of LLM-generated Boolean strategies, outputs varied and could contain invalid MeSH terms. Meanwhile, the FollowIR benchmark found that existing retrieval models often treated detailed relevance instructions as basic keywords and struggled with the long-form information; instruction-following improved after targeted training. None of these results licenses the rule “write naturally” or “write more.” Each says to match the expression to the component that receives it, inspect any transformation, and test the resulting route on the actual task.

The box shows an interface. It does not tell you what object the system will construct from your input, which mechanism will receive it, or which other search method should run alongside it.

What a system can do with your query#

When a system writes the Boolean query

A search interface can use an NLP parser⁴¹ or a language model to turn a natural-language request into fields, filters and Boolean clauses before retrieval. For example, Web of Science Smart Search makes its interpretation visible:

Web of Science Smart Search makes the transformation visible

Primo NDE Natural Language Search likewise documents classifying the input, generating a Boolean query and expanding it into OR-connected permutations. In both products, the generated query remains visible and editable.

The transformation changes what is sent to retrieval, not necessarily the retriever. Natural-language input may still lead to Boolean admission and lexical ranking; visibility matters because an omitted concept or misread constraint changes the candidate set before ranking.

The useful questions are what the system changed and when. The operations below can be combined, and they are not mutually exclusive.

Table 8.3 — What a system can do with a query, when it happens and the main risk it introduces.

Pseudo-relevance feedback breaks an upstream-only picture. It is retrieval-dependent because the second query cannot be constructed until the first ranking exists. Yet a workflow that always performs those two passes can still be fixed and non-agentic: the query contents adapt to the results, while the sequence was prescribed in advance. Chapter 9 separates that content adaptation from the question of who chooses the next action.

A system might identify a date filter, generate three reformulations, send each to lexical and dense retrievers, fuse their candidate lists and rerank the result. Calling the interface “natural-language search” reveals none of those internal decisions.

Query2doc and HyDE: generating text to retrieve text#

Query2doc prompts an LLM to generate a pseudo-document and uses its language to expand the original query. The expanded text can help a sparse retriever such as BM25 by supplying vocabulary likely to appear in relevant documents, and it can also be used with dense retrieval.

HyDE, short for Hypothetical Document Embeddings, uses a related idea differently. An LLM writes a hypothetical document that might answer the query. A dense encoder embeds that generated document, and the system retrieves real documents whose vectors are nearby.

HyDE searches from the vector of an imagined relevant document. The imagined document is a retrieval aid, not evidence.

The distinction matters. Query2doc uses generated language to enrich a query; HyDE uses a generated document as a pivot into a dense vector space. Both can bridge vocabulary gaps, but both can also import unsupported assumptions. Retrieval grounds the search in real indexed documents, but it does not retroactively make every detail in the generated text true.

Query interpretation and transformation in current discovery products#

Current research products illustrate different combinations of interpretation, transformation and retrieval. In each case the useful question is what the system inferred, what retrieval input it produced and what machinery received that input.

Table 8.4 — Query interpretation and transformation in current discovery products, and what each case illustrates.Vendor documentation checked August 2026.

Natural-language filter extraction is useful but limited

Some academic search tools can also recognise a limited set of constraints in a natural-language request, then turn them into fields, filters or refinements. Recognition is query understanding; constructing the fielded or filtered input is transformation. Neither operation is equivalent to supporting every filter, field tag or complex Boolean relationship available in the underlying database. A comparative review in Katina Magazine illustrates these differences, while the product documentation defines the currently supported controls:

Table 8.5 — Natural-language filter extraction across products, with the documented boundary in each case.Vendor documentation checked August 2026.

The practical lesson is to distinguish query-derived filters from filters the user selects explicitly after retrieval. Evaluators should test which natural-language formulations activate each supported constraint, whether the generated filter is visible, and what happens when the request contains an unsupported or ambiguous limit.

Primo VE NDE Natural Language Search is distinct from the Primo Research Assistant discussed earlier, which documents an LLM converting a question into Boolean variations before CDI retrieval and embedding reranking in a multi-stage pipeline.

These examples are not one natural-language architecture. A product may interpret with a conventional parser or generative model, transform the input into a structured query, and then use Boolean, lexical, vector or hybrid retrieval. The interface label alone does not reveal which operations occurred.

Understanding interprets the request. Transformation changes what a retrieval pass receives.

This is especially important for evidence searches. Ranking cannot rescue a relevant document if a rewrite, compulsory clause, filter or route prevented it from becoming a candidate; feedback can also reinforce an unrepresentative first ranking. Evaluation and reporting should therefore distinguish the original input, every transformed input, the selected routes and any intermediate results used to construct a later query.

All of these operations can appear in a non-agentic workflow. A designer can prescribe five reformulations, two retrievers, one fusion rule and one reranker in advance. Even a two-pass PRF workflow can be fixed in its sequence while adapting the contents of its second query. Chapter 9 asks the separate question: who chooses what happens next?

Chapters 7 and 8 showed a system generating several queries, routing them to different retrievers, fusing the results and reranking the candidates. None of that is necessarily agentic. The whole sequence may have been fixed before any user arrived. A separate question is whether the system can look at what came back and choose what to do next.

Chapter 8 also separated a task brief from the queries it may cause. The brief is input to the controller: it defines scope, criteria, sources, coverage, stopping conditions and the required record. The controller then turns that specification into retrieval inputs and tool calls. Calling the brief a query hides the layer that interprets it.

Pseudo-relevance feedback is a useful boundary case. Its second query depends on the first ranking, so its content adapts to retrieval. But if the system always runs the same two passes, its control sequence is fixed. A rule that decides whether to search again makes the control flow adaptive; a model allowed to choose the next query, route, tool or stopping point from what it observes may make it agentic.

Fixed, adaptive and agentic are not the same thing#

Table 9.1 — Fixed, adaptive and agentic control arrangements, distinguished by how the next action is decided.

The middle row is the one usually left out, and it is where a great many products actually sit—including most of the first wave of academic deep search. It is also the row most often mistaken for the third, because a model may well be doing the choosing: reading a yield and deciding to broaden is a decision, and a language model can make it. What makes the arrangement adaptive rather than agentic is that the alternatives were drawn before the question arrived. Ai2 have described Asta Find Paper—the tool their earlier Paper Finder became—in close to these terms: a semi-rigid flow, whose predefined structure is influenced at key points by model decisions. Librarians have been executing that second row by hand for decades: run the search, look at the yield, broaden or narrow, run it again.

Read the three rows as landmarks on a continuum rather than three species, running from a wholly handcrafted pipeline to a system given tools and a goal and left to work out the sequence. Real products sit between, and the same product can sit differently in different modes—which is why the rest of this chapter asks where a tool sits rather than which box it belongs in.

Agentic search is search in which a model, working within a given set of tools, instructions and limits, selects at least some subsequent retrieval actions from the task and from what it has observed—rather than executing a sequence fixed in advance.

Two things that definition deliberately withholds. It does not say the model is unconstrained: an agent works inside a harness—which tools it may call, which sources it can reach, how many rounds it is allowed, when it must stop. Tool here does not mean a product a library licenses. It means one specific operation the model has been given permission to invoke—run this search, fetch this record’s references, restrict to these years—described precisely enough that the model can decide when to call it and what to pass in. It does not say the model is reasoning from first principles either. The harness is usually where the consequential design decisions live, and it is usually the part a vendor does not describe.

Deep search can still be agentic search#

The opening diagram sorted tools by output format and by how extensively they retrieve. Agency is a third property, and it lies on neither axis.

Table 9.2 — Deep search and deep research compared by search trajectory and by main output.

Deep tells us how far the search goes. Agentic tells us how the next direction is chosen.

This resolves a common misreading. A tool that retrieves repeatedly, revises its searches in response to what it finds, and then hands you a ranked set of papers is doing agentic deep search. It has not thereby become deep research. It does not have to write a report to qualify as agentic, and writing one would not make it more so. The upper-left quadrant of the opening diagram—iterative search—is exactly where agentic deep search belongs.

The converse matters just as much. A deep research tool that returns a long, fluent, well-cited report may have run an entirely fixed workflow underneath. Length and polish of output are not evidence of agency; they are evidence of generation.

Agentic RAG is narrower than agentic search#

Basic retrieval-augmented generation (RAG) retrieves and then generates: what retrieval returns is placed in the model’s context so the answer it writes is grounded in those sources. Agentic RAG lets a model choose retrieval actions while working towards a generated response. Agentic search is broader than both: it may end with a ranked set of papers, passages or evidence and never generate prose at all.

Keeping these apart prevents RAG, deep research and agentic search from collapsing into a single marketing category—which, from the outside, is exactly what they tend to look like.

Where current academic tools actually sit#

These distinctions would be idle if products sorted themselves neatly into them, so it is worth testing whether they do. The probe I used, written up as “Deep Research, Shallow Agency: What Academic Deep Research Can and Can’t Do”, was deliberately ordinary: find the papers that could have been cited by this article but were not. It needs no specialist knowledge and no capability these tools lack—identify the paper, extract its reference list, find related work on the same topic, then subtract the second from the third. What it does not do is match any product’s predetermined workflow.

Most of the specialised academic tools failed it, and the shape of each failure is more informative than the verdict. Ai2 Asta Find Paper treated the request as a lookup and stopped once it had found the target article. Consensus Deep Search returned papers the article already cited, despite running four sub-agents. Undermind found related work, most of it already in the reference list. Elicit’s screening pipeline—search, set criteria, screen, extract—has no step at which is this one already cited? could be asked. SciSpace Agents came closest, reasoning about the comparison and attempting it, then foundering on matching the metadata. General-purpose assistants with web search passed, and did so with no academic connector at all.

That last result is worth sitting with, because the obvious reading of it is wrong. The academic tools were not out-thought. Several showed every sign of having understood the request—Undermind asked clarifying questions that implied it knew exactly what was wanted—and then had no operation available that would perform the subtraction. An agent can only choose among the tools it has. Reasoning does not substitute for a missing capability, and no amount of iteration will invent one: a loop can only re-select from the same menu. From outside, the two failures are hard to tell apart, since a tool that lacked a capability and a tool that had it and reasoned poorly produce the same empty result.

Sorting the tools themselves is a separate exercise, done in April 2026 and written up as “From Fixed Search Workflows to Agentic Academic Search”. It describes how these products were built at that point, and it will date quickly.

Table 9.3 — Where current academic tools sat at the time of testing. The rows are positions on the spectrum, not three species.Tested April 2026; products have changed since.

Two things in that table are worth more than the names.

First, the same vendor appears at both ends. Undermind’s original deep search ran a predefined structure; Undermind Projects does not. Agency is a property of the mode you are using, not of the brand printed at the top of the page—which is precisely why a vendor’s marketing copy cannot answer the question for you.

Second, Ai2 Asta Find Paper sits in the upper-left quadrant of the opening diagram—iterative search, returning records rather than a report—while running a predefined structure. Repeated retrieval is not evidence of agency: a model may be making calls inside that structure without ever selecting a move its designers had not already provided. The cell exists, and it is occupied.

One caveat about the evidence. That testing used a single task type, probing one kind of multi-step reasoning. A tool could fail it and still perform well on other agentic tasks, or pass it through a lucky sequence of actions rather than robust reasoning. Treat the table as an illustration of the distinction, not a league table.

Agency changes the sequence, not the retrieval#

Every action available to an agent is one this book has already explained. It can construct or revise a Boolean query, run BM25, run dense or learnt-sparse retrieval, follow citations, apply metadata filters, choose a database, fuse result lists, rerank candidates, inspect a shortlist—and decide whether to go round again.

Agency governs which of those operations run, in what order, and when to stop. It does not change how BM25 scores a document, how an ANN index finds neighbours or what a cross-encoder does with a shortlist. The retrieval machinery underneath is the machinery described in the preceding chapters.

Scopus AI's routing is the smallest useful illustration. Deciding per question whether to search lexically, over embeddings, or both is a single agentic choice—one selection, made once, before retrieval. Its lexical path is unchanged by having been chosen; so is its vector path. Nothing about the retrieval improved. What changed is that a fixed blend became a decision, and a decision is something that can be right or wrong for a given question, and that a librarian may want to see recorded.

Agentic is not a synonym for dense, hybrid, multi-stage or iterative.

Agency completes a set of three independent properties. Hybrid describes which retrieval signals contribute; multi-stage describes the order in which operations are applied; agentic describes who decided that order. They combine freely. Repeated model-directed BM25 searches are agentic but not hybrid. Fixed parallel lexical and dense retrieval with reciprocal rank fusion is hybrid and multi-stage but not agentic. A production pipeline may be all three at once, and a product may advertise one while possessing only that one. As with every other distinction in this book, the interface label settles none of it.

The clearest demonstration of this comes from a recent development in the products themselves. Scite and the Wiley AI Gateway now expose their search as MCP servers. The Model Context Protocol is a convention for describing a service to a model rather than to a programmer: the server publishes what operations it offers, what each one expects and what it returns, and a model reads that description while it is working and decides whether to call it. Wiley’s returns full-text chunks likely to be relevant to the query alongside the paper’s metadata. A librarian can point a general-purpose agent at these and assemble a retrieval system whose controller is theirs and whose retrievers are the vendors’.

Notice what does and does not change in that arrangement. Scite’s index, its analysis pipeline, its ranking and its citation-statement evidence are the same whether you use its own interface or call it from your own agent. What changes is who decides when to call it, what to ask it, and whether to ask again. That is the whole of what agency contributes—and it is why an agentic system built on a weak retriever is still a weak search.

Two things MCP is not. It is not a replacement for an API: it is a description layer over a service the vendor already runs, which is why the index and the ranking do not change, and why a library’s authentication and entitlements apply exactly as before. An MCP endpoint is not a route around a subscription. Nor is it a rival to RAG. Appendix D separates that pair, which are the two most often confused.

What agency costs, and what it buys#

More agency is not simply better. It is a trade, and for a great many searches it is a bad one.

Table 9.4 — What agency costs and what it buys, against a fixed or adaptive workflow.

That final row is the whole argument for agency, and it is a real one. A fixed pipeline is only ever as good as its designer’s imagination, and researchers routinely ask questions no product designer anticipated. But it is worth being honest about how often that applies. A great deal of academic searching is a known task type, and paying an agent to rediscover a procedure a librarian could have specified is a poor bargain.

Two further difficulties are specific to retrieval, and the second is the serious one.

The menu is invisible as well as the path. A trace tells you what a system did; it does not tell you what it could have done. When a platform can theoretically do anything, a user has no reliable way to discover what it can actually do, or how the vendor has configured its agents. This is not the black-box problem—it is not about hidden reasoning but about hidden capability. A user has no way of knowing that a tool’s academic search cannot extract references from a paper while its web search can, so a capable system fails a task it could have completed, and the failure looks like incapacity.

Recall failures compound across rounds. The shortlist bottleneck now applies once per round, and the rounds are not independent. A reranker cannot rescue a document the retriever never returned. In an agentic loop, round two is chosen on the basis of what round one returned—so a concept missed at the start may never be recovered, because nothing downstream has any reason to look for it. Iteration compounds recall failures as readily as it repairs them.

None of this is an argument against agentic search. It is an argument for asking which arrangement a task actually needs—and, when the answer is agency, for asking to see the trajectory.

What the loop is for#

This chapter has been careful about what agency does not do, and that care was earned. But it leaves the balance wrong, because there is something agency does that nothing else in this book can—and it is not what the marketing claims. Agency does not make a retrieval better. It changes what happens after a bad one.

Consider the query that returns nothing. In a conventional catalogue that is the end of the interaction: the searcher is told there are no results and left to work out why. I tested the alternative by pointing a model, through an MCP server, at my own library’s Primo index, and replaying real queries from our search logs that had returned zero results. A user looking for databases on autism gets nothing, because no database is called that. The loop broadens—psychology, medicine, education, disability—and returns PsycINFO, Education Research Complete and SocINDEX. cost of living becomes inflation, household expenditure and consumer prices, and reaches the economics databases. Rolex watches is recognised as a question about luxury goods and produces Statista and Passport. Misspellings resolve on the way past: statis to Statista, refinituv to LSEG.

The comparison that matters is not against a human. It is against Primo’s own Natural Language Search, which converts a question into a Boolean string in a single pass. On simple misspellings it does well. On autism it produced a creditable expansion—(autism) OR (autistic disorder) OR (ASD)—which still returned zero results, because the problem was never the synonyms. One-shot transformation cannot learn from a failure it does not see. The loop can, because it reads the empty result and treats it as information.

What is doing the work here is worth being precise about, because it is not the obvious answer. It is not the embeddings: the retriever underneath is a keyword endpoint. It is not MCP, which is only plumbing—a plain CSV of our database holdings, handed to the same model, reproduced most of the result. It is not even model capability: a much smaller model performed nearly as well as the largest one I tried. What remains is the loop itself, and the freedom it confers to broaden a concept, shift granularity and try again. This is the strongest evidence in the book for a claim Chapters 3 and 4 made on other grounds: lexical retrieval is not obsolete. Wrapped in something that can fail and retry, an ordinary keyword search outperformed a cleverer single rewrite over the same index.

For librarians this lands close to home. Which database should I use? is a question the literature has worried at for twenty years, and it has resisted every interface redesign aimed at it. This is the first mechanism I have seen that looks capable of absorbing most of it. It also moves the design question. The thing to ask of a search system stops being does it return the right result for a well-formed query and becomes does it recover when the query is poor—which is a different property, and one Chapter 11 has to find a way to measure.

The limits fall exactly where a library’s knowledge does. The loop works because the model has read the open web—libguides, vendor pages, reviews—and can therefore guess that autism research lives in PsycINFO. A database the open web has barely heard of gets no such help, and a newly licensed one none at all. Coverage, field quirks, embargo dates, what your institution actually subscribes to this year: these are what a model cannot infer and a library holds. The work does not disappear. It moves from answering the routine question to supplying what the loop cannot know, and to the specialised request—this variable, these companies, 2005 to 2020—where it still runs out.

Agency does not improve a search. It gives a failed search somewhere to go.

From the result screen, many retrieval failures look alike: a relevant record is missing, or an irrelevant one sits near the top. The useful question is which evidence would distinguish several possible explanations, because more than one can apply at once.

Before any of that, establish that the record could have been returned at all. A record absent from the collection, still inside an indexing lag, or excluded by a date limit, format facet or access filter was never eligible, and no amount of vocabulary work will retrieve it. A record that was retrieved but fell outside the result boundary was matched; a cutoff hid it. Where a system indexes passages rather than whole documents, a claim split across separate indexed units may leave no single unit carrying enough evidence to rank. And when a returned result simply lacks a word that was typed, the four explanations in Chapter 4 apply first: the term may have been removed during analysis or rewriting, or retained and left unmatched. These checks are cheap, and they eliminate causes that no amount of representation work would address.

Once eligibility is established, the practical workflow works through the missing record’s language, its representation, the retriever’s training and the matching behaviour that produced the result.

Table 10.1 — A diagnostic map for retrieval failures. The rows are prompts for investigation, not an exhaustive or mutually exclusive taxonomy.

The four rows are different kinds of claim, which is why they cannot simply be tried in turn. Vocabulary mismatch is a relationship visible between a query and a relevant record. OOV is a property relative to a named vocabulary. OOD is a relationship between training and deployment. A failure of negation, direction or composition is a limit in matching or model capability.

The rest of the chapter takes those diagnostic questions in order, beginning with the problem librarians have always managed. A synonym or controlled term may sometimes work around lexical OOV; it does not by itself address OOD transfer or a failure of logic and relation.

Vocabulary mismatch: a problem librarians already know#

Suppose a search for heart attack recovery misses a relevant article titled:

Rehabilitation after myocardial infarction

The query and the article express sufficiently related concepts using different lexical forms. The vocabulary mismatch problem long predates neural retrieval. Furnas and colleagues (1987) showed how rarely people choose the same words for the same objects and operations; later IR work states the problem directly as lexically different but semantically related query and document terms (Jeong et al., 2021).

Librarians have never needed an embedding to recognise this. They respond with synonyms, spelling and morphological variants, controlled vocabulary, citation searching, pearl growing and iterative reformulation. In strict Boolean search, OR and subject headings make those bridges explicit. Query transformation may add them automatically.

BM25 changes the ranking rule, not the identity of the terms. It can rank a record from partial lexical overlap, and stemming or analysis may connect morphological variants, but BM25 does not itself make heart attack synonymous with myocardial infarction.

A suitably trained dense retriever offers another bridge. It can place the complete query and document representations near one another without exact overlap. That is one modern response to vocabulary mismatch, not the origin of the problem and not a guarantee that every paraphrase will be connected.

The earlier delulu example is the same diagnosis in newer clothing: a searcher writes delulu about getting this job while a relevant page discusses unrealistic expectations during a job search. The age or register of the expressions may affect whether a learnt model connects them, but the visible query–document relation is still vocabulary mismatch.

Dense retrieval is a newer response to an old retrieval problem.

Out-of-vocabulary (OOV): missing from which vocabulary?#

Suppose a DOI, gene name or chemical identifier is visibly present in a record, yet searching it returns nothing useful or behaves unpredictably. The first question is not whether the string is simply “too new” for search. It is which component failed to preserve it.

Out of vocabulary is meaningful only relative to a named vocabulary. An analysed query term can be absent from the index vocabulary, a complete word can be absent from an older word-level model’s vocabulary, or a string can be present as model tokens without having acquired a useful retrieval representation. Those are not the same condition.

Table 10.2 — “Out of whose vocabulary?” separates literal OOV from a representation or training-exposure problem.

The first two questions concern literal vocabularies. The third does not. It describes a representation or training-exposure gap. Keeping it subordinate to the OOV investigation prevents “the tokeniser can encode the pieces” from being mistaken for “the retriever has learnt a useful representation.”

A query term can therefore be missing from the index but encodable by a model tokeniser. It can be tokenisable but poorly represented. It can appear exactly in the collection while being divided awkwardly by the model. The earlier sections on index analysis, word and subword tokenisation and the difference between model tokens and indexed terms explain why.

rizzlord is partly compositional. A model whose tokeniser produces useful pieces resembling rizz and lord, and whose training gave those pieces informative contextual representations, may infer a plausible meaning. Another model may split the same string very differently. A pure byte-level model can encode any valid UTF-8 form of iykyk, frfr or skibidi; a character-level model can do so only when its inventory or fallback covers the characters. As Appendix B explains, either may encode the expression without knowing how it is being used.

Tokenising a term is not the same as understanding it.

The reverse case is equally important. Suppose an identifier called DELULU-427 appears in three indexed documents and exact identity matters.

If the query and documents are analysed consistently, a lexical system can match the surface form directly. A keyword field might preserve the complete string; an ordinary text field might split it into delulu and 427. BM25 can give rare matching components substantial weight without ever having encountered the identifier during model training.

A standard single-vector dense retriever can usually encode the string as model tokens, but may blur it with related strings or concepts after contextualisation and pooling. Research on salient-phrase-aware dense retrieval documents this weakness while also showing that better training can mitigate it (Chen et al., 2022).

This is why a lexical path remains important for gene and protein identifiers, chemical strings, dataset titles, model names, article identifiers, product codes, personal names and newly coined terminology. The lesson is not that dense retrieval always fails on identifiers. It is that omitting exact lexical matching creates avoidable risk when exact identity determines relevance.

Out-of-distribution (OOD): when deployment differs from training#

A search tool may look excellent on ordinary questions yet repeatedly miss relevant legal or biomedical papers. It may find one answer-bearing passage but perform poorly when a librarian asks for all studies meeting review criteria. Every word can be familiar while the retrieval behaviour is wrong for the setting or purpose.

In neural IR, out of distribution is used broadly for situations in which deployment differs from the distributions or retrieval conditions represented during training. This can include shifts in queries or corpora and, in some IR work, shifts in retrieval task and relevance pattern.

Domain shift is a familiar case, but the retrieval literature does not use one universally fixed partition. For a librarian investigating a failure, three possible sites of shift are especially useful:

• Query shift: users ask in a different style, register, length or intent, or submit query types the training data scarcely contained.
• Corpus or domain shift: the collection differs in discipline, genre, language, document type, time period or terminology.
• Task or relevance shift: the deployment rewards a different kind of match. A model trained to find answer-bearing web passages may be used for high-recall evidence searching, known-item retrieval or citation discovery.

All the words in the new setting may be familiar. A model can therefore be OOD without encountering an OOV token and without facing a simple vocabulary mismatch. Conversely, a new discipline can create OOD transfer, specialised OOV strings and query–document vocabulary mismatch at the same time.

This is the other side of Chapter 5’s retrieval-training story. That chapter showed that positive pairs, negatives and training labels teach a model what should rank together. Once a model has learnt one distribution of queries, documents and relevance judgements, OOD asks how well those relationships transfer elsewhere.

The retrieval literature operationalises that problem more broadly than subject domain. BEIR (Thakur et al., 2021) was designed to test zero-shot generalisation across heterogeneous retrieval tasks and domains. MS-Shift isolates changes in query semantics, intent and length (Lupart et al., 2023). Work on generative IR additionally tests query variations, unseen query types, unseen tasks and corpus expansion (Liu et al., 2025).

A single poor result cannot prove OOD, especially when a vendor does not disclose the training distribution. The evidence becomes stronger when performance is plausible on familiar benchmark-like queries but degrades systematically for one discipline, collection, query style or retrieval purpose. That is why the practical response is comparative local evaluation, followed where possible by adaptation, retraining or replacement—not simply adding another synonym.

BM25 has a limited kind of robustness here. Its matching rule does not import a learnt semantic interpretation from another dataset, and its principal term statistics come from the target collection. It still depends on analysers, fields, parameters and query formulation, and it cannot bridge different terminology by itself. BEIR found it a robust zero-shot baseline while some more expensive neural and reranking approaches performed better on average. That is evidence about those benchmark tasks, not a law that lexical systems always transfer well or dense systems always transfer badly.

A learnt retriever may transfer its theory of relevance badly.

OOD is the question of whether what was learnt travels to this retrieval situation.

When every word is known: failures of negation, relations and composition#

Sometimes no vocabulary is missing, the query need not be OOD, and lexical overlap is nearly perfect. What fails is the relationship among the pieces.

Compare three retrieval requirements:

• A causes B, not merely a passage saying that B causes A.
• The intervention does not increase risk, not one saying that it does increase risk.
• Two concepts participating in the required relationship, not simply occurring independently in the same passage.

A bag-of-words lexical score preserves the presence of the terms but not necessarily their direction, scope or roles. BM25 may score the opposed passages similarly because the lexical evidence is almost identical. Strict Boolean preserves logic that the searcher states explicitly, but ordinary term conjunction still does not encode who caused what or which proposition a negation governs.

Phrase operators, positional indexes, proximity scoring and n-grams can preserve adjacency and local order. They are valuable callbacks to Chapters 2 and 3, but adjacency alone does not solve causality, negation or relation binding.

Single-vector dense retrieval can also blur a decisive local difference when it pools a whole query or passage into one representation. A cross-encoder reads query and candidate jointly and therefore provides the strongest architectural opportunity to model these relationships. ColBERT-style late interaction preserves more token-level evidence than a pooled bi-encoder, but it still encodes query and document separately and can fail badly on logic and negation. NevIR (Weller et al., 2024) found cross-encoders strongest and late-interaction models next, while most evaluated systems still performed at or below a random ranking baseline. Neither architecture guarantees correct logic, and neither can rescue a record omitted from the first-stage shortlist.

This is a capability failure rather than one formally standard IR category. It is still a useful diagnostic lens because the intervention differs from synonym expansion or domain adaptation. BoolQuestions (Zhang et al., 2024) and NevIR document why Boolean-logic and negation tests belong in a retrieval evaluation.

All the terms can be present while the relationship that determines relevance is lost.

Diagnose before choosing a remedy#

A result screen supplies symptoms, not causes. Work from a known relevant record or a controlled contrast whenever possible, confirm the record was eligible and inside the result boundary, then ask:

1. Does the relevant record simply use different terminology? Investigate vocabulary mismatch.
2. Does an identifier, new coinage or specialised string disappear or behave inconsistently? Ask out of whose vocabulary, and inspect index fields and analysis separately from model tokenisation and representation.
3. Does the model work plausibly elsewhere but poorly for this discipline, corpus, query style or retrieval purpose? Investigate OOD transfer, and compare performance on a local query set before adapting or replacing anything.
4. Are all the words present but their relationships wrong? Investigate negation, relations and composition.

Table 10.1 gives the remedy worth testing first in each case. Reach for it only after the question above it has an answer: the order matters more than the list, because the same remedy applied to the wrong diagnosis is what makes a search look unfixable.

The answers can overlap. A newly coined chemical identifier may be OOV for one component, weakly represented by another and concentrated in a corpus unlike the model’s training data. A specialised query may combine OOD transfer with a logic failure. Diagnosis therefore means accumulating evidence and testing interventions.

What different first-stage approaches preserve#

Every diagnosis above ends at the same place: something the retrieval method did not preserve. That is worth stating directly, because it is what separates the four lenses from a list of complaints about products.

Table 10.3 — Three contrasting first-stage retrieval patterns used throughout this book: what each preserves, its characteristic strength and its blind spot.

These are three reference patterns, not a complete list of first-stage methods. Learnt sparse retrieval and ColBERT-style late interaction can also operate at the first stage, as Chapter 7 explained. Query transformation changes or creates retrieval inputs; retrieval control routes them. Rerankers compare a shortlist after retrieval. Agentic search describes how subsequent actions are selected and sequenced. A real system may combine all of them.

That is why asking whether a product “uses vectors” tells us very little. The same holds for most of the vocabulary a vendor is likely to use: Appendix D sets out which labels name a representation, which name a matching method, which name a pipeline stage, and which name only an interface or an aspiration.

No first-stage approach preserves everything equally well. Hybrid and multi-stage systems exist partly because there is no single representation that eliminates vocabulary, distribution and composition problems. The practical conclusion is diagnostic: determine what evidence was lost before choosing Boolean expansion, lexical retrieval, dense retrieval, adaptation, reranking or hybridisation.

Relevance is a judgement, not a property#

No record is relevant in itself. It is relevant to someone, for some purpose, at some point in their work. The same paper can be exactly what a researcher tracing a method needs, useless to an undergraduate who wanted an overview, and excluded on sight by a systematic reviewer screening against inclusion criteria. Three verdicts, one record, and none of them wrong.

This has a consequence that governs everything else in the chapter. A retrieval system cannot be correct. It can only be closer to or further from somebody’s judgement about somebody’s need. Every number below is downstream of that judgement, and inherits whatever the judge understood the question to be.

Librarians are already practised at this. A reference interview is relevance elicitation: establishing what would count as a useful answer before going to look for one. What research evaluation adds is only that the judgements get written down, so that two systems can be compared against the same ones.

It is worth separating two things that both get called relevance. A record can be on topic—about the right subject—and still be no use, because it has been superseded, retracted, studies the wrong population, or answers a neighbouring question. Most retrieval metrics measure the first and are silent about the second. That gap is not a flaw in the metrics; it is a limit on what they are evidence for.

A metric does not measure relevance. It measures agreement with whoever judged relevance.

Precision and recall, and the choice librarians already make#

Given a set of results and a set of judgements, two questions can be asked. Of everything the system returned, how much was useful? And of everything useful, how much did the system return? The first is precision; the second is recall.⁴²

Table 11.1 — Precision and recall: what each counts, what it costs, and which library task leans on it.

The two errors described in Chapter 2 can now be named. A weak alternative added to an OR admits records that should not have been competing, which costs precision. A relevant record failing a compulsory AND never enters the result set at all, which costs recall. The chapter argued that the second error was the more serious one; the vocabulary here says why. Precision can be repaired by looking further down a list. Recall cannot be repaired by anything downstream.

The trade between them is real and unavoidable. Any system can achieve perfect recall by returning the entire database, and near-perfect precision by returning only the one record it is most confident about. Neither is a search. Every practical retrieval configuration is a position taken between those two failures, and the right position depends entirely on the task—which is why the two rows of the table above describe recognisably different working days.

This also settles something left loose in Chapter 7. A first-stage retriever is fast, shallow and broad; a reranker is slow, careful and narrow. Their division of labour is exactly the trade above, split across two stages. The first stage decides recall, because it decides which documents exist as far as the rest of the pipeline is concerned. Everything after it is competing for precision within that set.

Every stage after the first can improve precision. None of them can improve recall.

Recall has an awkward practical problem that precision does not. Measuring it requires knowing how many relevant records the collection holds, and in a live database of hundreds of millions of records nobody knows that number. Precision only requires judging what came back. This asymmetry is why serious searchers rarely measure recall in the absolute and instead compare: run two strategies and see what each finds that the other missed, or seed the evaluation with records already known to be relevant and check how many are recovered.⁴³ Both are practical. Neither yields the true denominator, and a reported recall figure should always prompt the question of where its denominator came from.

Rank changes the question#

As Chapter 3 established, ranked retrieval makes position part of the result. Precision and recall treat a result list as a bag: a record is either in it or not, and the order is irrelevant. That is a reasonable description of a Boolean result set exported for screening. It is a poor description of a ranked list, where position strongly determines what gets looked at, and where the difference between first place and fortieth matters enormously even though both are in the set.

The simplest repair is to stop measuring the whole list. Precision@k is precision calculated over the first k results only—precision@10 being common because ten roughly matches a first page. Here k is an evaluation depth: it need not equal the retriever’s top-k output boundary or a reranker’s candidate budget. The measure is easy to explain and captures what informal first-page comparisons are usually doing.

Its weakness is that it treats every position inside the cut-off as equal: a relevant record at rank 1 and the same record at rank 10 score identically, and anything at rank 11 scores nothing at all. Three refinements are common enough that a librarian will meet them in vendor material and research papers, and each encodes a different idea of what a good ranking is.

Mean reciprocal rank (MRR) looks only at where the first useful result appears, and rewards it for appearing early. It suits tasks with essentially one right answer—finding a known paper, resolving a citation, answering a factual question—and it is the natural metric behind a complaint that the right record was there, just too far down. It is blind by construction to everything after that first hit, so it says nothing about whether a system found one relevant record or fifty.⁴⁴

Mean average precision (MAP) rewards a system for placing many relevant records high, not merely the first one. It is the closer fit for the literature-search tasks librarians support, where the aim is a good set rather than a single answer. Its limitation is that it treats relevance as binary: a record is relevant or it is not, and a marginally useful paper counts the same as a decisive one.⁴⁵

Normalised discounted cumulative gain (nDCG) removes both restrictions. It accepts graded judgements—highly relevant, partly relevant, not relevant—and discounts each record’s contribution by how far down the list it sits, so a decisive paper at rank 2 counts for more than a marginal one at rank 9. It is the metric most often quoted when a retrieval system reports an improvement, and the one most worth recognising for that reason alone.⁴⁶

Table 11.2 — Rank-aware metrics: what each rewards, what it ignores, and when a librarian would ask for it.

Two cautions about the linked explanations above. They are written for recommender systems, so they speak of users and recommendations where this book would say queries and results; the arithmetic is identical and the substitution is safe. And all four measures need judgements before they can produce anything at all, which returns the problem to the first section of this chapter: someone has to decide what counted.

Quoting a metric is choosing a definition of better. Ask which one, and whether it is the one your task cares about.

Test collections, and what a benchmark number is worth#

To compare two retrieval systems, something has to be held still. The standard arrangement fixes three things: a collection of documents, a set of topics or queries, and a set of relevance judgements pairing the two. Run both systems over the same collection with the same topics, score them against the same judgements, and the difference is attributable to the systems. That arrangement is the Cranfield paradigm, named for the experiments that established it in the 1960s, and essentially all retrieval evaluation since has been a variation on it.⁴⁷

One step in it does not scale. Judging every document in a collection against every topic is impossible once the collection is large, so evaluations instead use pooling: run many different systems, collect the union of their highest-ranked results, judge that pool, and treat everything unjudged as not relevant.

That last move is a convenience with a consequence worth stating plainly. Unjudged is not the same as irrelevant, but the scoring cannot tell them apart. A retriever that surfaces something genuinely useful which no pooled system found is scored as though it found nothing at all—which biases the measurement, mildly and systematically, against approaches unlike the ones already in the pool.⁴⁸

This is the machinery behind the BEIR result cited when OOD transfer was introduced. BEIR assembles retrieval tasks from a range of domains and tests models on collections they were not trained on. Reading that finding properly: BM25 held up across collections it had never seen, judged by other people’s assessors, on other people’s topics—and several more expensive neural systems did better on average across the same set. Both halves of that sentence are conditional on the same apparatus.

Which is why a leaderboard position transfers so poorly to a library. It is the measurement counterpart of the argument already made about training data: a model trained on one literature learns that literature’s notion of relevance, and a model evaluated on one set of topics has been shown to work for those topics. Neither says much about a collection of chemistry theses, a non-English corpus, or the queries your researchers actually type.

A benchmark result is evidence about a benchmark. It becomes evidence about your library only when your queries resemble its topics.

Building an evaluation set you own#

The practical response is not to distrust benchmarks but to build a small one of your own. It need not be a research instrument. What a library needs is closer to a regression test: a fixed set of queries whose good answers are known, which can be rerun whenever something changes.

The queries. Thirty to fifty is enough to be informative and few enough to be rerun by one person in an afternoon. Take them from real practice—reference enquiries, interlibrary loan requests, search consultations, questions from database training—so that they carry the disciplines, languages and document types the library actually holds. Include, deliberately, the searches already known to go badly. Those are the ones a vendor demonstration will never contain, and the ones a regression will show up first.

The judgements. For each query, record the handful of items that ought to surface. Exhaustive judgement is neither achievable nor necessary here; known relevant records are enough to detect a system getting worse. Where staff can distinguish a decisive record from a merely useful one, mark that too—it is what makes graded measures available later.

The depth. Judge to a fixed cut-off, ten or twenty results, and keep it fixed. A stable depth is what makes the exercise both affordable and comparable across runs; a depth that drifts turns a regression test back into an anecdote.

Four probes are worth planting on purpose, because each tests a failure this book has already described and no vendor demonstration will include them:

1. An identifier, gene name, chemical string or product code, where exact identity decides relevance—the gap case, and the quickest test of whether an exact lexical route survives.
2. A phrase occurring only in the full text of a paper you hold, which tests the indexed unit rather than the ranking.
3. A query whose known relevant record shares almost none of its words—the mismatch case, and the one semantic retrieval is supposed to fix.
4. Two near-identical queries with opposite meaning, to see whether composition survives.

What this produces is not a score worth publishing. It is a baseline. Rerun it after a model update, an index change, a platform migration or before a renewal, and a regression that no release note mentioned becomes visible—which is precisely the failure mode a library has no other way to catch, since the system underneath a saved search can change without notice.

The obvious objection is that such a set measures your queries against your judgements and generalises to nobody. That is correct, and it is the point rather than a defect. It is the only evaluation whose definition of relevance the library chose.

What a number cannot settle#

Measurement answers a narrow question well: on these queries, against these judgements, which system ranked better. It does not explain why one lost—that is the diagnosis of Chapter 10, and a metric will happily report a decline without indicating whether the cause was vocabulary mismatch, an OOV or representation problem, OOD transfer, a logic failure or a changed cut-off.

Averages are the most common way to be misled. A benchmark gain of two points is entirely compatible with a system that has become systematically worse on one discipline, one language or one document type, because the mean absorbs it. The per-query differences are where the information is: a handful of queries that moved sharply matters more than a small shift in the average, and a difference smaller than the run-to-run variation is not a difference at all.⁴⁹

So when a vendor reports an improvement, the useful questions are the ones this chapter has supplied. On which queries, drawn from where? Judged by whom, to what depth, and with graded or binary judgements? Which metric, and against which baseline? And, most usefully of all: does the gain survive on the queries the library already knows to be hard?

None of this requires reading a loss function or running a significance test. It requires knowing that a number is a claim about a specific comparison, and asking which comparison it was.

The preceding sections described how search systems transform queries, select candidates, combine retrieval routes, rerank results and sometimes choose their own next action. Three consequences follow for libraries. Searches must be documentable. Retrieval decisions must be inspectable enough to support an accurate mental model. And products must be evaluated as complete pipelines rather than by their marketing labels.

The three sections below take those in turn: what a library must be able to record, what it must be able to see, and what it should therefore require and test before licensing anything.

Reproducibility depends on what is fixed—and what is recorded#

Librarians often use reproducible to mean that another searcher can understand what was done and rerun it. No retrieval method guarantees an identical result set forever. Database coverage, indexing, controlled vocabulary, ranking configurations and model versions can all change.

The useful question is therefore not simply “Is this search reproducible?” but:

Which parts of the retrieval pipeline were fixed, documented, exportable and versioned?

Three practical points follow. Reproducibility is conditional: the same query does not guarantee the same records or the same ranking once indexes, models and configurations move. Record what the system actually searched: saving the natural-language question is not enough when the system generated a Boolean query, expansions, subqueries, hypothetical text or an agentic sequence. Separate rerunning from comparison: an exact future recreation may be impossible, while a saved query, trace and exported result set still let a later run be compared against the original.

At minimum, a record should cover four areas of the pipeline:

Table 12.1 — The four pipeline areas a reproducible search record must cover.

LLM-based rewriting adds a specific reproducibility risk. Sampling settings can make generation deliberately variable, while model updates, prompt changes and hosted inference infrastructure can affect outputs even when a temperature of zero or greedy decoding is used. This does not mean that every LLM call must vary; it means that identical output should not be assumed unless the complete system provides and tests that guarantee. A technical account of why deployed inference can vary is provided in Thinking Machines Lab’s discussion of LLM nondeterminism.

Across all methods, save the search date, reported count, retrieved identifiers and—where possible—a raw export. When an LLM rewrites the query, saving only the natural-language question is insufficient: preserve the exact internal query, subqueries, expansion terms or hypothetical document actually passed to retrieval. A rerun may then be compared with the original even when the live database or model has changed.

Determinism is conditional on a fixed system. LLM output is not automatically deterministic, and reproducibility is also a documentation practice. Transparency helps explain a result, but it does not guarantee that a future system will return the same records.

This aligns with PRISMA-S, which recommends reporting the complete strategy, database and platform, limits, dates and result counts, while recognising that some proprietary or similarity-based discovery operations may not be fully replicable.

Interpretability requires an inspectable retrieval pipeline#

Documentation records what was done. It does not, on its own, explain why any particular record appeared. That is a separate property, and it needs separate vocabulary.

Transparency means that information about the system is exposed. Interpretability means that a librarian can trace why a record was included, excluded, scored or reranked. Understandability means that the librarian can form an accurate, usable mental model of the system without having to inspect every equation or model weight.

These qualities overlap, but they are not interchangeable. A vendor can publish a broad description of its ranking system without making a particular result interpretable. A detailed score display can still be hard to understand. And a search may be fully documented for a later rerun while leaving the reason for an individual result opaque.

Can a librarian trace why a record was included, scored, reranked or excluded—and identify which parts of that account remain unknown?

Three distinctions do most of the work in answering it. A highlighted term is not necessarily the cause of a result. A generated explanation is not necessarily a faithful execution trace. And an explanation can be perfectly understandable without exposing a single model weight.

A faithful explanation is an execution trace: it is derived from the actual query, index, candidates and scoring path used for that result. A plausible post-hoc explanation may still be helpful for orientation, but it should be labelled as an interpretation rather than presented as proof of what the system used. Highlighted words and generated rationales are especially easy to mistake for causal evidence when they may only describe a result after it was chosen.

What a useful inspection trail contains:

1. Original query: what the searcher entered, including fields, limits and filters.
2. Transformed query: the parsed, expanded, decomposed or generated form actually sent onward.
3. Candidate source: which retriever, indexed unit and route supplied the record.
4. Matched evidence: Boolean clauses, lexical contributions, token alignments or other evidence the system can faithfully expose.
5. Scoring and reranking: the stages that ordered the candidate and any cut-offs or approximate-search settings that shaped the shortlist.
6. Fusion and display: how component results were combined and how deduplication, diversification, snippets or labels changed what the searcher finally saw.
7. Trajectory, where the system is agentic: the sequence of actions actually taken—each query, each source, each route—together with the observations that informed the next step and the point at which searching stopped.

This is also why a short explanation need not expose every engineering detail. The aim is an accurate mental model and an actionable audit trail: enough information to diagnose an unexpected result, compare retrieval routes and recognise the limits of the explanation.

A search can be reproducible yet poorly understood if the saved query and identifiers do not reveal how records were selected or ranked. A clear-looking explanation does not make a search reproducible unless the relevant inputs, versions and outputs can also be recorded and rerun.

Choosing and governing search tools#

Most librarians will never train a retriever. They will license one, or license a database containing one. Documentation and inspection are therefore not only personal practices; they are things a library has to require in advance, because neither can be retrofitted to a product that was not built to expose them.

When a product advertises semantic, natural-language or AI search, the distinctions above become five groups of procurement and evaluation questions. Vendor answers will arrive in vendor vocabulary; Appendix D is a crosswalk from those labels to the mechanisms below, and is worth having to hand while the questions are being asked.

Before comparing query boxes, ask what the product accepts as a retrieval starting point: only text, or also controlled fields, a seed record, citation links, saved judgements and API predicates? For an agentic mode, ask whether both the task brief and the queries generated from it can be exported. An attractive text box may conceal that the product narrowed the search method before ranking began.

Query interpretation, transformation and control

1. What does the system infer, change and decide? Does it recognise fields, filters or intent without changing the input; correct, map, expand or generate retrieval inputs; choose retrieval routes; or construct a later query from earlier results? Can users inspect and export the interpretation, every transformed input, each route and any feedback records?

2. Who controls the retrieval trajectory? Is the sequence fixed in advance, governed by rules written in advance, or selected by a model from intermediate results? If a model chooses, what tools, sources and iteration limits bound it? Can users see every query it ran, every source it consulted, every route it took and the point at which it decided to stop?

Retrieval coverage and architecture

3. Which retrieval and ranking components are used? Is there Boolean retrieval, BM25-like lexical ranking, dense retrieval or reranking? Do the components operate in parallel, sequentially or both, how many candidates survive each cut-off, and does the system select components by query or always combine them?

4. What role does the neural model perform? A vendor naming a model has not thereby described an architecture. Is it parsing or rewriting the query, producing the first-stage query and passage vectors, expanding the vocabulary of a sparse index, jointly reranking candidates, or generating the final answer? These are different jobs with different failure modes, and a model can be present without doing the one you assumed.

5. What is the indexed unit? Is the system retrieving titles, abstracts, passages, sections or complete articles? A product searching abstracts cannot retrieve evidence that occurs only in the full text, however sophisticated its vectors are.

6. How are passage results assembled? If five chunks from the same paper match, are their scores combined, is only the best retained, or are five near-duplicate results shown?

7. How is the hybrid combined? A product may call itself hybrid because it contains two retrievers even when one signal rarely affects the final ranking. Ask where component lists are fused, whether a later stage reranks the merged candidates, what cut-offs apply, and whether users can inspect, compare or control the component result sets.

Exactness and failure testing

7. Is exact lexical matching retained? Can users search identifiers, chemical strings, gene names, dataset titles and quoted phrases without semantic expansion quietly changing the task?

8. How is text tokenised and analysed? Ask how the lexical path treats case, punctuation, hyphens and identifiers, and whether query and document analysis are compatible. For a dense model, ask whether it uses word, subword, character or byte inputs and how unfamiliar or newly coined strings are tested.

9. How are exactness, negation and exclusions tested? Ask for examples involving identifiers, contradictory claims, NOT-like intent and near-duplicate names — not only attractive paraphrase demonstrations.

10. What trained the relevance model? Web queries, question–answer pairs, citation links and scholarly relevance judgements teach different behaviours. How was the model adapted to the disciplines and document types being searched?

Auditability and repeatability

11. Can users inspect why a result appeared? Can they move from the displayed record backwards through the retrieval route, transformed inputs, inferred constraints, any feedback records, matched evidence, ranking or reranking stages, and final fusion or display rules? Does the product distinguish a faithful execution trace from a helpful but post-hoc explanation?

12. Can the search be documented and rerun? Can users export the original input; exact generated Boolean query, filters, expansions, subqueries or hypothetical text; retrieval route; ranking mode; model version; index date; k or threshold; and retrieved identifiers? When the same search is rerun, does the system reuse the saved transformation or ask an LLM to generate it again?

13. Are model, index and ranking changes disclosed? Does the vendor announce when an update may change results a user previously obtained, and is there any way to tell after the fact which version produced a saved search?

Institutional evaluation

14. How is performance evaluated, and on whose queries? Average benchmark scores can hide failures on the query types that matter locally. A serious evaluation uses the institution’s own disciplines, vocabulary, languages, document types and known relevant records.

15. Are failure cases part of the evaluation? A demonstration chosen by a vendor shows what the system does well. The searches that matter for procurement are the ones a library already knows to be difficult.

16. Can the library compare behaviour across an update? If a saved set of local test queries can be rerun after a model, index or ranking change, a library can detect a regression that no release note mentions.

None of these questions requires reading a loss function. All of them distinguish a retrieval design from a marketing label.

Libraries do not need access to every model weight or engineering detail. They need enough visibility to establish what was actually searched, enough documentation to rerun or compare it later, and enough control to test whether the system works for their own collections and users.

The tokeniser may be able to encode every piece of the query. The source article may be divided into sensible chunks. Each chunk may receive a dense vector, while a separate lexical system indexes its analysed terms. The final results may blend both signals.

That still does not mean the retrieval system has any rizz.