Interactive companion
BM25 treats matching words as weighted clues and adds their contributions into a graded score. This laboratory lets you change the evidence and watch the ranking respond. Editing a record changes its length and the collection average, so other records’ scores can change too.
The tour fixes query and scoring parameters; use its prescribed controls, then switch to the sandbox to experiment freely.
No stemming, no stop-word removal, no phrase or field weighting — plain word matching, so you can see exactly what BM25 itself contributes.
Click a term to switch its evidence off. The bar next to each term shows how unusual it is in this collection.
deluluA real pipeline hands only the first k records to whatever comes next. Records below the line were retrieved and scored; they were simply not passed on.
Watch Record D's delulu segment. The first repetitions matter; the twentieth barely registers.
For the guided-tour query, this adds no matching words. It changes the document length and collection average; sandbox queries may match words in the added text.
| f(q,D) | How many times the term occurs in this document |
| IDF(q) | How unusual the term is across the collection |
| |D| ÷ avgdl | This document's length relative to the average |
| k₁ | How quickly repetition stops helping |
| b | How strongly document length matters |
Defaults are k₁ = 1.2 and b = 0.75. Nothing in the guided tour depends on changing them. Set b to 0 to switch document-length normalisation off entirely.
This lab uses the Lucene inverse-document-frequency form, ln(1 + (N − df + 0.5) ÷ (df + 0.5)), which stays positive. Other implementations use variants that can go negative for very common terms, so two systems both described as “using BM25” can rank differently.
What the engine does underneath these numbers — the term dictionary, the posting traversal, the top-k heap and the threshold θ that lets it skip work — is walked through in Appendix C, using the same scoring form as this page.
Changing k₁ and b here does not reproduce Primo, Scopus or PubMed. Those systems wrap lexical evidence of this kind inside considerably richer pipelines.
Companion to “How Search Decides What You See”. The collection is deliberately tiny and fully visible so that every number on this page can be checked by hand.