Semantic Search vs Keyword Search: What Changed with AI

Ask a search box "what is the longest a line of code should be?" and the right answer, in Python's style guide, sits under the heading "Maximum Line Length". It never uses the word "longest". A search that matches words will struggle. A search that matches meaning will find it at once.

Now ask where __all__ should be placed in a module. This time the exact string is the whole point, and the search that matches words wins.

That is semantic search vs keyword search in two questions. Neither is the modern one or the old one. They are different instruments, and the useful question is which fails where.

Two ways to rank

Keyword search scores a document by the words it shares with the query. The standard formula is BM25, the default in search engines such as Elasticsearch. A document scores higher when it contains the query's words, more so when those words are rare across the collection, with adjustments so that repeating a word many times and being a long document do not win unfairly (Elastic: similarity settings).

Semantic search turns the query and every document into vectors with an embedding model and ranks documents by how close their vectors are to the query's. Because the model was trained to place similar meanings together, it can match synonyms, paraphrases and even misspellings, where a keyword engine can only find lexical matches (Sentence Transformers: semantic search). See embeddings explained.

A test

The script searches a small real collection: six Python Enhancement Proposals, public-domain documents from peps.python.org, split into 148 sections. It asks 40 questions, each written for this article and each answered by one known section. Three methods rank the sections: BM25 from the rank_bm25 library, a small embedding model, and a hybrid of the two.

import json
import re

import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer

sections = json.load(open("../../genai/pep_sections.json", encoding="utf-8"))
questions = json.load(open("../../genai/pep_questions.json", encoding="utf-8"))
ids = [s["id"] for s in sections]
texts = [s["title"] + "\n" + s["text"] for s in sections]


def words(text):
    return re.findall(r"[a-z0-9_]+", text.lower())


keyword_index = BM25Okapi([words(t) for t in texts])
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
vectors = model.encode(texts, normalize_embeddings=True)


def order(scores):
    return list(np.argsort(-np.asarray(scores), kind="stable"))


def search(question):
    by_keyword = order(keyword_index.get_scores(words(question)))
    by_meaning = order(vectors @ model.encode(question, normalize_embeddings=True))
    fused = np.zeros(len(texts))
    for ranking in (by_keyword, by_meaning):              # reciprocal rank fusion with k = 60
        for position, doc in enumerate(ranking, start=1):
            fused[doc] += 1 / (60 + position)
    return {"keyword": by_keyword, "meaning": by_meaning, "hybrid": order(fused)}


ranks = {"keyword": [], "meaning": [], "hybrid": []}
for q in questions:
    target = ids.index(q["answer_in"])
    for method, ranking in search(q["question"]).items():
        ranks[method].append(ranking.index(target) + 1)

print(f"{len(sections)} sections, {len(questions)} questions")
print("method    right section first   in the top 5   outside the top 10")
for method, r in ranks.items():
    r = np.array(r)
    print(f"{method:8} {np.sum(r == 1):>19}   {np.sum(r <= 5):>12}   {np.sum(r > 10):>18}")

print()
print("rank of the right section:  keyword  meaning  hybrid")
for i in (2, 18, 7, 23, 26):
    print(f"{questions[i]['question'][:52]:52} {ranks['keyword'][i]:>4} {ranks['meaning'][i]:>8} {ranks['hybrid'][i]:>7}")

148 sections, 40 questions
method    right section first   in the top 5   outside the top 10
keyword                   19             29                    9
meaning                   21             33                    6
hybrid                    21             36                    4

rank of the right section:  keyword  meaning  hybrid
What is the longest a line of code should be?          14        1       2
How should functions be named?                         70        2      13
Where should __all__ and __version__ be placed in a     1        4       2
Where does the phrase 'Readability counts' come from    1        6       1
How can I give a complicated type a shorter name?      88       15      40

• Overall, meaning beat keywords, but not by much. The embedding model put the right section first for 21 questions of 40, BM25 for 19.
• Combining them was better than either. The hybrid had the right section in its top five for 36 questions, against 33 and 29, and left only 4 outside the top ten.
• "The longest a line should be": keyword rank 14, meaning rank 1. The section says "maximum" and "79 characters". No shared words, same idea.
• "How should functions be named?": keyword rank 70. Every word in the question is common across documents about Python, so BM25 had nothing distinctive to hold on to.
• The question about module-level names: keyword rank 1, meaning rank 4. __all__ is an exact, rare string, which is what keyword search is for.
• "Readability counts": keyword 1, meaning 6. A quoted phrase is a lexical question.
• "Give a complicated type a shorter name": both failed. The answer is under "Type aliases", and nothing in the question says alias. This is a vocabulary gap that neither method closed.

Forty questions on one collection is a small test. It is enough to show the general point: the two methods make different mistakes.

Hybrid search

Because the mistakes differ, running both and merging the results is the usual production answer. Elastic's documentation recommends it and suggests a method called reciprocal rank fusion for the merge (Elastic: hybrid search).

Reciprocal rank fusion ignores the scores, which are on different scales and cannot be compared, and uses only the positions. Each document receives one divided by a constant plus its rank in each list, and the sums decide the final order (Elastic: reciprocal rank fusion). A document that is near the top of either list ends up near the top overall. It needs no tuning, which is much of its appeal.

Hybrid is not free of trade-offs. In the test, "how should functions be named" fell from rank 2 under meaning alone to rank 13 in the hybrid, pulled down by a keyword rank of 70. Fusion improves the average and can hurt an individual query.

Which to use

1. Start with keyword search if the content is full of exact terms that people type: product codes, error messages, function names, legal references. It is cheap, fast and explainable.
2. Add semantic search when people ask in their own words, when queries are questions, or when the vocabulary of users and documents differs.
3. Use hybrid for anything feeding a language model. A RAG system has to cope with both kinds of query, and a missed passage becomes a wrong answer. See RAG architecture explained.
4. Measure before choosing. Thirty or forty real questions with known answers will tell you more than any comparison article, this one included. See retrieval evaluation.

Two things often matter more than the choice of method. How documents are split decides what can be found at all: see chunking strategies for RAG. And at scale, the vectors need an index built for nearest-neighbour search: see vector databases explained.

Common mistakes

• Replacing keyword search with embeddings and losing exact-match queries that used to work.
• Judging by a few favourite queries instead of a test set.
• Comparing raw scores across methods. A BM25 score and a cosine similarity are not on the same scale.
• Using an embedding model that does not know your domain's terms and assuming meaning will carry it.
• Ignoring the vocabulary gap. If users say "shorter name" and the documents say "alias", add synonyms, better titles or a rewriting step. No ranking method fixes missing words by itself.

Semantic search and document indexing are part of the Vector Databases and Retrieval module of the Generative AI course.

Sources

• Semantic Search (Sentence Transformers documentation) — Sentence Transformers
• Similarity settings (Elasticsearch Reference) — Elastic
• Hybrid search (Elastic Docs) — Elastic
• Reciprocal rank fusion (Elasticsearch Reference) — Elastic
• rank_bm25: a collection of BM25 algorithms in Python (GitHub) — Dorian Brown
• Python Enhancement Proposals (Python Software Foundation) — Python Software Foundation