Embeddings Explained: The Foundation of Modern AI

A computer can compare two numbers easily. It cannot easily compare "How do I reset my password?" with "I forgot my login details". The two sentences share almost no words and mean nearly the same thing.

Embeddings solve this by turning text into numbers in a particular way: pieces of text with similar meaning get similar numbers. Once that is done, finding related text becomes arithmetic. Search by meaning, recommendations, clustering and retrieval for LLMs all build on it.

What an embedding is

An embedding is a list of numbers, a vector, produced by a model for a piece of text. OpenAI's documentation puts the idea in one line: the distance between two vectors measures how related the texts are, with small distances meaning high relatedness (OpenAI: vector embeddings).

The individual numbers mean nothing on their own. There is no position for "is about passwords". Meaning is carried by where the whole vector sits relative to others.

Think of a map. A single coordinate tells you little, but two cities with close coordinates are near each other. An embedding is a coordinate on a map of meaning, with hundreds of dimensions instead of two.

Seeing it

The script uses a small open model, all-MiniLM-L6-v2, which maps a sentence to 384 numbers (Hugging Face: all-MiniLM-L6-v2). It embeds six sentences, compares every pair, and then finds the closest sentence to three queries. It ran on a laptop.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
sentences = ["How do I reset my password?",
             "I forgot my login details and need to change them.",
             "Can I get my money back if I cancel?",
             "What is the refund policy for cancelled orders?",
             "The bank raised interest rates again.",
             "We sat on the river bank and watched the boats."]
vectors = model.encode(sentences, normalize_embeddings=True)
print("shape:", vectors.shape, "  length of each vector:", round(float(np.linalg.norm(vectors[0])), 3))
print("first sentence, first six numbers:", [round(float(x), 3) for x in vectors[0][:6]])

similarity = vectors @ vectors.T            # cosine similarity, because the vectors have length 1
print()
print("cosine similarity between every pair")
print("     " + "".join(f"{i + 1:>6}" for i in range(len(sentences))))
for i, row in enumerate(similarity):
    print(f"{i + 1:>4} " + "".join(f"{value:>6.2f}" for value in row))

print()
for query in ("I can't sign in to my account", "money and lending", "a walk by the water"):
    scores = vectors @ model.encode(query, normalize_embeddings=True)
    best = int(np.argmax(scores))
    print(f"{query!r:34} -> {best + 1}. {sentences[best]}  ({scores[best]:.2f})")

shape: (6, 384)   length of each vector: 1.0
first sentence, first six numbers: [0.007, -0.063, -0.072, -0.032, -0.043, 0.07]

cosine similarity between every pair
          1     2     3     4     5     6
   1   1.00  0.55  0.29  0.16  0.10  0.03
   2   0.55  1.00  0.19  0.12  0.02 -0.01
   3   0.29  0.19  1.00  0.61  0.20  0.02
   4   0.16  0.12  0.61  1.00  0.15 -0.06
   5   0.10  0.02  0.20  0.15  1.00  0.29
   6   0.03 -0.01  0.02 -0.06  0.29  1.00

"I can't sign in to my account"    -> 1. How do I reset my password?  (0.56)
'money and lending'                -> 5. The bank raised interest rates again.  (0.44)
'a walk by the water'              -> 6. We sat on the river bank and watched the boats.  (0.35)

• Six sentences became six rows of 384 numbers. The first six of them are shown, and they are not interpretable. That is normal.
• Sentences 1 and 2 score 0.55, although they share only the word "my". Both are about getting back into an account.
• Sentences 3 and 4 score 0.61. "Money back if I cancel" and "refund policy for cancelled orders" are the same question.
• Sentences 5 and 6 share the word "bank" and score 0.29. The model reads the word in its sentence, so a bank that sets interest rates and a river bank land far apart.
• Unrelated pairs are near zero. Password help and river boats score 0.03.
• Each query found its sentence without sharing its words. "I can't sign in" matched the password question. "A walk by the water" matched the river.

Measuring similarity

The usual measure is cosine similarity: the angle between two vectors, ignoring their length. It runs from 1 for the same direction, through 0 for unrelated, to negative values for opposed.

The script normalises each vector to length 1. After that, cosine similarity is a plain dot product, which is why the comparison is one line of matrix multiplication. Many systems store normalised vectors for this reason.

Two cautions about the numbers:

• Scores are relative, not absolute. 0.55 was a strong match here. Another model might put the same pair at 0.8. Compare scores within one model, and choose thresholds by testing, not by intuition.
• A score is not a probability. It ranks candidates. It does not say how likely they are to be correct.

Where they come from

The first widely used embeddings were for single words. Word2vec learned a vector for each word from the company it keeps in large amounts of text (Mikolov et al.: word2vec). Its weakness is visible in the example above: one vector for "bank" cannot serve both meanings.

Modern embedding models run the whole text through a transformer, so each word is read in context, and return one vector for the passage. Sentence-BERT showed how to train such models so that the vectors can be compared directly, and made large-scale comparison practical: finding the most similar pair among 10,000 sentences fell from about 65 hours to about 5 seconds (Reimers and Gurevych: Sentence-BERT).

The model used here was trained on more than a billion pairs of related sentences, pulling the members of each pair together and pushing unrelated sentences apart. See attention explained for the architecture underneath.

What they are used for

The first two are where most developers meet embeddings. See semantic search vs keyword search and RAG architecture explained. Storing and searching millions of vectors is the job of a vector database.

Choosing a model

There is no single winner. A large benchmark of embedding models across 8 kinds of task and 112 languages found that no method dominated all of them (Muennighoff et al.: MTEB). A model that leads on similarity may be ordinary at retrieval. Test on your own data.

Check the languages. The model in this article is for English. Text in Hindi or Telugu needs a multilingual model.

Check the input limit. This model truncates anything beyond 256 word pieces, roughly a long paragraph. Text past the limit is silently ignored, which is one reason documents are split into chunks before embedding. See chunking strategies for RAG.

Weigh the size. Longer vectors can hold more, and they cost more to store and search. 384 numbers per text is small. Hosted models commonly return 1,536 or more, and some let you shorten the vector to save space.

Do not mix models. Vectors from two models live on different maps. If you change the model, every stored text must be embedded again.

Common mistakes

• Comparing vectors from different models.
• Embedding very long documents whole. The tail is cut off, and what remains is blurred into one average.
• Setting a similarity threshold without testing it.
• Treating similarity as correctness. The nearest passage is the most similar one, not necessarily the one that answers the question.
• Using an English model on other languages.
• Embedding the query and the documents with different preprocessing.

Embeddings and similarity search are part of the Advanced LLM Concepts module of the Generative AI course.

Sources

• Vector embeddings (OpenAI API documentation) — OpenAI
• all-MiniLM-L6-v2 model card (Hugging Face) — Sentence Transformers
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (arXiv) — Reimers and Gurevych
• Efficient Estimation of Word Representations in Vector Space (arXiv) — Mikolov et al.
• MTEB: Massive Text Embedding Benchmark (arXiv) — Muennighoff et al.