LlamaIndex vs LangChain: Which Framework to Pick

"LlamaIndex or LangChain?" is one of the most common questions from people starting with LLM applications. The usual answers describe philosophies: LlamaIndex for data, LangChain for chains and agents. That is roughly right, but it hides how much the two now overlap, and it skips the question that matters more: does the choice change your results?

This comparison builds the same retrieval step in both frameworks, over the same documents with the same embedding model, and measures what comes out.

What each framework is for

LlamaIndex describes itself as a Python toolkit for building LLM-powered agents over your data. Its centre is context augmentation: getting your documents, databases and APIs in front of the model at the moment it answers, of which retrieval-augmented generation is the best-known form. It offers a high-level API that goes from documents to answers in a few lines, lower-level parts you can replace, and agents and workflows on top (LlamaIndex: framework).

LangChain now presents itself around agents. Its central piece, create_agent, is a minimal, configurable harness around the model loop: the prompt, the tools and middleware. LangGraph sits underneath for lower-level orchestration, and Deep Agents offers a fuller, ready-made agent (LangChain: overview).

Both connect to the same model providers, the same vector databases and the same embedding models. The difference is mostly where each one starts.

The same task in both

The script loads six Python Enhancement Proposals, which are in the public domain or CC0 (PEP 1), embeds them with the same small local model in both frameworks, and asks 40 questions. A question counts as answered if a known answer phrase appears in the three chunks retrieved.

import json
import statistics
from pathlib import Path

from transformers import AutoTokenizer

# Same documents, same embedding model, same questions: only the framework and the chunking change
EMBED = "sentence-transformers/all-MiniLM-L6-v2"
PEPS = {8: "pep-0008.rst", 20: "pep-0020.rst", 257: "pep-0257.rst", 484: "pep-0484.rst", 498: "pep-0498.rst", 572: "pep-0572.rst"}
texts = {n: Path("peps", f).read_text(encoding="utf-8") for n, f in PEPS.items()}
questions = json.loads(Path("pep_questions.json").read_text(encoding="utf-8"))
word_pieces = AutoTokenizer.from_pretrained(EMBED)

# ---- LlamaIndex: documents go into an index; chunking and embedding happen inside it
from llama_index.core import Document, Settings, VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.embed_model = HuggingFaceEmbedding(model_name=EMBED)
li_docs = [Document(text=t, metadata={"pep": n}) for n, t in texts.items()]


def llamaindex(**splitter):
    extra = {"transformations": [SentenceSplitter(**splitter)]} if splitter else {}
    index = VectorStoreIndex.from_documents(li_docs, **extra)
    retriever = index.as_retriever(similarity_top_k=3)
    chunks = [node.get_content() for node in index.docstore.docs.values()]
    return chunks, lambda q: [hit.get_content() for hit in retriever.retrieve(q)]


# ---- LangChain: you pick and connect the pieces yourself
from langchain_core.documents import Document as LCDocument
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

lc_embed = HuggingFaceEmbeddings(model_name=EMBED)
lc_docs = [LCDocument(page_content=t, metadata={"pep": n}) for n, t in texts.items()]


def langchain(chunk_size, chunk_overlap):
    splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
    chunks = splitter.split_documents(lc_docs)
    store = InMemoryVectorStore.from_documents(chunks, lc_embed)
    return [c.page_content for c in chunks], lambda q: [d.page_content for d in store.similarity_search(q, k=3)]


LIMIT = Settings.embed_model.max_length
print(f"the embedding model reads at most {LIMIT} word pieces of each chunk")
print(f"LlamaIndex default splitter: {SentenceSplitter().chunk_size} tokens, overlap {SentenceSplitter().chunk_overlap}\n")
print(f"{'setup':<30} {'chunks':>6} {'median size':>12} {'over limit':>11} {'text per question':>18} {'answer in top 3':>16}")
for name, (chunks, retrieve) in [
    ("LlamaIndex, default splitter", llamaindex()),
    ("LlamaIndex, 200-token chunks", llamaindex(chunk_size=200, chunk_overlap=20)),
    ("LangChain, 1,000 characters", langchain(1000, 200)),
    ("LangChain, 500 characters", langchain(500, 50)),
]:
    sizes = [len(word_pieces.tokenize(c)) for c in chunks]
    over = sum(s > LIMIT for s in sizes)
    found, sent = 0, []
    for q in questions:
        hits = retrieve(q["question"])
        found += any(q["phrase"].lower() in hit.lower() for hit in hits)
        sent.append(sum(len(word_pieces.tokenize(hit)) for hit in hits))    # what an LLM would have to read
    print(f"{name:<30} {len(chunks):>6} {statistics.median(sizes):>12.0f} {over:>11} "
          f"{statistics.median(sent):>18.0f} {found:>9} of {len(questions)}")

the embedding model reads at most 256 word pieces of each chunk
LlamaIndex default splitter: 1024 tokens, overlap 200

setup                          chunks  median size  over limit  text per question  answer in top 3
LlamaIndex, default splitter       69         1023          69               2896        28 of 40
LlamaIndex, 200-token chunks      338          190          10                525        28 of 40
LangChain, 1,000 characters       289          240         109                670        28 of 40
LangChain, 500 characters         593          107           0                304        27 of 40

What the output shows

• The framework made no difference. Every setup found the answer for 27 or 28 of the 40 questions. Same embedding model, same documents, same results.
• The defaults made a large difference elsewhere. LlamaIndex's default splitter makes chunks of 1,024 tokens. The embedding model reads only the first 256 word pieces of any text and ignores the rest (Hugging Face: all-MiniLM-L6-v2). So every one of the 69 default chunks was embedded from roughly its first quarter.
• Big chunks cost context. With default chunks, the three results for a median question held 2,896 word pieces of text for the model to read. With 200-token chunks, 525, for the same number of answers found. That is a fifth of the input tokens on every question, and less irrelevant text for the model to be distracted by.
• LangChain's units are different. Its recursive splitter measures chunk size in characters, not tokens (LangChain: splitting recursively). At 1,000 characters, 109 of 289 chunks were still longer than the embedding model reads, because these documents contain code, which turns into many word pieces per character.
• The test favours long chunks. A long chunk is more likely to contain an answer phrase somewhere. That is why the default setup could match the others despite the truncation. Measured by what the model has to read, it was the worst.

The general lesson is that retrieval quality comes from chunking, embedding and search settings, and both frameworks expose all of them. See chunking strategies for RAG.

How the code differs

The task is the same, but the shape of the code is not.

LlamaIndex is index-first. You hand documents to an index, and splitting, embedding and storing happen inside it, using global Settings for the models. Its sentence splitter tries to respect sentence boundaries (LlamaIndex: node parsers). The defaults are a convenience and a risk. The index embeds with OpenAI unless you set an embedding model, as the script does.

LangChain is assembly-first. You pick a splitter, run it, pick an embedding model and a vector store, and connect them. Nothing happens that you did not write, which takes more code and hides less.

Choosing

Mixing is common too: a LlamaIndex retriever can serve as a tool inside a LangChain agent.

Whichever you choose, measure retrieval on your own questions, and set the chunk size against the embedding model's limit rather than accepting a default. For the full picture of retrieval-augmented generation, see RAG architecture explained. For a first project in one framework, see the LangChain tutorial.

Common mistakes

• Choosing a framework to fix retrieval quality. The settings decide it, not the brand.
• Accepting default chunk sizes without checking the embedding model's input limit.
• Mixing units: tokens in one splitter, characters in another, word pieces in the embedder.
• Leaving default models in place and sending documents to a hosted API without meaning to.
• Judging retrieval by hit rate alone, ignoring how much text the model has to read.

LangChain and the other Gen AI frameworks are covered in the Gen AI Frameworks and Tools module of the Generative AI course.

Sources

• LlamaIndex Framework (LlamaIndex documentation) — LlamaIndex
• Node Parser Modules (LlamaIndex documentation) — LlamaIndex
• LangChain overview (LangChain documentation) — LangChain
• Splitting recursively (LangChain documentation) — LangChain
• all-MiniLM-L6-v2 model card (Hugging Face) — sentence-transformers
• PEP 1: PEP Purpose and Guidelines (Python Enhancement Proposals) — Python Software Foundation