How Large Language Models (LLMs) Work Under the Hood

A large language model does one thing. Given a piece of text, it produces a score for every token that could come next. A chatbot, a code assistant and a summariser are all built on that single step, repeated.

That sentence explains how LLMs work better than most diagrams, and it is easy to test, because small models are free to download and run. This guide runs one and reads the numbers.

The step

Stage 3 is the Transformer, the architecture built on attention that current language models share (Vaswani et al.: Attention Is All You Need). See how transformers changed AI for that part. This article follows the whole loop.

Watching it happen

The script loads GPT-2, a small open model from 2019. It has 124 million parameters and a vocabulary of 50,257 tokens (Hugging Face: GPT-2 model card). The models you use today are far larger and far better, but the mechanism below is the same.

It ran on a laptop, with no graphics card.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2").eval()
print(f"parameters: {sum(p.numel() for p in model.parameters()):,}   vocabulary: {tok.vocab_size:,} tokens")

ids = tok("The capital of France is", return_tensors="pt").input_ids
print("the prompt as token ids:", ids[0].tolist())
with torch.no_grad():
    scores = model(ids).logits[0, -1]            # one score for every token in the vocabulary
print("scores produced for the next position:", len(scores))


def likeliest(scores, temperature, k=5):
    probabilities = torch.softmax(scores / temperature, dim=-1)
    best = torch.topk(probabilities, k)
    return "  ".join(f"{tok.decode(i)!r} {p:.3f}" for p, i in zip(best.values, best.indices))


print()
for temperature in (1.0, 0.5, 2.0):
    print(f"temperature {temperature}:  {likeliest(scores, temperature)}")


def generate(ids, steps, sample):
    """Generation is the same step in a loop: choose a token, append it, run the model again."""
    for _ in range(steps):
        with torch.no_grad():
            scores = model(ids).logits[0, -1]
        if sample:
            choice = torch.multinomial(torch.softmax(scores, dim=-1), 1)
        else:
            choice = scores.argmax().view(1)
        ids = torch.cat([ids, choice.view(1, 1)], dim=1)
    return tok.decode(ids[0])


print()
print("always the likeliest token:", repr(generate(ids, 10, sample=False)))
for seed in (1, 2):
    torch.manual_seed(seed)
    print(f"sampled, seed {seed}:           {generate(ids, 10, sample=True)!r}")

parameters: 124,439,808   vocabulary: 50,257 tokens
the prompt as token ids: [464, 3139, 286, 4881, 318]
scores produced for the next position: 50257

temperature 1.0:  ' the' 0.085  ' now' 0.048  ' a' 0.046  ' France' 0.032  ' Paris' 0.032
temperature 0.5:  ' the' 0.397  ' now' 0.128  ' a' 0.118  ' France' 0.058  ' Paris' 0.058
temperature 2.0:  ' the' 0.005  ' now' 0.004  ' a' 0.004  ' France' 0.003  ' Paris' 0.003

always the likeliest token: 'The capital of France is the capital of the French Republic, and the capital'
sampled, seed 1:           "The capital of France is upset with Sir Bennington, London's top novelist"
sampled, seed 2:           'The capital of France is Madrid, which down that angry road dugins the'

• The prompt became five numbers. The model never sees letters. See tokenization in LLMs.
• The output is 50,257 scores, one for each token in the vocabulary. That list is all the model produces.
• There is no answer, only a distribution. The likeliest next token is the, at 8.5%. Paris is fifth, at 3.2%. A stronger model would put most of the weight on Paris, but it would still be a distribution.
• Temperature reshapes the same scores. At 0.5 the leader rises to 39.7%. At 2.0 the five are nearly level. Nothing about the model changes, only how boldly it is read.
• Always taking the likeliest token goes in circles: "the capital of the French Republic, and the capital". Hugging Face's documentation notes that this strategy starts to repeat itself on longer text (Hugging Face: generation strategies).
• Sampling is livelier, and one run says Madrid. The sentence is grammatical, confident and false.

That last line is the most useful thing in the output. The model was not lying and did not malfunction. It drew a plausible token from its distribution, and plausible is all it was trained to be.

Where the ability comes from

A model like this is made by pretraining: showing it enormous amounts of text and, at every position, asking it to predict the next token. Nobody labels anything. The text is its own answer key, which is why the method is called self-supervised (Hugging Face: how do Transformers work).

The parameters are the numbers adjusted during that training. Each wrong prediction nudges them slightly. After enough text, predicting the next token well requires having absorbed grammar, facts, styles and a good deal of how arguments and programs are put together. None of this is stored as sentences. It is spread across the parameters as tendencies.

A pretrained model continues text. It does not yet answer questions. Anthropic's documentation says such models are not inherently good at following instructions, and that further training, including fine-tuning and feedback from human raters, turns them into assistants (Anthropic: glossary).

From text-continuer to chat

A conversation with an assistant is still one piece of text. Behind the interface, the system's instructions, your messages and the model's earlier replies are joined into a single sequence with markers for who is speaking. The model continues that sequence, one token at a time, until it produces a marker that means it has finished.

This is why the wording of a prompt matters: it is the text being continued. See prompt engineering techniques.

The dials

Temperature controls how adventurous the choice is. Low values stay close to the likeliest tokens. High values give unlikely tokens a real chance. Use low values for extraction and code, higher ones for brainstorming.

Other sampling settings restrict the choice to the likeliest candidates, so that a very unlikely token cannot be picked by bad luck.

A maximum length stops the loop.

One caution. Even at temperature 0, a hosted model may not give identical output for identical input, and Anthropic's glossary says so plainly. If your application needs repeatability, test for it instead of assuming it.

What follows from the design

Four practical facts come straight from the mechanism.

Fluent is not the same as true. The model produces likely text. When the likely text is also correct, which is often, it looks like knowledge. When it is not, you get Madrid. See how to reduce LLM hallucinations.

It knows two things only: what training left in its parameters, and what is in the text in front of it. The second is the context window, a working memory that is separate from training. See context windows explained.

It does not learn from your conversation. Using a model does not change its parameters. Anything it seems to remember is text that was put back into the prompt.

New or private information has to be supplied. If a fact is not in the parameters, the reliable way to get it into an answer is to put it in the prompt. Doing that automatically is called retrieval-augmented generation. See RAG architecture explained.

Common misunderstandings

• "It looks the answer up." A plain model has no database. It can search only if it has been connected to a tool that does.
• "It decides what is true." It decides what is likely.
• "The same prompt gives the same answer." Only if nothing is sampled, and not always then.
• "A bigger model has no limits." Bigger models have sharper distributions and wider knowledge. The mechanism, and its failure mode, are unchanged.
• "It thinks before it writes." Each token is produced from everything before it. Models that appear to plan do so by writing their reasoning as more tokens.

How LLMs work, in training and at inference, is part of the Transformers and Large Language Models module of the Generative AI course.

Sources

• GPT-2 model card (Hugging Face) — OpenAI community on Hugging Face
• Attention Is All You Need (arXiv) — Vaswani et al.
• How do Transformers work? (Hugging Face LLM Course) — Hugging Face
• Generation strategies (Hugging Face Transformers documentation) — Hugging Face
• Glossary (Claude Platform Docs) — Anthropic