Few-Shot vs Zero-Shot Prompting: A Practical Guide

There are two ways to tell a language model what you want. You can describe the task, or you can show it. Zero-shot prompting does the first: an instruction and nothing else. Few-shot prompting adds a handful of worked examples before the real input.

Few-shot vs zero-shot prompting is usually presented as a simple upgrade: add examples, get better results. Often that is true. The test in this guide shows it is not a law, and why you should measure.

The two approaches

The idea that a model can pick up a task from a few demonstrations in its prompt, with no training at all, was the headline result of the GPT-3 paper (Brown et al.: Language Models are Few-Shot Learners). Nothing in the model changes. The examples are simply more text to continue from. See how LLMs work.

What examples are for

Vendor guidance agrees on the main use. Google's documentation says few-shot prompts are often used to regulate the formatting, phrasing and scoping of responses, and recommends including examples (Google: prompt design strategies). Anthropic calls examples one of the most reliable ways to steer output format, tone and structure, and suggests three to five that are relevant and varied (Anthropic: prompting best practices).

Both add warnings. Google notes that too many examples can make the model overfit to them. Anthropic asks for diversity so that the model does not pick up unintended patterns. The test below runs straight into both.

A test

The task is sorting customer messages into six intents, all about bank cards and easy to confuse: a card that has not arrived, a question about delivery time, a lost card, a card that does not work, a declined payment, and activation. The messages are real ones from Banking77, a public dataset of banking queries (Casanueva et al.: Banking77), published under a Creative Commons licence (PolyAI: task-specific datasets).

A small open model classifies the same 48 messages under six different prompts.

import csv
import json
import random
import urllib.request
from collections import Counter

LABELS = {"card_arrival": "a card was ordered and has not arrived yet, or the customer wants to track it",
          "card_delivery_estimate": "the customer asks how long delivery will take or needs the card by a certain date",
          "lost_or_stolen_card": "the card has been lost or stolen",
          "card_not_working": "the customer has the card and it does not work",
          "declined_card_payment": "a payment made with the card was declined",
          "activate_my_card": "the customer wants to activate a card they have received"}


def load(name, per_label):
    rows = list(csv.DictReader(open(f"../../genai/banking77/{name}.csv", encoding="utf-8")))
    return [(r["text"], label) for label in LABELS
            for r in [r for r in rows if r["category"] == label][:per_label]]


def ask(prompt):
    body = {"model": "llama3.2:3b", "prompt": prompt, "stream": False,
            "options": {"temperature": 0, "seed": 42, "num_predict": 12}}
    request = urllib.request.Request("http://localhost:11434/api/generate", data=json.dumps(body).encode(),
                                     headers={"Content-Type": "application/json"})
    return json.load(urllib.request.urlopen(request, timeout=600))["response"].strip().split("\n")[0].strip(" .`'\"")


def prompt_for(message, definitions, examples):
    labels = "\n".join(f"{name}: {meaning}" if definitions else name for name, meaning in LABELS.items())
    shots = "".join(f"Message: {text}\nLabel: {label}\n\n" for text, label in examples)
    return (f"Classify the customer message into exactly one of these labels:\n{labels}\n"
            f"Reply with the label only.\n\n{shots}Message: {message}\nLabel:")


test = load("test", 8)                                   # 48 real customer messages, 8 for each label
three = load("train", 3)
shuffled = random.Random(0).sample(three, len(three))
CONDITIONS = [("label names only", False, []),
              ("names + 1 example per label", False, load("train", 1)),
              ("names + 3 examples per label", False, three),
              ("the same 3, in shuffled order", False, shuffled),
              ("names + one-line definitions", True, []),
              ("definitions + 1 example per label", True, load("train", 1))]

print(f"{len(test)} test messages, {len(LABELS)} labels, 8 messages for each")
print("prompt                              correct   accuracy   most frequent reply")
for name, definitions, examples in CONDITIONS:
    replies = [ask(prompt_for(text, definitions, examples)) for text, _ in test]
    correct = sum(reply == label for reply, (_, label) in zip(replies, test))
    top, count = Counter(replies).most_common(1)[0]
    print(f"{name:34} {correct:>8}   {correct / len(test):>8.0%}   {top} x{count}")
print()
print("first example in the 3-per-label prompt:", three[0][1], "  after shuffling:", shuffled[0][1])

48 test messages, 6 labels, 8 messages for each
prompt                              correct   accuracy   most frequent reply
label names only                         34        71%   lost_or_stolen_card x16
names + 1 example per label              35        73%   card_arrival x19
names + 3 examples per label             21        44%   card_arrival x34
the same 3, in shuffled order            37        77%   card_arrival x16
names + one-line definitions             44        92%   lost_or_stolen_card x9
definitions + 1 example per label        36        75%   card_arrival x19

first example in the 3-per-label prompt: card_arrival   after shuffling: declined_card_payment

• Label names only: 71%. The model has a fair idea of what the names mean.
• One example per label: 73%. One more message right out of 48. That is noise, not an improvement.
• Three examples per label: 44%. Worse than giving none. The model answered card_arrival for 34 of the 48 messages. Only 8 belong there. The examples were listed label by label, and card_arrival came first.
• The same 18 examples, shuffled: 77%. Nothing changed except the order, and accuracy moved by 33 points.
• One-line definitions, no examples: 92%. The largest gain came from explaining the labels in words.
• Definitions plus one example per label: 75%. Adding examples to the strongest prompt made it weaker.

So on this model and this task, examples were the wrong tool. The labels were confusable, and what the model lacked was a statement of where one ends and the next begins. A definition supplies that directly. An example supplies it only by implication, along with accidental signals such as which label appears first or most often.

Why order and choice matter

This instability is documented. A study of few-shot prompting found that the format, the choice of examples and even their order could move accuracy from near chance to near the state of the art, because models lean towards certain answers for reasons that have nothing to do with the input (Zhao et al.: Calibrate Before Use). That study saw a pull towards answers near the end of the prompt. The small model here, given examples grouped by label, was pulled towards the label at the start. Different models, different biases, same lesson: examples are not neutral.

Larger models are steadier, and on a different task the ranking could reverse. That is the point. The effect of examples is an empirical question about your model and your data.

Choosing and arranging examples

1. Start zero-shot with a precise instruction. Define each label or field in a sentence. It is the cheapest thing to try, and here it was the most effective.
2. Add examples for what words cannot convey: an output format, a tone, a borderline case.
3. Balance them. The same number for each label, or the model learns the imbalance.
4. Vary them. Cover the awkward cases, not five versions of the easy one.
5. Mix the order, and check whether the result changes when you reorder. If it does, the prompt is fragile.
6. Keep the format identical across examples and the real input.
7. Stop adding when results stop improving. Every example is paid for on every call. See context windows explained.
8. Measure on held-out inputs that are not among the examples.

When each fits

For the last row, see fine-tuning vs RAG. For prompts that ask the model to reason before answering, see chain-of-thought prompting. For the wider toolkit, see prompt engineering techniques.

Common mistakes

• Assuming examples always help.
• Listing examples grouped by label, so the first or last label dominates.
• Using only easy examples.
• Testing on the examples themselves.
• Skipping the definition of the task and hoping the examples explain it.
• Leaving a dozen examples in a prompt that runs a million times a day.

Zero-shot and few-shot prompting are part of the Prompt Engineering module of the Generative AI course.

Sources

• Language Models are Few-Shot Learners (arXiv) — Brown et al.
• Prompt design strategies (Gemini API documentation) — Google
• Prompting best practices (Claude Platform Docs) — Anthropic
• Calibrate Before Use: Improving Few-Shot Performance of Language Models (arXiv) — Zhao et al.
• Efficient Intent Detection with Dual Sentence Encoders (arXiv) — Casanueva et al.
• Task-specific datasets, including Banking77 (GitHub, PolyAI) — PolyAI