Evaluating LLM Outputs: Metrics, Tools and Best Practices

A prompt change looks better on the three examples you tried. A new model sounds smarter in a demo. Neither tells you whether the application got better, and LLM applications fail in ways three examples never show. Evaluation turns those impressions into evidence.

OpenAI's guidance names the habit to avoid: "vibe-based evals", judging a system by whether it seems to work. It recommends evaluating early and often, logging everything, automating scoring where possible, and calibrating automated scores against human judgement (OpenAI: evaluation best practices).

Ways to score an output

Anthropic's evaluation guide ranks them by practicality. Code-based grading is the fastest and most reliable but lacks nuance. Human grading is the most flexible but slow and expensive. LLM-based grading is fast and flexible, but its reliability should be checked first. The guide also favours more questions graded automatically over fewer graded by hand (Anthropic: build evaluations). Overlap metrics need particular care: research on LLM evaluators notes that BLEU and ROUGE correlate poorly with human judgement on open-ended tasks (Liu et al.: G-Eval).

Can an LLM judge be trusted?

Using a model to grade a model is the most popular shortcut, and the most debated. The study that introduced MT-Bench found strong judges such as GPT-4 agreed with human preferences more than 80% of the time, about as often as humans agree with each other. It also documented the biases: preferring the answer in a particular position, preferring longer answers, and preferring their own style (Zheng et al.: Judging LLM-as-a-Judge).

The same researchers released their expert human votes, 3.3K pairwise preferences under CC BY 4.0 (Hugging Face: MT-Bench human judgments). That makes it possible to check any judge against people. The script takes 40 answer pairs where the human majority was clear, asks a small local model to pick the better answer in both orders, and compares the results with two yardsticks: a rule that the longer answer wins, and GPT-4's verdicts as recorded in the dataset.

import json
import urllib.request
from collections import Counter

import pandas as pd

URL = "http://localhost:11434/api/chat"
MODEL = "llama3.2"
OPTIONS = {"temperature": 0, "seed": 42, "num_ctx": 4096, "num_predict": 20}

# MT-Bench human judgments (CC BY 4.0): expert votes between two models' answers. First turn only.
human = pd.read_parquet("human.parquet").query("turn == 1").copy()
gpt4 = pd.read_parquet("gpt4_pair.parquet").query("turn == 1").copy()


def pair_key(r):
    return (r.question_id, *sorted([r.model_a, r.model_b]))


def winner_name(r):
    return {"model_a": r.model_a, "model_b": r.model_b}.get(r.winner, "tie")


for df in (human, gpt4):
    df["key"] = df.apply(pair_key, axis=1)
    df["won"] = df.apply(winner_name, axis=1)

rows = []
for key, group in human.groupby("key"):
    top, n = Counter(group["won"]).most_common(1)[0]
    if top == "tie" or n <= len(group) / 2:
        continue                                   # no clear human majority
    r = group.iloc[0]
    answers = {r.model_a: r.conversation_a[1]["content"], r.model_b: r.conversation_b[1]["content"]}
    if max(len(a) for a in answers.values()) > 1500:
        continue                                   # keep prompts small enough for a laptop
    rows.append({"key": key, "question": r.conversation_a[0]["content"], "answers": answers, "human": top})

print(f"{len(rows)} pairs have a clear human majority and short answers; 40 are sampled")
sample = pd.DataFrame(rows).sample(40, random_state=1).sort_values("key").to_dict("records")

SYSTEM = ("You are an impartial judge. Compare two AI assistants' answers to the user's question for helpfulness, "
          "relevance, accuracy, depth and detail. Do not let the order or the length of the answers influence you. "
          "Reply in JSON with the key winner set to A or B.")
SCHEMA = {"type": "object", "properties": {"winner": {"type": "string", "enum": ["A", "B"]}}, "required": ["winner"]}


def judge(question, first, second):
    prompt = f"[Question]\n{question}\n\n[Answer A]\n{first}\n\n[Answer B]\n{second}"
    body = {"model": MODEL, "stream": False, "options": OPTIONS, "format": SCHEMA,
            "messages": [{"role": "system", "content": SYSTEM}, {"role": "user", "content": prompt}]}
    req = urllib.request.Request(URL, json.dumps(body).encode(), {"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=900) as r:
        return json.loads(json.load(r)["message"]["content"])["winner"]


gpt4_verdict = dict(zip(gpt4["key"], gpt4["won"]))
tally = Counter()
for p in sample:
    x, y = sorted(p["answers"])
    ax, ay = p["answers"][x], p["answers"][y]
    original = x if judge(p["question"], ax, ay) == "A" else y
    swapped = y if judge(p["question"], ay, ax) == "A" else x
    longer = x if len(ax) > len(ay) else y
    tally["first shown"] += (original == x) + (swapped == y)
    tally["longer"] += longer == p["human"]
    tally["original"] += original == p["human"]
    tally["swapped"] += swapped == p["human"]
    if original == swapped:
        tally["consistent"] += 1
        tally["consistent and right"] += original == p["human"]
    g = gpt4_verdict.get(p["key"], "tie")
    tally["gpt4 verdicts"] += g in p["answers"]
    tally["gpt4 right"] += g == p["human"]

n = len(sample)
print(f"\n{'judge':<44} agrees with the human majority")
print(f"{'rule: the longer answer wins':<44} {tally['longer']} of {n}")
print(f"{'llama3.2 3B, answers in one order':<44} {tally['original']} of {n}")
print(f"{'llama3.2 3B, same answers swapped':<44} {tally['swapped']} of {n}")
print(f"{'llama3.2 3B, kept only when both orders agree':<44} {tally['consistent and right']} of {tally['consistent']}"
      f" (consistent on {tally['consistent']} of {n} pairs)")
print(f"{'GPT-4 judge, as recorded in the dataset':<44} {tally['gpt4 right']} of {tally['gpt4 verdicts']}"
      f" (tie or inconsistent on {n - tally['gpt4 verdicts']})")
print(f"\nllama3.2 picked the answer shown first in {tally['first shown']} of {2 * n} judgements")

404 pairs have a clear human majority and short answers; 40 are sampled

judge                                        agrees with the human majority
rule: the longer answer wins                 32 of 40
llama3.2 3B, answers in one order            30 of 40
llama3.2 3B, same answers swapped            33 of 40
llama3.2 3B, kept only when both orders agree 24 of 25 (consistent on 25 of 40 pairs)
GPT-4 judge, as recorded in the dataset      26 of 31 (tie or inconsistent on 9)

llama3.2 picked the answer shown first in 43 of 80 judgements

What the output shows

• The small judge agreed with people most of the time, 30 or 33 of 40 depending on which answer was shown first.
• Swapping the answers changed its verdict on 15 of 40 pairs. It was consistent on only 25. A judge that changes its mind when two answers trade places is guessing on those pairs.
• When it was consistent, it was almost always right: 24 of 25. Running both orders and keeping only agreed verdicts turns an unreliable judge into a reliable one for part of the data, and identifies the rest for a person.
• It leaned slightly towards the first answer, picking it in 43 of 80 judgements, which is position bias of a mild kind.
• "The longer answer wins" scored 32 of 40. In this sample the stronger models also wrote longer answers, so length predicted the human verdict well. That is a warning, not a recommendation. A judge that is barely better than a length rule may be rewarding length, the verbosity bias the MT-Bench study describes.
• GPT-4's recorded verdicts agreed on 26 of the 31 pairs where it chose, and it declined or contradicted itself on 9. Strong judges are better, not perfect.

Forty pairs give rough proportions, not precise rates. The method is the point: before trusting a judge, measure it against human labels on your own data.

Using LLM judges well

• Write a rubric, with criteria and examples, and ask for a constrained verdict such as A or B, or a score from 1 to 5.
• Judge both orders in pairwise comparisons, and treat disagreement as "unsure".
• Compare against trivial baselines such as length. A judge must beat them to be worth its cost.
• Calibrate against people on a sample, and repeat when the judge model changes.
• Use a strong judge, and avoid judging a model with itself where self-preference can creep in.

Building the evaluation set

Start from real traffic: logged questions, support tickets, the cases that went wrong. Add edge cases, adversarial inputs and questions with no good answer. For each, record what a good output must contain, or the correct label. Keep the set under version control next to the prompts it tests, and run it on every change of prompt, model or retrieval settings. See prompt templates and versioning.

Separate measurements for separate failures. Retrieval quality needs its own metrics; see retrieval evaluation. Invented answers need their own count; see how to reduce LLM hallucinations.

Tools

Evaluation can start as a short script, like the one above. Dedicated tools add test matrices, caching, reports and CI integration. promptfoo, for example, is an open-source command-line tool and library for evaluating and red-teaming LLM applications (promptfoo: intro).

Hosted platforms change. OpenAI is retiring its own Evals platform, read-only from 31 October 2026 and scheduled to shut down on 30 November 2026, as its evaluation guide notes. Keep evaluation sets and scoring logic in your own repository so a vendor change does not erase them.

In production

Offline evaluation catches regressions before release. After release, sample real conversations for review, track user feedback such as ratings and escalations, and watch for drift as users ask new kinds of questions. Feed the failures back into the evaluation set. For safety checks, see guardrails for LLMs.

Common mistakes

• Judging by a handful of examples.
• Trusting an LLM judge without measuring it against human labels.
• Pairwise judging in one order only.
• Using ROUGE or BLEU for open-ended answers.
• An evaluation set that never changes while the product does.
• Keeping the only copy of your evals in a vendor's dashboard.

LLM evaluation, prompt evaluation and cost and latency tuning form the Evaluation and Optimization module of the Generative AI course.

Sources

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv) — Zheng et al.
• MT-Bench human judgments (Hugging Face) — LMSYS
• Evaluation best practices (OpenAI API documentation) — OpenAI
• Define success criteria and build evaluations (Claude Platform Docs) — Anthropic
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (arXiv) — Liu et al.
• Intro (promptfoo documentation) — promptfoo