Building a Customer Support Bot with LLMs

Most of what customers ask a support team has been asked before: where is my card, why did my transfer fail, how do I get a refund. That repetition is why support is one of the most common uses of language models. It is also where a confident wrong answer costs most.

In 2022, Air Canada's website chatbot told a passenger he could book at full fare and claim a bereavement discount afterwards. The policy said otherwise. When he took the airline to a Canadian tribunal, the airline argued that the chatbot was responsible for its own actions. The tribunal disagreed and ordered the airline to pay (BBC: Air Canada chatbot ruling). A support bot speaks for the company.

So the design question is not only how to answer, but when to answer and when to hand over.

The architecture

A support bot that holds up in production usually has five parts:

The language model is essential in generation and helpful in routing. The rest is ordinary software, and that is where most of the reliability comes from.

Routing: the step that decides everything

If the bot misunderstands the question, nothing after it can help. Routing can be done by an LLM, by a trained classifier, or by comparing meanings with embeddings. The last is cheap, fast and easy to measure, so the script uses it on a real dataset.

BANKING77 contains 13,083 customer queries to a bank, each labelled with one of 77 intents, released by PolyAI for research on intent detection (Casanueva et al.: BANKING77) under the CC BY 4.0 licence (Hugging Face: PolyAI/banking77). The script embeds every training message with a small sentence model (Hugging Face: all-MiniLM-L6-v2), represents each intent by the average of its messages, and routes each test query to the closest intent.

from collections import Counter

import numpy as np
import pandas as pd
from sentence_transformers import SentenceTransformer

# Banking77 (CC BY 4.0): real customer messages to a bank, each labelled with one of 77 intents
train = pd.read_csv("banking77/train.csv")
test = pd.read_csv("banking77/test.csv")
print(f"{len(train):,} training messages, {len(test):,} test messages, {train.category.nunique()} intents")

# Route by meaning: embed every message and compare a new one with each intent's average
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
train_vecs = model.encode(train.text.tolist(), normalize_embeddings=True, batch_size=128)
test_vecs = model.encode(test.text.tolist(), normalize_embeddings=True, batch_size=128)

intents = sorted(train.category.unique())
centroids = np.stack([train_vecs[(train.category == i).to_numpy()].mean(axis=0) for i in intents])
centroids /= np.linalg.norm(centroids, axis=1, keepdims=True)

scores = test_vecs @ centroids.T
best = scores.argmax(axis=1)
ranked = np.sort(scores, axis=1)
confidence = ranked[:, -1] - ranked[:, -2]          # margin between the best and second-best intent
predicted = np.array(intents)[best]
right = predicted == test.category.to_numpy()
print(f"routing every message: {right.mean():.1%} to the right intent")

# A bot should answer only when it is confident, and hand the rest to a person
print(f"\n{'margin needed':>13} {'answered by bot':>16} {'right when it answers':>22} {'sent to a person':>17}")
for threshold in (0.0, 0.02, 0.05, 0.08, 0.12):
    auto = confidence >= threshold
    print(f"{threshold:>13.2f} {auto.mean():>16.1%} {right[auto].mean():>22.1%} {(~auto).mean():>17.1%}")

print("\nmost frequent mix-ups (true intent -> routed to)")
mixups = Counter(zip(test.category[~right], predicted[~right]))
for (truth, guess), n in mixups.most_common(6):
    print(f"{n:>3}  {truth} -> {guess}")

10,003 training messages, 3,080 test messages, 77 intents
routing every message: 84.9% to the right intent

margin needed  answered by bot  right when it answers  sent to a person
         0.00           100.0%                  84.9%              0.0%
         0.02            90.9%                  89.5%              9.1%
         0.05            78.9%                  93.8%             21.1%
         0.08            66.6%                  96.2%             33.4%
         0.12            48.2%                  97.9%             51.8%

most frequent mix-ups (true intent -> routed to)
 15  supported_cards_and_currencies -> fiat_currency_support
 12  beneficiary_not_allowed -> failed_transfer
  9  topping_up_by_card -> transfer_into_account
  9  pending_transfer -> transfer_timing
  8  declined_transfer -> declined_card_payment
  6  card_delivery_estimate -> card_arrival

What the output shows

• Routing everything gets 84.9% right. Good for a few lines of code and no training, and not good enough to answer every customer automatically.
• Confidence buys accuracy. The script measures how far ahead the best intent is of the second best. Answering only when that margin is at least 0.05, the bot handles 78.9% of queries and gets 93.8% of those right. The other 21.1% go to a person.
• Every threshold is a business decision. Raising the bar to 0.08 lifts accuracy to 96.2% but sends a third of queries to people. The right point depends on what a wrong answer costs.
• The mix-ups are between neighbours. Supported cards and currencies against fiat currency support, a pending transfer against transfer timing. These are where a clarifying question works better than a guess, and where intents might be merged.

An LLM router would do better on many of these, at higher cost per message. The method of measuring stays the same: accuracy on a labelled test set, at the confidence level where the bot will actually answer.

Answering: grounded or not at all

Once the intent is known, the reply should come from approved content: the policy, the help article, the account data. Retrieve it, give it to the model, and require the answer to rest on it. A bot that answers from the model's general knowledge will eventually invent a policy. See how to reduce LLM hallucinations and, for the retrieval part, RAG architecture explained.

For anything that changes an account, such as refunds, cancellations or address changes, the model should prepare the action and a person or a confirmation step should approve it. See function calling and tool use.

Handover is a feature

Customers do not mind a handover. They mind repeating themselves. A good handover:

• Passes the full conversation and what the bot already looked up.
• Happens early for complaints, legal matters, vulnerable customers and anything the bot is unsure about.
• Is always available when the customer asks for a person.
• Tells the customer what happens next and how long it will take.

Measure the right things

Anthropic's guide to support bots suggests measuring how accurately the bot understands queries, how accurate its answers are, and how well it escalates, with a target of 95% or higher for correct escalation. It describes a deflection rate, the share of queries handled without a person, of 70 to 80% as a typical aim (Anthropic: customer support agent).

Deflection alone is a dangerous target, because a bot that never escalates scores perfectly on it. Track it together with accuracy on answered queries, escalation accuracy, repeat contacts and customer satisfaction. For building the test sets behind those numbers, see evaluating LLM outputs. For protecting the bot from manipulation, see guardrails for LLMs.

Common mistakes

• Optimising deflection without measuring wrong answers.
• Letting the model answer policy questions from its own knowledge.
• Hiding the route to a person.
• Handover without context, so customers start again.
• No confidence threshold, so every query gets an answer.
• Launching without a labelled test set of real customer messages.

Building chatbots and assistants with LLMs is part of the Building Gen AI Applications module of the Generative AI course.

Sources

• Efficient Intent Detection with Dual Sentence Encoders (arXiv) — Casanueva et al.
• PolyAI/banking77 dataset (Hugging Face) — PolyAI
• all-MiniLM-L6-v2 model card (Hugging Face) — sentence-transformers
• Customer support agent (Claude Platform Docs) — Anthropic
• Airline held liable for its chatbot giving passenger bad advice (BBC Travel) — BBC