Generative AI and LLM Interview Questions

Generative AI and LLM interview questions have shifted from "what is RAG?" to "how would you know your RAG system works?" and "what will this cost per month?". Those are questions you answer with numbers. This guide works through them on a real dataset, BANKING77: 13,083 customer banking queries, each labelled with one of 77 intents (Casanueva et al., arXiv). The task stands in for a common LLM application: given a customer's message, retrieve similar labelled examples to help a model classify or answer it.

import os

os.environ["HF_HUB_OFFLINE"] = "1"                       # use the locally cached models only
import numpy as np
import pandas as pd
from sentence_transformers import SentenceTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from transformers import AutoTokenizer

train = pd.read_csv("banking77/train.csv")              # Banking77: customer banking queries labelled with 77 intents
test = pd.read_csv("banking77/test.csv").sample(1000, random_state=0)
print(f"G1 knowledge base: {len(train):,} labelled queries, {train.category.nunique()} intents | evaluation: {len(test)} held-out queries")


def evaluate(q_vecs, kb_vecs, name):
    sims = q_vecs @ kb_vecs.T
    top = np.argsort(-sims, axis=1)[:, :5]
    hits = train.category.to_numpy()[top] == test.category.to_numpy()[:, None]
    first = np.where(hits.any(axis=1), hits.argmax(axis=1) + 1, np.inf)
    print(f"   {name:<28} recall@1 {hits[:, 0].mean():.1%} | recall@5 {hits.any(axis=1).mean():.1%} | MRR@5 {(1 / first).mean():.3f}")


print("\nG2 retrieval quality: does the nearest stored query have the same intent?")
tfidf = TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True).fit(train.text)
evaluate(tfidf.transform(test.text).toarray(), tfidf.transform(train.text).toarray(), "TF-IDF (keywords)")
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu")
kb = model.encode(train.text.tolist(), batch_size=128, normalize_embeddings=True)
evaluate(model.encode(test.text.tolist(), batch_size=128, normalize_embeddings=True), kb, "all-MiniLM-L6-v2 embeddings")

print("\nG3 token budget with the GPT-2 tokenizer")
tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
lengths = np.array([len(tok.encode(t)) for t in test.text])
print(f"   tokens per customer query: mean {lengths.mean():.1f} | 95th percentile {np.percentile(lengths, 95):.0f} | max {lengths.max()}")
instruction = "Classify the customer's banking query into one of the intents shown in the examples. Reply with the intent only."
examples = "\n".join(f"Query: {q}\nIntent: {c}" for q, c in train[["text", "category"]].head(5).itertuples(index=False))
prompt_tokens = len(tok.encode(instruction + "\n" + examples)) + int(lengths.mean())
print(f"   prompt with instruction + 5 retrieved examples + query: about {prompt_tokens} tokens")
for monthly in (100_000, 1_000_000):
    print(f"   {monthly:>9,} queries a month -> {monthly * prompt_tokens / 1e6:,.1f} million input tokens")

G1 knowledge base: 10,003 labelled queries, 77 intents | evaluation: 1000 held-out queries

G2 retrieval quality: does the nearest stored query have the same intent?
   TF-IDF (keywords)            recall@1 72.6% | recall@5 92.0% | MRR@5 0.802
   all-MiniLM-L6-v2 embeddings  recall@1 91.4% | recall@5 96.7% | MRR@5 0.934

G3 token budget with the GPT-2 tokenizer
   tokens per customer query: mean 12.8 | 95th percentile 28 | max 78
   prompt with instruction + 5 retrieved examples + query: about 159 tokens
     100,000 queries a month -> 15.9 million input tokens
   1,000,000 queries a month -> 159.0 million input tokens

"How do you measure retrieval quality?"

Name the metrics and say what they mean for the product.

• Recall@k: the share of queries where at least one of the top k retrieved items is relevant. Here, relevant means a stored query with the same intent.
• MRR (mean reciprocal rank): the average of 1 divided by the rank of the first relevant result, so finding it first scores 1, second 0.5. It rewards putting the right answer at the top.

Then use the numbers. With embeddings, recall@1 was 91.4% and recall@5 96.7%: if the application passes five examples to the model, the right intent is among them for nearly 97% of queries.

"Keyword search or embeddings?"

Compare them on your own data rather than answering from principle. TF-IDF keyword retrieval reached 72.6% recall@1 and an MRR of 0.802. The all-MiniLM-L6-v2 sentence embeddings, a small model that maps text to 384-dimensional vectors for semantic search (Hugging Face: all-MiniLM-L6-v2), reached 91.4% and 0.934. Customers phrase the same need in many ways, "card hasn't arrived" and "still waiting for my new card", and embeddings capture that similarity where shared keywords do not.

A strong answer adds the trade-offs: embeddings need a model and a vector index; keyword search is cheaper, exact for codes and names, and easier to explain. Many production systems combine them.

"How many tokens will this use?"

Break the prompt into parts and count each:

• The query: customer messages averaged 12.8 tokens with the GPT-2 tokenizer, with a 95th percentile of 28. Budget for the tail, not the mean.
• The fixed instruction and five retrieved examples: together with an average query, about 159 tokens.
• Volume: at 100,000 queries a month that is about 15.9 million input tokens; at a million queries, about 159 million.

Multiply by your provider's current price per million input tokens, and add output tokens, which are usually priced separately. Say clearly that the tokenizer used here, GPT-2's, differs from those of commercial models, so the counts are an estimate to refine with the real tokenizer.

How to present numbers in an interview

• State the metric and the threshold that matters: "We need the right intent in the top five at least 95% of the time."
• Compare against a baseline: the keyword method here was the baseline that justified embeddings.
• Separate evaluation data from the knowledge base, as the code does with held-out test queries; otherwise the numbers are inflated.
• Give the cost as a formula, tokens per request times requests times price, so it updates when prices change.

Building an evaluation set quickly

Interviewers often follow up with "but where do the labels come from?" Explain a practical route. Start with a few hundred real queries from logs, with personal data removed. Have domain experts label the correct answer or document for each, and record disagreements, because ambiguous cases tell you where the system will struggle. Keep this set fixed and separate from anything used to build or tune the system, and add newly discovered failure cases over time. Even two or three hundred carefully labelled examples give far more trustworthy numbers than impressions from trying a few queries by hand.

When you have no numbers yet

Sometimes the interviewer describes a system that does not exist and asks how good it will be. Do not guess a figure. Describe the measurement you would run, the baseline you would compare against, and the threshold that would make the system worth shipping. That answer shows the same quantitative habit as quoting real results, and it is honest.

Other quantitative questions to expect

• How would you choose chunk size for long documents, and how would you measure the effect on recall?
• How would you set a similarity threshold to say "I don't know"?
• How would you estimate latency for retrieval plus generation?
• How many labelled examples do you need to evaluate reliably?
• How would you detect quality falling after a model or prompt change?

For the conceptual questions, transformers, retrieval-augmented generation, fine-tuning, agents and safety, see generative AI interview questions.

Retrieval, evaluation and LLM application design are taught in the Generative AI course.

Sources

• Efficient Intent Detection with Dual Sentence Encoders (Casanueva et al., arXiv:2003.04807) — arXiv
• sentence-transformers/all-MiniLM-L6-v2 (Hugging Face model card) — Hugging Face / sentence-transformers