NLP Interview Questions for Data Scientists

NLP interview questions have moved towards transformers and large language models, but the fundamentals are still tested, because they still decide most real text projects: how text becomes numbers, how to evaluate without fooling yourself, and how to choose a threshold when mistakes have different costs. The answers below are measured on a real dataset with classic, fast models, which also gives you the baseline any fancier model must beat.

The data is the UCI SMS Spam Collection: 5,574 real English text messages labelled spam or ham, the spam reported by users on a UK forum and the ham drawn from a university's SMS corpus (UCI: SMS Spam Collection). Message texts are not printed below; the results are.

import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, precision_recall_curve
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline

sms = pd.read_csv("SMSSpamCollection", sep="\t", header=None, names=["label", "text"], quoting=3)   # UCI SMS Spam Collection
sms["spam"] = sms["label"].eq("spam").astype(int)
print(f"N1 {len(sms):,} messages, {sms['spam'].mean():.1%} spam, {sms['text'].duplicated().sum()} exact duplicate texts")

train, test = train_test_split(sms, test_size=0.3, stratify=sms["spam"], random_state=0)
seen = test["text"].isin(set(train["text"]))
print(f"\nN2 random split: {seen.mean():.1%} of test messages also appear in training "
      f"({seen[test['spam'] == 1].mean():.1%} of test spam)")
unique = sms.drop_duplicates("text")
u_train, u_test = train_test_split(unique, test_size=0.3, stratify=unique["spam"], random_state=0)


def f1(vectorizer, tr=train, te=test):
    model = make_pipeline(vectorizer, LogisticRegression(max_iter=2000, C=10)).fit(tr["text"], tr["spam"])
    return f1_score(te["spam"], model.predict(te["text"])), model


print(f"   word TF-IDF F1 on the random split {f1(TfidfVectorizer())[0]:.3f} | after removing duplicates first {f1(TfidfVectorizer(), u_train, u_test)[0]:.3f}")

print("\nN3 representations, duplicates removed (F1 on spam)")
for name, vec in (("word counts", CountVectorizer()), ("word TF-IDF", TfidfVectorizer()),
                  ("word 1-2-grams TF-IDF", TfidfVectorizer(ngram_range=(1, 2), min_df=2)),
                  ("character 2-5-grams TF-IDF", TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 5), min_df=2))):
    print(f"   {name:<28} {f1(vec, u_train, u_test)[0]:.3f}")

_, model = f1(TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 5), min_df=2), u_train, u_test)
p = model.predict_proba(u_test["text"])[:, 1]
precision, recall, thresholds = precision_recall_curve(u_test["spam"], p)
ok = precision[:-1] >= 0.99
print(f"\nN4 blocking real messages is costly: at 99% precision the model still catches {recall[:-1][ok].max():.1%} of spam "
      f"(threshold {thresholds[ok][np.argmax(recall[:-1][ok])]:.2f}); at the default 0.5 threshold it catches "
      f"{((p >= 0.5) & (u_test['spam'] == 1)).sum() / u_test['spam'].sum():.1%}")

words = make_pipeline(TfidfVectorizer(), LogisticRegression(max_iter=2000, C=10)).fit(u_train["text"], u_train["spam"])
vocab, coefs = words[0].get_feature_names_out(), words[1].coef_[0]
print("\nN5 strongest spam words:", list(vocab[np.argsort(coefs)[-8:][::-1]]))
print("   strongest ham words: ", list(vocab[np.argsort(coefs)[:8]]))

print(f"   word TF-IDF with English stop words removed: F1 {f1(TfidfVectorizer(stop_words='english'), u_train, u_test)[0]:.3f}")

train_vocab = set(CountVectorizer().fit(u_train["text"]).vocabulary_)
test_tokens = CountVectorizer().build_analyzer()
tokens = [t for text in u_test["text"] for t in test_tokens(text)]
print(f"\nN6 out-of-vocabulary: {np.mean([t not in train_vocab for t in tokens]):.1%} of test word tokens never appear in training")

N1 5,574 messages, 13.4% spam, 403 exact duplicate texts

N2 random split: 9.6% of test messages also appear in training (17.4% of test spam)
   word TF-IDF F1 on the random split 0.917 | after removing duplicates first 0.914

N3 representations, duplicates removed (F1 on spam)
   word counts                  0.921
   word TF-IDF                  0.914
   word 1-2-grams TF-IDF        0.935
   character 2-5-grams TF-IDF   0.944

N4 blocking real messages is costly: at 99% precision the model still catches 91.8% of spam (threshold 0.41); at the default 0.5 threshold it catches 89.8%

N5 strongest spam words: ['txt', 'text', 'call', 'chat', 'uk', '150p', 'free', 'www']
   strongest ham words:  ['me', 'my', 'can', 'gt', 'lt', 'that', 'going', 'am']
   word TF-IDF with English stop words removed: F1 0.910

N6 out-of-vocabulary: 8.5% of test word tokens never appear in training

N1. What do you check first in a text dataset?

Class balance and duplicates. 13.4% of messages are spam, so accuracy would flatter any model, and F1 on the spam class or precision and recall are the measures to report. 403 messages are exact repeats of another, which is common in real text: templates, forwarded messages, campaigns that send the same spam to many people. Also read a sample of messages from each class yourself, because counts alone will not show encoding problems or labels that look wrong.

N2. How can duplicates distort evaluation?

A random split puts copies of the same message on both sides. 9.6% of test messages also appeared in training, and for spam 17.4%, because spam campaigns repeat. A model can score well on those by memory. Here, removing duplicates before splitting changed F1 only slightly, from 0.917 to 0.914, but the check is cheap, and on other data the gap can be large. Say you would deduplicate, or split by source or time, before you trust a text score.

N3. Bag of words, TF-IDF or n-grams?

Measure them. With duplicates removed, plain word counts scored an F1 of 0.921, TF-IDF 0.914, word 1-2-grams 0.935 and character 2-5-grams 0.944 (scikit-learn: feature extraction). Character n-grams did best because spam plays with spelling and symbols: prices such as "150p", shortened words, odd spacing. They match pieces of words, so variants still share features. TF-IDF did not beat raw counts on these short messages, a reminder that the textbook default is not automatically better.

What to add: sentence embeddings or a fine-tuned transformer would likely do better still, at a much higher cost. Start with a measured baseline like this one so you can show what the extra cost buys.

N4. How would you set the threshold for a spam filter?

From the cost of each mistake. Blocking a real message is usually worse than letting one spam through, so fix a precision target and maximise recall under it (scikit-learn: precision-recall curve). At 99% precision, the character n-gram model still caught 91.8% of spam, using a threshold of 0.41. The default 0.5 threshold caught 89.8%, so it was more cautious than necessary. Choosing the threshold from the business requirement, rather than accepting 0.5, is the answer interviewers want.

N5. How do you explain what the model learned?

For a linear model, read the coefficients. The strongest spam words were "txt", "text", "call", "chat", "uk", "150p", "free" and "www", calls to action and premium-rate pricing. The strongest ham words were personal and conversational: "me", "my", "can", "going", "am", plus "lt" and "gt", fragments of encoded symbols that appear in the ham source. That last detail matters: if one class's messages carry formatting artefacts from how they were collected, a model can learn the collection process instead of the content. Spotting it is exactly what error analysis is for.

Should you remove stop words?

Not by default. Removing scikit-learn's English stop-word list lowered F1 from 0.914 to 0.910. The ham class depends on exactly those words: "me", "my", "can" and "am" were among its strongest signals, and most of them are on the stop-word list. Stop words carry style and intent, which is often what separates classes in short texts. Remove them when they are pure noise for your task, such as keyword search over long documents, and measure the effect either way.

N6. What happens with words the model has never seen?

They are ignored by word-based models. 8.5% of word tokens in the test set never appeared in training: names, typos, new slang, new product codes. Character n-grams and subword tokenisers soften this by breaking unknown words into known pieces, which is one reason modern language models use subword tokens; see tokenization in LLMs.

How to answer NLP questions

• Start with a strong, cheap baseline and measure every upgrade against it.
• Deduplicate before splitting and report precision and recall on the minority class.
• Choose the threshold from the cost of errors.
• Read what the model learned, and look for artefacts of how the data was collected.

Modern representations are covered in embeddings explained and how transformers changed AI, and the wider path in the NLP roadmap for 2026. For the full interview round, see data science interview questions and answers.

Text analytics and NLP are part of the Data Science course.

Sources

• SMS Spam Collection (UCI Machine Learning Repository) — UCI Machine Learning Repository
• Feature extraction (scikit-learn documentation) — scikit-learn
• precision_recall_curve (scikit-learn documentation) — scikit-learn