MLOps Interview Questions and Answers, Part 2: The ML System Design Round

Part 1 of the MLOps interview questions and answers covers the interview rounds and a real drift case. This part is the ML system design round, where candidates often draw tidy boxes (feature store, model server, monitoring) and stop. Interviewers are listening for something else: questions about the label, a check for leakage, a model chosen for its cost as well as its accuracy, and a serving design sized with numbers.

Here is one design question worked end to end, with the numbers measured rather than guessed: "Bookings arrive through our website and agents. Design a service that scores each new booking's cancellation risk, for up to 200 requests per second at peak." The data is real booking history from two Portuguese hotels (Antonio, de Almeida and Nunes, 2019).

import math, pickle, time, warnings
import numpy as np, pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
warnings.filterwarnings("ignore")

# Hotel booking demand (CC BY 4.0): predict at booking time whether a booking will be cancelled
h = pd.read_csv("hotels.csv")
h["nights"] = h.stays_in_weekend_nights + h.stays_in_week_nights
h["city"] = (h.hotel == "City Hotel").astype(int)
h["children"] = h.children.fillna(0)
FEATURES = ["lead_time", "adr", "nights", "adults", "children", "booking_changes", "previous_cancellations",
            "total_of_special_requests", "required_car_parking_spaces", "is_repeated_guest", "city"]

# Leakage check: reservation_status is recorded after the outcome, so it must never be a feature
print("D1 leakage check: is_canceled by reservation_status")
print(pd.crosstab(h.reservation_status, h.is_canceled).to_string())

train, test = h[h.arrival_date_year < 2017], h[h.arrival_date_year == 2017]      # train on the past, test on later arrivals
models = {"logistic regression": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
          "gradient boosting": HistGradientBoostingClassifier(random_state=0),
          "random forest (300 trees)": RandomForestClassifier(n_estimators=300, random_state=0, n_jobs=1)}

print(f"\nD2 train {len(train):,} bookings (2015-16), test {len(test):,} (2017)")
rows = []
for name, model in models.items():
    model.fit(train[FEATURES], train.is_canceled)
    auc = roc_auc_score(test.is_canceled, model.predict_proba(test[FEATURES])[:, 1])
    lat = []
    for i in range(300):                                  # one request at a time, as an online API would
        start = time.perf_counter(); model.predict_proba(test[FEATURES].iloc[[i]]); lat.append(time.perf_counter() - start)
    start = time.perf_counter(); model.predict_proba(test[FEATURES]); batch_s = time.perf_counter() - start
    rows.append({"model": name, "auc": round(auc, 3), "p50_ms": round(1000 * np.percentile(lat, 50), 1),
                 "p99_ms": round(1000 * np.percentile(lat, 99), 1), "batch_rows_per_s": round(len(test) / batch_s),
                 "size_kb": round(len(pickle.dumps(model)) / 1e3)})
table = pd.DataFrame(rows).set_index("model")
print(table[["auc", "size_kb"]].to_string())
print("\nD3 measured speed on one CPU core of this laptop")
print(table[["p50_ms", "p99_ms", "batch_rows_per_s"]].to_string())

# Sizing for the stated requirement: 200 requests/s at peak, keep each worker below 60% busy
PEAK_QPS, UTILISATION = 200, 0.6
print(f"\nD4 single-threaded workers needed for {PEAK_QPS} requests/s at peak")
for name, r in table.iterrows():
    workers = math.ceil(PEAK_QPS * (r.p50_ms / 1000) / UTILISATION)
    print(f"   {name:<27} {workers} worker(s)")
print(f"   nightly batch alternative: scoring all {len(test):,} test bookings took "
      f"{len(test) / table.batch_rows_per_s.min():.1f} s with the slowest model")

D1 leakage check: is_canceled by reservation_status
is_canceled             0      1
reservation_status              
Canceled                0  43017
Check-Out           75166      0
No-Show                 0   1207

D2 train 78,703 bookings (2015-16), test 40,687 (2017)
                             auc  size_kb
model                                    
logistic regression        0.745        2
gradient boosting          0.780      364
random forest (300 trees)  0.742   558895

D3 measured speed on one CPU core of this laptop
                           p50_ms  p99_ms  batch_rows_per_s
model                                                      
logistic regression           3.4     6.8           2953727
gradient boosting             6.5     9.1            466721
random forest (300 trees)    35.9    46.9             10591

D4 single-threaded workers needed for 200 requests/s at peak
   logistic regression         2 worker(s)
   gradient boosting           3 worker(s)
   random forest (300 trees)   12 worker(s)
   nightly batch alternative: scoring all 40,687 test bookings took 3.8 s with the slowest model

Step 1: question the label

The first table answers two questions interviewers love. What exactly is a "cancellation"? Here the label is 1 for 43,017 cancelled bookings and also for 1,207 no-shows. Whether a no-show should count as a cancellation is a business decision, because the hotel may treat them very differently. Ask before modelling.

Is there leakage? The reservation status matches the label perfectly, because it is recorded after the outcome. Including it would produce a near-perfect model offline and a useless one in production, where the status is unknown at booking time. Every feature must be available when the prediction is made. The model here uses only booking-time facts: lead time, rate, nights, party size, changes, special requests, history and hotel.

Step 2: split by time

The model is trained on 2015 and 2016 arrivals and tested on 2017, the way it would be used: trained on the past, scoring the future. A random split would mix future bookings into training and flatter the score.

Step 3: compare models on more than accuracy

Gradient boosting was the most accurate and small. The random forest was no better than logistic regression, yet over 550 MB, and in this run it took about 36 ms per single request against under 10 ms for the other two. scikit-learn's documentation explains why: for tree ensembles, latency grows with the number and depth of trees (scikit-learn: computational performance). A model that size also slows container start-up, deployments and rollbacks. In an interview, saying "I would not ship the forest, and here is the measurement" is exactly the reasoning being tested.

Step 4: size the online service

With measured median latency per request and a target of keeping each worker under 60% busy, the arithmetic is simple: requests per second times seconds per request, divided by the utilisation target. At 200 requests per second, gradient boosting needed 3 single-threaded workers in this run, and the random forest about four times as many. Add headroom for failures, such as one extra worker per zone, and measure again behind the real API, because network and serialisation add latency the in-process numbers exclude.

Step 5: ask whether it needs to be online at all

Scoring in bulk is much faster than one row at a time; scikit-learn reports differences of one to two orders of magnitude, and here the slowest model scored all 40,687 test bookings in about 4 seconds. If the business only needs risk scores for tomorrow's arrivals, a nightly batch job is simpler and cheaper than a service. If risk must be known while the guest is still booking, for example to ask for a deposit, it must be online. See batch vs real-time inference for the trade-offs.

Completing the design

Once the numbers are settled, the rest of the design is about keeping them true:

• Serving: a small API around the gradient boosting model, as in serving models with FastAPI and Docker.
• Feature parity: the same feature code at training and serving time, with golden predictions checked on every deploy.
• Monitoring: cancellation rate and score distribution by channel, because the mix of booking channels shifts over time.
• Retraining: on a schedule, with the time-based evaluation repeated before each release.

Follow-up questions to expect

• "Traffic doubles at a festival. What happens?" The worker count scales linearly with requests, so autoscaling on request rate or CPU, with a minimum that covers normal peaks, is the answer, plus a load test at twice the peak.
• "The model server goes down. What does the booking site do?" Fall back to a default decision, such as no deposit request, rather than blocking bookings. A risk score should never become a single point of failure.
• "How would you know the model is getting worse?" Labels arrive only after the arrival date, so monitor input and score distributions daily and accuracy once outcomes are known.

How to present it in the interview

Start with the questions (label, timing of features, latency need), then the measurement plan, then the boxes. A design that says "3 workers, because each request takes under 10 ms and we keep them 60% busy" is more convincing than any diagram.

ML system design, serving and monitoring are part of the MLOps course.

Sources

• Hotel booking demand datasets (Antonio, de Almeida and Nunes, Data in Brief, 2019) — Data in Brief (Elsevier)
• Computational performance (scikit-learn documentation) — scikit-learn