How to Read a Machine Learning Research Paper

Research papers are where new methods appear first, months or years before tutorials catch up. They are also dense, written for other researchers, and easy to misread. Reading one from the first word to the last on a first attempt usually ends halfway through the methods section, confused. The fix is not more effort but a different order: read in passes, each with a specific goal, and stop as soon as you have what you need.

This guide uses the three-pass method from S. Keshav's short and widely shared note, How to Read a Paper (Keshav: How to Read a Paper), and applies it to a paper that changed the field: Attention Is All You Need, which introduced the Transformer (arXiv: Attention Is All You Need).

Before you start: choose the paper and the goal

Decide why you are reading. To know what a method is, a first pass may be enough. To use it, you need the second. To build on it or review it, you need all three. Find papers through arXiv, conference proceedings and the references of papers you already trust, and prefer a recent survey when entering a new area: one survey can replace twenty first passes.

Pass one: five to ten minutes for the big picture

Keshav's first pass reads only the title, abstract and introduction, the section and subsection headings, and the conclusions, then glances at the references. At the end, you should be able to answer the five Cs:

• Category: what type of paper is it: a new method, an analysis, a benchmark, a system?
• Context: which other papers and ideas does it build on?
• Correctness: do the assumptions appear valid?
• Contributions: what are the main claims?
• Clarity: is it well written?

For the Transformer paper, the abstract answers most of these. Category: a new architecture. Context: sequence models built from recurrent or convolutional networks with attention. Contribution: a model "based solely on attention mechanisms", reporting 28.4 BLEU on English-to-German translation and 41.8 on English-to-French, "after training for 3.5 days on eight GPUs". That is already enough to explain the paper in one sentence, and to decide whether to continue.

Pass two: up to an hour on figures, tables and the argument

The second pass reads the whole paper but skips proofs and dense derivations. Spend most of the time on figures and tables, because that is where the claims are either supported or not. Note references you have not read that seem important; they are your background reading. By the end, you should be able to summarise the paper's main thrust, with supporting evidence, to someone else.

For every results table, ask:

• Compared with what? Are the baselines strong and recent, and tuned as carefully as the new method?
• Measured how? Which metric and dataset, and is the test data truly unseen? Evaluation metrics explained covers what common metrics hide.
• How much variation? Are there error bars or several runs, or is one number presented as final? The NeurIPS paper checklist asks authors to report error bars or statistical significance for exactly this reason (NeurIPS: paper checklist).
• At what cost? Compute, data and training time. A small gain that needs ten times the compute is a different result from a free one.
• What is missing? Ablations, where parts of the method are removed one at a time, show which parts actually matter.

The Transformer paper earns trust on several of these: its results table compares against published models and also lists the estimated training cost of each, and it includes a table of variations of the architecture, an ablation study that shows how results change as components change.

Separate what is shown from what is suspected

Papers mix three kinds of statement: results that are measured, explanations the authors believe, and speculation. Read the verbs. In section 3.2.1, the Transformer authors explain why they divide attention scores by the square root of the key dimension d_k: "We suspect that for large values of d_k, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients." A footnote adds that if the components of the query and key are independent with mean 0 and variance 1, their dot product has variance d_k (arXiv: Attention Is All You Need, HTML). "We suspect" is an honest hedge: the scaling is part of the method, but the reason is offered as a belief, not tested in the paper. Statements like that are good candidates for checking yourself.

Pass three: re-implement to understand

Keshav describes the third pass as virtually re-implementing the paper: making the same assumptions as the authors and re-creating the work, which exposes both its innovations and its hidden assumptions. For a machine learning paper, the most direct way is to code the core equation. Scaled dot-product attention is Attention(Q, K, V) = softmax(QKᵀ / √d_k) V:

import numpy as np

def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)        # subtract the max for numerical stability
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

def attention(Q, K, V, scale=True):
    scores = Q @ K.T
    if scale:
        scores = scores / np.sqrt(K.shape[-1])
    weights = softmax(scores)
    return weights @ V, weights

rng = np.random.default_rng(0)
for d_k in (4, 64, 512):
    q, k = rng.normal(size=(10_000, d_k)), rng.normal(size=(10_000, d_k))
    print(f"d_k {d_k}: variance of q·k = {np.var((q * k).sum(axis=1)):.1f}")

d_k 4: variance of q·k = 4.0
d_k 64: variance of q·k = 65.3
d_k 512: variance of q·k = 509.7

The footnote's arithmetic holds: the variance of the dot product tracks d_k. Now the consequence the authors suspected, with 5 queries attending over 8 keys of dimension 512:

d_k = 512
Q, K, V = rng.normal(size=(5, d_k)), rng.normal(size=(8, d_k)), rng.normal(size=(8, 16))
output, weights = attention(Q, K, V)
print("output shape:", output.shape, "| weights per query sum to", weights.sum(axis=1).round(6))
for scale in (False, True):
    _, weights = attention(Q, K, V, scale)
    print("scaled" if scale else "unscaled", "largest weight per query:", weights.max(axis=1).round(3))

output shape: (5, 16) | weights per query sum to [1. 1. 1. 1. 1.]
unscaled largest weight per query: [0.515 1.    0.993 1.    1.   ]
scaled largest weight per query: [0.334 0.584 0.221 0.318 0.353]

Without scaling, most queries put almost all their weight on a single key: the softmax is saturated, the regime where its gradients become tiny. With scaling, attention is spread across keys. Twenty lines of NumPy turned a sentence you might have skimmed into something you understand and could explain in an interview. Deep learning explained covers the backpropagation that those gradients feed.

Keep notes that compound

After each paper, write a short entry: the one-sentence summary, the five Cs, the key figure or table, what you doubted, and the references to read next. Over months, these notes become a map of your field and make later papers much faster to read, because most papers build on a handful of earlier ones.

Common mistakes

• Reading linearly on the first attempt. Skim first; most papers do not deserve a full read, and the ones that do are easier the second time.
• Getting stuck on the maths. Skip derivations in the second pass and return in the third, if the paper matters to you.
• Taking reported gains at face value. Check baselines, variation and cost before believing a table.
• Ignoring the limitations section. When present, it is the most honest part of the paper; when absent, ask what it would say.
• Reading alone. Explaining a paper to others, or reading public peer reviews where a venue publishes them, reveals what you missed.

Neural networks, CNNs and sequence models, the background most deep learning papers assume, make up the deep learning module of the Data Science course.

Sources

• How to Read a Paper — S. Keshav, University of Waterloo
• Attention Is All You Need — arXiv, 2017
• Attention Is All You Need (HTML version, section 3.2.1) — arXiv
• NeurIPS Paper Checklist — NeurIPS