vLLM and Model Serving: Scaling LLM Inference

Running a model for yourself is easy: load it, send a prompt, wait. Serving the same model to five hundred people at once is a different problem, and it is mostly not about the model. It is about scheduling and memory.

LLM serving systems such as vLLM exist because the obvious approaches waste most of an expensive GPU. Seeing why requires one fact about how text is generated.

One token per step

A model produces its answer one token at a time. Each token needs a full pass through the network, a decoding step. A 300-token answer is 300 steps. See how LLMs work.

Each step is limited less by arithmetic than by memory traffic. The GPU has to read the model's weights to produce a token, and moving those billions of numbers takes longer than computing with them. Anyscale's engineers describe inference as memory-IO bound for this reason, and conclude that throughput depends on how large a batch fits in GPU memory (Anyscale: continuous batching).

That is the key. If the weights are read once per step anyway, the same step can produce the next token for eight requests, or thirty-two, for little extra cost. The question is how to keep those slots full.

Three ways to schedule

Continuous batching makes the decision at every step instead of once per group. The idea, sometimes called iteration-level scheduling, came from research on serving systems, and it is now standard in serving software.

A simulation

The script serves the same 200 requests three ways. Their answers vary in length, mostly short with a long tail, as real traffic does. It counts decoding steps, how busy the eight slots were, and how long requests waited.

It is a model with stated simplifications. Every step is assumed to take the same time whatever the batch size, prompt processing is ignored, and all requests arrive at once.

import random
import statistics

random.seed(7)
# 200 requests arrive together. Most answers are short, a few are long, as in real traffic.
lengths = [min(1000, int(random.lognormvariate(4.6, 0.8))) for _ in range(200)]
SLOTS = 8          # how many sequences the server can decode side by side
print(f"{len(lengths)} requests, output length median {statistics.median(lengths):.0f} tokens, "
      f"longest {max(lengths)}, total {sum(lengths):,} tokens")


def one_at_a_time(lengths):
    finished, clock = [], 0
    for n in lengths:
        clock += n                      # one decoding step makes one token
        finished.append(clock)
    return finished, clock


def static_batches(lengths):
    finished, clock = [], 0
    for i in range(0, len(lengths), SLOTS):
        batch = lengths[i:i + SLOTS]
        finished += [clock + n for n in batch]
        clock += max(batch)             # the batch is released only when its longest answer ends
    return finished, clock


def continuous(lengths):
    waiting = list(enumerate(lengths))
    running, finished, clock = [], [0] * len(lengths), 0
    while waiting or running:
        while waiting and len(running) < SLOTS:       # fill free slots at every step
            running.append(list(waiting.pop(0)))
        clock += 1
        for job in running:
            job[1] -= 1
        for job in [j for j in running if j[1] == 0]:
            finished[job[0]] = clock
            running.remove(job)
    return finished, clock


print()
print("scheduling              decode steps   tokens per step   slots busy   median wait   95th percentile")
for name, schedule in (("one at a time", one_at_a_time), ("static batches of 8", static_batches),
                       ("continuous, 8 slots", continuous)):
    finished, steps = schedule(lengths)
    capacity = steps * (1 if schedule is one_at_a_time else SLOTS)
    p95 = sorted(finished)[int(0.95 * len(finished))]
    print(f"{name:22} {steps:>13,}   {sum(lengths) / steps:>15.2f}   {sum(lengths) / capacity:>10.0%}   "
          f"{statistics.median(finished):>11,.0f}   {p95:>15,}")

# Memory: every token a sequence holds needs a key and a value vector in every layer.
layers, width, bytes_each = 12, 768, 2              # GPT-2 small, 16-bit numbers
per_token = 2 * layers * width * bytes_each
print()
print(f"key-value cache per token for a 12-layer, 768-wide model: {per_token / 1024:.0f} KB; "
      f"for 8 sequences of 1,024 tokens: {8 * 1024 * per_token / 2**20:.0f} MB")
layers, width = 32, 4096                            # a typical 7-billion-parameter shape
per_token = 2 * layers * width * bytes_each
print(f"the same for a 32-layer, 4,096-wide model: {per_token / 1024:.0f} KB per token, "
      f"{8 * 4096 * per_token / 2**30:.0f} GB for 8 sequences of 4,096 tokens")

200 requests, output length median 90 tokens, longest 910, total 25,505 tokens

scheduling              decode steps   tokens per step   slots busy   median wait   95th percentile
one at a time                 25,505              1.00         100%        11,880            24,817
static batches of 8            8,020              3.18          40%         3,682             7,758
continuous, 8 slots            3,319              7.68          96%         1,565             3,101

key-value cache per token for a 12-layer, 768-wide model: 36 KB; for 8 sequences of 1,024 tokens: 288 MB
the same for a 32-layer, 4,096-wide model: 512 KB per token, 16 GB for 8 sequences of 4,096 tokens

• One at a time took 25,505 steps for 25,505 tokens, and the median request waited 11,880 steps.
• Static batches cut that to 8,020 steps, but the slots were busy only 40% of the time. Each group waited for its longest answer while seven finished requests held their places.
• Continuous batching needed 3,319 steps, kept the slots 96% busy and produced 7.68 tokens per step. The median wait fell to 1,565 steps.
• The slow tail improved most. The 95th-percentile request finished at step 3,101, against 7,758 with static batches.

The proportions, not the absolute numbers, are the lesson. With the same hardware, the scheduling policy alone changed throughput by more than a factor of two over static batching. Anyscale's own benchmarks of real systems reported much larger gains over naive batching.

The memory that limits the batch

Each sequence in a batch carries state: for every token so far, a key and a value vector in every layer. This is the key-value cache, and it is what lets the model avoid recomputing the whole history at each step. See attention explained.

The last lines of the output show how quickly it grows. A small 12-layer model needs 36 KB per token. A 32-layer model of a common 7-billion-parameter shape needs 512 KB per token, so eight conversations of 4,096 tokens hold 16 GB of cache, on top of the model's weights.

The cache is also hard to allocate. A sequence's final length is unknown when it starts, so simple servers reserve the maximum for every request, and most of that reservation is never used. The PagedAttention paper describes exactly this waste: the cache is large, grows and shrinks unpredictably, and is lost to fragmentation, which limits the batch size (Kwon et al.: PagedAttention).

PagedAttention borrows the operating system's answer to the same problem. The cache is split into small fixed-size blocks allocated as a sequence grows, like pages of virtual memory, so almost nothing is reserved and unused. The authors report near-zero waste and 2 to 4 times the throughput of earlier systems at the same latency.

vLLM

vLLM is the serving engine built around PagedAttention. Its documentation lists continuous batching, prefix caching, many quantization formats, speculative decoding and an OpenAI-compatible API among its features (vLLM documentation).

Starting a server is one command, and any OpenAI client can then talk to it. These follow the vLLM quickstart (vLLM: quickstart):

pip install vllm
vllm serve Qwen/Qwen2.5-1.5B-Instruct

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-1.5B-Instruct",
       "messages": [{"role": "user", "content": "Summarise PEP 8 in one sentence."}]}'

The quickstart lists Linux as the operating system and supported GPUs or other accelerators. These commands were not run for this article, because they need that hardware.

Because the API matches OpenAI's, an application written against a hosted model can often be pointed at a self-hosted one by changing a base URL and a model name.

Choosing a setup

The local runner in the second row has the same trade-offs in miniature. Ollama processes one request at a time per model by default, lets you raise that with a setting, and notes that memory needs grow with the number of parallel requests times the context length. Requests beyond its queue limit are rejected (Ollama: FAQ).

What to measure

• Throughput: tokens per second across all users.
• Time to first token: how long a user waits before anything appears.
• Latency percentiles: the 95th and 99th, not only the average, since the slowest requests are what users remember.
• GPU memory: weights plus cache, with headroom for long conversations.
• Cost per million tokens, so that self-hosting can be compared with an API. See cost optimization for LLM applications.

Common mistakes

• Benchmarking with one request at a time. A serving system shows its value under load.
• Testing only short prompts. Long contexts are where memory runs out.
• Reporting the average latency and hiding the tail.
• Ignoring the cache when sizing GPUs. The weights are not the only thing that must fit.
• Self-hosting for a few thousand requests a day, when an API would be cheaper and simpler.

For running fine-tuned adapters on a served model, see LoRA and QLoRA.

Model deployment strategies, including serving models behind an API, are part of the Deployment and Production module of the Generative AI course.

Sources

• Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv) — Kwon et al.
• vLLM documentation — vLLM project
• Quickstart (vLLM documentation) — vLLM project
• How continuous batching enables 23x throughput in LLM inference (Anyscale) — Anyscale
• FAQ (Ollama documentation) — Ollama