LoRA and QLoRA: Efficient Fine-Tuning Explained

Fine-tuning a language model used to mean updating every one of its weights. For a model with billions of parameters that needs a great deal of memory, produces a full copy of the model for each task, and is out of reach for most teams.

LoRA and QLoRA changed that. LoRA trains a small add-on while the original model stays frozen. QLoRA compresses the frozen model so the whole procedure fits on a single graphics card. Between them they are how most open models are fine-tuned today.

The idea behind LoRA

Fine-tuning changes a weight matrix by some amount. Call the original matrix W and the change ΔW. Full fine-tuning learns all of ΔW, which is as large as W.

LoRA, short for low-rank adaptation, makes a bet: the useful change can be written as the product of two much smaller matrices, B and A. If W is 768 by 2,304, ΔW has about 1.8 million entries. With a rank of 8, A and B together have 8 times (768 + 2,304), which is 24,576. Only those are trained. The original weights are frozen and never touched (Hu et al.: LoRA).

The paper reports that, against full fine-tuning of GPT-3, this cut trainable parameters by a factor of 10,000 and memory by a factor of 3, with quality on par or better on the models it was tried on.

Two details make it practical. One of the two matrices starts at zero, so at the beginning the adapter changes nothing and training starts from the original model's behaviour. And after training, the product can be added into W, so the adapted model runs at exactly the original speed (Hugging Face: LoRA).

A real run

The script uses the PEFT library to put LoRA on GPT-2, a small open model, and trains it for 60 steps on paragraphs from PEP 8, the Python style guide (PEP 8). The aim is modest: nudge the model towards the wording of that document, and measure whether it predicts unseen paragraphs of it better. It ran on a laptop without a graphics card.

import json
import os
import re

import torch
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.manual_seed(0)
tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
tok.pad_token = tok.eos_token
model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
total = sum(p.numel() for p in model.parameters())
frozen_before = model.transformer.h[0].attn.c_attn.weight.clone()

# 1. The arithmetic: one adapted matrix in each of 12 layers, 768 inputs and 2,304 outputs.
print(f"full fine-tuning trains all {total:,} parameters")
for r in (4, 8, 16):
    extra = 12 * r * (768 + 2304)
    print(f"LoRA rank {r:>2}: {extra:>9,} trainable parameters, {extra / total:.2%} of the model")

model = get_peft_model(model, LoraConfig(r=8, lora_alpha=16, target_modules=["c_attn"], fan_in_fan_out=True))
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"PEFT reports for rank 8: {trainable:,} trainable")

# 2. A small real task: adapt GPT-2 to the wording of PEP 8.
pep8 = [s["text"] for s in json.load(open("../../genai/pep_sections.json", encoding="utf-8")) if s["pep"] == 8]
paragraphs = [" ".join(p.split()) for text in pep8 for p in re.split(r"\n\s*\n", text) if len(p.split()) >= 15]
batch = lambda texts: tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=64)
train, held_out = paragraphs[:160], batch(paragraphs[160:200])


def loss_on(data):
    labels = data.input_ids.masked_fill(data.attention_mask == 0, -100)
    return model(**data, labels=labels).loss


model.eval()
with torch.no_grad():
    before = loss_on(held_out).item()
optimizer = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-3)
model.train()
for step in range(60):
    data = batch(train[step % 20 * 8: step % 20 * 8 + 8])
    loss_on(data).backward()
    optimizer.step()
    optimizer.zero_grad()
model.eval()
with torch.no_grad():
    after = loss_on(held_out).item()
print()
print(f"loss on 40 held-out paragraphs: {before:.2f} before, {after:.2f} after 60 training steps")
unchanged = torch.equal(frozen_before, model.base_model.model.transformer.h[0].attn.c_attn.weight)
print("original weights unchanged:", unchanged)

model.save_pretrained("adapter")
adapter_mb = os.path.getsize("adapter/adapter_model.safetensors") / 1e6
print(f"adapter saved: {adapter_mb:.1f} MB, against {total * 4 / 1e6:.0f} MB for the full model")

full fine-tuning trains all 124,439,808 parameters
LoRA rank  4:   147,456 trainable parameters, 0.12% of the model
LoRA rank  8:   294,912 trainable parameters, 0.24% of the model
LoRA rank 16:   589,824 trainable parameters, 0.47% of the model
PEFT reports for rank 8: 294,912 trainable

loss on 40 held-out paragraphs: 3.97 before, 3.62 after 60 training steps
original weights unchanged: True
adapter saved: 1.2 MB, against 498 MB for the full model

• Rank 8 trains 294,912 parameters, 0.24% of the model. The library's count matches the arithmetic: 12 layers, each with 8 times (768 + 2,304).
• Doubling the rank doubles the count. Rank is the dial for how much the adapter can learn.
• The loss on held-out text fell from 3.97 to 3.62. Sixty steps on a laptop moved the model measurably towards the document's style.
• The original weights are bit-for-bit unchanged. Everything learned is in the adapter.
• The adapter is 1.2 MB. The model it adapts is 498 MB.

That last pair of numbers is the practical revolution. You can keep one copy of a base model and a folder of small adapters, one for each task or customer, and load the one you need. Hugging Face's own example is a 6 MB adapter for a 700 MB model (Hugging Face: PEFT quicktour).

A caution on scale: this run is a demonstration. Adapting GPT-2 to a style guide proves the mechanism, not a useful product.

The settings that matter

By default the PEFT library adapts the query and value projections of each attention block. Adapting more layers adds parameters and capacity. A higher rank is not automatically better: with little data, a large adapter memorises it.

QLoRA: shrink the frozen part

LoRA reduces what is trained. The frozen model still has to sit in memory, and that is the larger problem. Weights are normally stored as 16-bit numbers, so a model with 7 billion parameters needs about 14 GB before any training begins.

QLoRA stores the frozen weights in 4 bits instead, a quarter of the space, and trains ordinary LoRA adapters on top. The paper's headline result is fine-tuning a 65-billion-parameter model on a single 48 GB graphics card while matching the quality of 16-bit fine-tuning (Dettmers et al.: QLoRA). At 4 bits, the 7-billion-parameter model above takes about 3.5 GB.

Normally a quantized model is not trained further, because the low precision makes training unstable. It works here because the quantized weights are never updated. Only the adapter, kept at full precision, learns (Hugging Face: PEFT quantization).

In code, QLoRA is a loading option placed in front of the same LoRA setup:

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import prepare_model_for_kbit_training

config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", quantization_config=config)
model = prepare_model_for_kbit_training(model)

This block follows Hugging Face's documentation. It needs a supported graphics card and was not run for this article.

What they do not fix

• Bad data. An adapter learns whatever the examples teach, including their mistakes.
• Missing knowledge. Fine-tuning is a poor way to add facts. Use retrieval for that. See fine-tuning vs RAG.
• Evaluation. A falling training loss is not proof of a better model. Hold out data, as the example does, and test on the task you care about. See evaluating LLM outputs.
• Serving. A tuned open model still has to be hosted. See vLLM and model serving.

A workflow

1. Confirm that prompting and retrieval are not enough.
2. Collect a few hundred clean examples of input and desired output, and set some aside.
3. Pick a base model that already does the task tolerably.
4. Start with rank 8 or 16 on the attention layers.
5. Train briefly, and compare held-out results with the untuned model.
6. Raise the rank or add target layers only if it underfits.
7. Merge the adapter for deployment, or keep it separate if you have several.

For the libraries involved, see Hugging Face Transformers: a practical guide.

Common mistakes

• Choosing a high rank by default.
• Evaluating on the training examples.
• Using an adapter with a different base model from the one it was trained on. It will load and produce nonsense, or fail.
• Forgetting the prompt format. A chat model tuned on text in one template must be used with the same template.
• Expecting 4-bit training to be free. It saves memory and costs some speed.

Fine-tuning and how it compares with prompting are covered in the Advanced LLM Concepts module of the Generative AI course.

Sources

• LoRA: Low-Rank Adaptation of Large Language Models (arXiv) — Hu et al.
• QLoRA: Efficient Finetuning of Quantized LLMs (arXiv) — Dettmers et al.
• LoRA (Hugging Face PEFT documentation) — Hugging Face
• Quicktour (Hugging Face PEFT documentation) — Hugging Face
• Quantization (Hugging Face PEFT documentation) — Hugging Face
• PEP 8: Style Guide for Python Code (Python Enhancement Proposals) — Python Software Foundation