The problem LoRA solves
Fine-tuning a large language model the traditional way means updating every one of its weights — often billions of numbers. That demands enormous GPU memory (you must store the weights, their gradients, and the optimiser state), takes a long time, and produces a full-size copy of the model for every task you fine-tune. Doing this for several tasks quickly becomes expensive and unwieldy. LoRA, short for Low-Rank Adaptation, was designed to get most of the benefit of fine-tuning at a tiny fraction of that cost.
How LoRA works
The key insight is that the change a model needs to learn for a new task is usually much simpler than the model itself — it can be captured by a low-rank update. Instead of editing a large weight matrix directly, LoRA freezes the original matrix and learns two small matrices, often called A and B, whose product approximates the needed change. If the original weight is a large square matrix, A and B are thin: one is tall-and-narrow, the other short-and-wide, and the narrow dimension between them is the rank (commonly 8, 16, or 32).
During training only A and B are updated; the original weights never move. At inference the small product is added back to the frozen weights, so the model behaves as if it had been fine-tuned. Because A and B are tiny relative to the full model, the number of trainable parameters drops by orders of magnitude, and the resulting adapter file can be just a few megabytes — small enough to store dozens of task-specific adapters and swap them in on demand.
LoRA vs full fine-tuning
Full fine-tuning is the most flexible option and can, in principle, change anything about the model, which makes it the better choice when you need the model to absorb large amounts of new knowledge or fundamentally change its behaviour. The cost is high memory, slow training, and a full model copy per task.
LoRA trades a little of that flexibility for huge efficiency gains. It typically matches full fine-tuning closely on task-specialisation and style-adaptation jobs while training far faster, using far less memory, and producing portable adapters. For the most common fine-tuning needs — making a model follow a domain’s tone, format outputs a certain way, or specialise in a task — LoRA is usually the right default.
QLoRA and the rest of the PEFT family
LoRA is the best-known member of a broader set of techniques called PEFT, parameter-efficient fine-tuning. The most important extension is QLoRA, which loads the frozen base model in 4-bit quantized precision to save even more memory and trains LoRA adapters on top. This makes it feasible to fine-tune very large models on a single modest GPU that could never hold them in full precision. Other PEFT methods include prefix tuning and adapter layers, but LoRA and QLoRA dominate in practice because they are simple, well-supported, and effective.
When to reach for LoRA
Choose LoRA when you want to adapt an existing strong model to a specific task, tone, or domain, especially if you have limited GPU memory or want to maintain many specialised variants cheaply. Reach for QLoRA when the base model is too large to fit otherwise. Consider full fine-tuning only when you genuinely need to teach the model substantial new knowledge or change its core behaviour and you have the hardware to support it. For most teams, LoRA is the practical sweet spot: it captures the bulk of the gains while keeping fine-tuning fast, cheap, and portable.