LoRA (Low-Rank Adaptation)

Full fine-tuning of a 70-billion-parameter model requires storing not only the 140 GB of model weights in GPU memory, but also gradients and Adam optimiser states that are three to four times larger, demanding a cluster of high-end GPUs and producing a brand-new 140 GB model file for every fine-tuned task. LoRA exploits the mathematical insight that when a pre-trained model adapts to a downstream task, the weight updates (`ΔW`) have a very low 'intrinsic rank'.

Instead of modifying a large weight matrix `W` (say, 4,096 by 4,096), LoRA freezes `W` completely and represents the update as the product of two skinny matrices, `A` (4,096 by `r`) and `B` (`r` by 4,096), where the rank `r` is typically between 8 and 64. Only `A` and `B` are trained, reducing trainable parameters to less than 1 per cent of the base model. When combined with 4-bit quantisation of the frozen base model (a technique known as QLoRA), an engineer can fine-tune a 70B model on a single workstation GPU, and the resulting adapter checkpoint is only 50 to 200 megabytes.

At inference time, serving engines like vLLM and LoRAX keep one shared base model in GPU memory and dynamically swap dozens of tenant- or task-specific LoRA adapters per request with negligible latency overhead.

LoRA transforms custom model training from a six-figure research project into a routine afternoon engineering job. A UK B2B SaaS platform can maintain a single base model in production while serving bespoke LoRA adapters for legal extraction, SQL generation, and customer-specific tone without multiplying GPU cluster costs.

Because LoRA constrains updates to a low-rank subspace, it can underperform full fine-tuning if you are trying to teach a model an entirely new human or programming language across billions of tokens. For 95 per cent of enterprise post-training tasks (formatting, classification, domain adaptation, and preference alignment), however, LoRA applied across all linear transformer layers matches full fine-tuning quality at a fraction of the cost.