# LoRA (Low-Rank Adaptation)

AI Glossary · Training, Fine-Tuning & Evals
Canonical URL: https://aisat.uk/glossary/lora
Author: Lucy Gilmore, Assistant General Counsel, Product, Privacy & AI at Sprinklr | Practical Privacy, Product & AI Governance & Defensible Risk Decisions (https://www.linkedin.com/in/lucy-gilmore/)
Breadcrumb: [Glossary](https://aisat.uk/glossary) > [Training & Evals](https://aisat.uk/glossary?category=training) > LoRA (Low-Rank Adaptation)

## TL;DR

> **LoRA (Low-Rank Adaptation)** is a parameter-efficient fine-tuning (PEFT) technique that freezes a pre-trained language model's original weights and injects small, trainable rank-decomposition matrices into its layers. It cuts fine-tuning memory and compute costs by over 90 per cent while producing tiny, hot-swappable adapters.

## How LoRA (Low-Rank Adaptation) works in practice

Full fine-tuning of a 70-billion-parameter model requires storing not only the 140 GB of model weights in GPU memory, but also gradients and Adam optimiser states that are three to four times larger, demanding a cluster of high-end GPUs and producing a brand-new 140 GB model file for every fine-tuned task. LoRA exploits the mathematical insight that when a pre-trained model adapts to a downstream task, the weight updates (`ΔW`) have a very low 'intrinsic rank'.

Instead of modifying a large weight matrix `W` (say, 4,096 by 4,096), LoRA freezes `W` completely and represents the update as the product of two skinny matrices, `A` (4,096 by `r`) and `B` (`r` by 4,096), where the rank `r` is typically between 8 and 64. Only `A` and `B` are trained, reducing trainable parameters to less than 1 per cent of the base model. When combined with 4-bit quantisation of the frozen base model (a technique known as QLoRA), an engineer can fine-tune a 70B model on a single workstation GPU, and the resulting adapter checkpoint is only 50 to 200 megabytes.

At inference time, serving engines like vLLM and LoRAX keep one shared base model in GPU memory and dynamically swap dozens of tenant- or task-specific LoRA adapters per request with negligible latency overhead.

## Why it matters and trade-offs

LoRA transforms custom model training from a six-figure research project into a routine afternoon engineering job. A UK B2B SaaS platform can maintain a single base model in production while serving bespoke LoRA adapters for legal extraction, SQL generation, and customer-specific tone without multiplying GPU cluster costs.

Because LoRA constrains updates to a low-rank subspace, it can underperform full fine-tuning if you are trying to teach a model an entirely new human or programming language across billions of tokens. For 95 per cent of enterprise post-training tasks (formatting, classification, domain adaptation, and preference alignment), however, LoRA applied across all linear transformer layers matches full fine-tuning quality at a fraction of the cost.

## Related AI concepts

- [Fine-Tuning](https://aisat.uk/glossary/fine-tuning): Fine-Tuning is the post-training process of updating a pre-trained model's weights on a curated dataset of task-specific input-output examples to lock in domain vocabulary, strict formatting or lower inference cost.
- [Quantisation](https://aisat.uk/glossary/quantisation): Quantisation is a model compression technique that converts neural network weights and activations from high-precision 16-bit or 32-bit floating-point numbers into lower-bit representations such as INT8, FP8 or INT4, reducing VRAM footprint and accelerating inference.
- [Small Language Model (SLM)](https://aisat.uk/glossary/small-language-model): A Small Language Model (SLM) is a compact neural language model, typically between one billion and ten billion parameters, optimised for low-latency, low-cost inference on a single GPU or on-device NPU for domain-specific workflows.
- [Large Language Model (LLM)](https://aisat.uk/glossary/large-language-model): A Large Language Model (LLM) is a deep transformer neural network containing billions of parameters trained on massive text and code corpora to comprehend, summarise, translate, reason over and generate human and programming languages.