Low-rank adaptation (LoRA)
LoRA is a parameter-efficient fine-tuning method (Hu et al., Microsoft, arXiv 2021-06-17). The pre-trained weights are frozen and small trainable low-rank matrices are injected into the transformer layers; only those are trained. The paper reports up to 10,000x fewer trainable parameters and about 3x less GPU memory than full fine-tuning of GPT-3 175B with Adam, quality on par with or better than full fine-tuning, and no extra inference latency because the low-rank update can be merged into the base weights. Because an adapter is small, many adapters can be stored and swapped over one base model.
Variants
- QLoRA (Dettmers et al., arXiv 2023-05-23): trains LoRA adapters on top of a 4-bit quantised base model (4-bit NormalFloat, double quantization, paged optimizers). The paper reports fine-tuning a 65B model on a single 48GB GPU while preserving 16-bit fine-tuning task performance.
- DoRA (Liu et al., NVIDIA, arXiv 2024-02-14, ICML 2024 oral): splits each weight into magnitude and direction and applies LoRA to the direction; reported to beat LoRA on LLaMA, LLaVA and VL-BART tasks with no extra inference overhead.
2026 status
LoRA and QLoRA are widely used (author’s view; no usage data cited) to customise open-weight models (open-source-ai) on modest hardware, and the Hugging Face PEFT library (see hugging-face) is a reference implementation of parameter-efficient methods, integrated with Transformers, Diffusers and Accelerate (PEFT docs, accessed 2026-10-05). Adapting an open-weight large-language-model this way is an alternative to supplying examples in the prompt (in-context-learning) or retrieval. Per-method support and hyperparameter guidance change often and were not re-verified; check the library docs.
Open items
- Current adapter-serving support per inference engine (for example multi-LoRA serving) not verified for 2026.
Sources
- https://arxiv.org/abs/2106.09685 - LoRA (accessed 2026-10-05)
- https://arxiv.org/abs/2305.14314 - QLoRA (accessed 2026-10-05)
- https://arxiv.org/abs/2402.09353 - DoRA (accessed 2026-10-05)
- https://huggingface.co/docs/peft/en/index (accessed 2026-10-05)
History of this note
The earlier version was a 2023 quotation of a leaked Google memo on open-source competition plus a ZDNet link; it said little about LoRA itself and was replaced.