Imagine: you have 24 GB of VRAM and a desire to teach an open language model to respond in your company's style. Full fine-tuning of a 7-billion-parameter model would require about 112 GB of VRAM — four times what you have. This is where LoRA and QLoRA come in: methods that let you fine-tune a model on a single GPU without buying a cluster. But behind the simplicity lie nuances that can cause training to collapse or results to fall short of expectations.
In this article, we'll break down how low-rank adapters work, why 4-bit quantization is needed, how much memory is actually required, which hyperparameters affect what, and when fine-tuning isn't needed at all.
Why fine-tuning isn't just "run train"
Fine-tuning is the continuation of training a pre-trained model on your data so it better solves a narrow task: responds in a certain style, follows a format, knows a subject area. Unlike prompt engineering, where you craft instructions, fine-tuning changes the model's weights. This gives more consistent behavior but requires data, computation, and an understanding of what exactly you're changing.
The key problem is scale. A 7B parameter model in fp16 takes about 14 GB. Full fine-tuning requires storing weights, gradients, and optimizer states (usually AdamW). Gradients — another 14 GB, optimizer states — twice the weights (for AdamW, first and second moments are stored). Total roughly 14 + 14 + 28 = 56 GB plus activations, which depend on batch size and sequence length. In practice, for a 7B model with batch size 1 and sequence length 512, you might need 60–80 GB. That's why full fine-tuning on a single consumer GPU is rare.
LoRA: low-rank adapters
LoRA (Low-Rank Adaptation) offers an elegant workaround. Instead of updating all weights of matrix W (size d×k), we freeze it and add two small matrices A (d×r) and B (r×k), where r is the rank, usually 8, 16, or 32. Forward pass: h = Wx + BAx. Only A and B are trained. The number of parameters drops from dk to r(d+k). For d=k=4096 and r=16, that's 131 thousand instead of 16.7 million — 128 times fewer.
In practice, this means that for a 7B model with r=16, trainable parameters will be around 4–8 million (depending on which layers the adapters are applied to). Memory for gradients and optimizer for them is tiny. The main memory goes to frozen weights in fp16 (14 GB) and activations. This approach allows fine-tuning a 7B model on a GPU with 24 GB (e.g., RTX 4090) with a small batch size.
Hyperparameters: rank, alpha, learning rate
- Rank (r) — the main parameter. Larger rank — more adapter capacity, but also higher risk of overfitting. For style tasks, r=8–16 is often enough; for complex subject areas — 32–64. Too small a rank (r=4) may fail to learn what's needed; too large — loses the point of LoRA (approaches full fine-tuning in parameter count).
- Alpha — scaling coefficient: ΔW = (alpha/r) * BA. Usually alpha is set equal to r or 2r. If alpha is too large, the adapter dominates and the model loses base knowledge; if too small, training is slow.
- Learning rate — for LoRA usually higher than for full fine-tuning: 1e-4 – 3e-4. Too high leads to adapter destruction, too low — to underfitting.
QLoRA: 4-bit quantization for even more savings
QLoRA goes further: it freezes the base weights in 4-bit format (NF4 — NormalFloat4), and trains LoRA adapters in 16-bit. This reduces memory for weights from 14 GB to about 3.5–4 GB for a 7B model. Additionally, double quantization (quantizing the quantization constants) and paged optimizers are used to manage memory peaks.
As a result, QLoRA allows fine-tuning a 7B model on a GPU with 8–12 GB VRAM (e.g., RTX 3060 12GB or even 8GB with a small batch). For a 13B model, about 10–12 GB is needed; for 70B — already 48+ GB, but still many times less than full fine-tuning.
However, quantization introduces noise: 4-bit representation loses precision. In practice, this may slightly reduce the quality of the final model compared to LoRA on fp16, but for many tasks the difference is unnoticeable. Important: only frozen weights are quantized, adapters are trained in full precision, so gradients are not affected by quantization.
How much VRAM is actually needed
Estimates for a 7B model (approximate, with batch size 1 and sequence length 512):
| Method | Memory for weights | Total VRAM (with gradients and activations) |
|---|---|---|
| Full fine-tuning (fp16) | 14 GB | 60–80 GB |
| LoRA (fp16) | 14 GB | 18–24 GB |
| QLoRA (4-bit) | 3.5–4 GB | 6–10 GB |
Figures are approximate: activations grow with sequence length and batch size. For length 1024, memory may increase by 20–30%.
Where the method falls short of full fine-tuning
LoRA and QLoRA are not a silver bullet. They may fail if:
- The task requires deep knowledge change (e.g., learning a new language or significant restructuring of the subject area). Low rank limits expressiveness.
- There is a lot of diverse data — full fine-tuning may yield better results by tuning all weights.
- Maximum precision is critical — quantization in QLoRA introduces noise, and in tasks with strict requirements (medicine, finance), it's better to use LoRA on fp16 or full fine-tuning.
Also important: LoRA adapters can be merged with the base model for inference, but this loses the ability to quickly switch between different adapters.
On what data training collapses
Common causes of failure:
- Too little data or it's monotonous. If you have 50 examples of one type, the model will overfit and respond template-like. For style tasks, you need at least several hundred diverse examples.
- Wrong format. The model expects a certain template (e.g., "### Question: ... ### Answer: ..."). If the format during training is different from inference, quality drops.
- Class imbalance. In classification or entity extraction tasks, a skew toward one class will bias predictions.
- Noisy labels. The model will learn labeling errors and reproduce them.
Alternatives: RAG and prompt engineering
Fine-tuning is not always the best choice. If the task is to give the model access to external knowledge (documents, database), RAG (Retrieval-Augmented Generation) is often more effective: you load relevant fragments into the prompt. It's cheaper, faster, and doesn't require training. Prompt engineering with few-shot examples can solve a style task without changing weights.
Trade-off: fine-tuning gives consistent behavior and doesn't require passing long context, but costs money and time. RAG is easier to update but depends on search quality. Prompt engineering is the fastest start but limited by context length and may be unstable.
Learning paths
Self-study: articles, Hugging Face PEFT documentation, tutorials. Plus — free, minus — no feedback, easy to miss important details (e.g., how to properly set up evaluation or deployment). Time — months of trial and error.
Courses: a structured path with practical case studies. On asibiont.com, the Fine-tuning LLM course covers LoRA, QLoRA, dataset preparation, hyperparameters, evaluation, and deployment, including RLHF and DPO. Learning is personalized to your level, with feedback via AI chat. It saves time but requires financial investment.
Practice on real projects: the best way to consolidate, but without a mentor you can get stuck at the debugging stage.
Who it suits and who it doesn't
Fine-tuning (and especially LoRA/QLoRA) suits you if you want to customize an open model for a narrow task and you have a GPU (at least 8–12 GB) and some data. It doesn't suit if the task can be solved with a prompt or RAG, if there's no labeled data, or if maximum precision on critical data is required — then consider full fine-tuning or other approaches.
If you're ready to understand the mechanics and avoid typical mistakes, the Fine-tuning LLM course provides a structured path: from data preparation to deployment. You can start on the course page.
You can start here: LLM Fine-Tuning.