Fine-tuning a large language model (LLM) used to be a privilege reserved for teams with deep pockets and even deeper GPU racks. Full fine-tuning of a 7B parameter model requires something like 112 GB of VRAM in mixed precision — that's an A100 80GB plus some clever offloading. But with the rise of parameter-efficient fine-tuning (PEFT) methods like LoRA and QLoRA, the game has changed. You can now adapt a 7B model on a single consumer GPU (e.g., RTX 3090 with 24 GB) and get results that rival full fine-tuning. This is a game-changer for startups, researchers, and hobbyists.
But here's the catch: while the math behind LoRA is simple, the art of crafting effective prompts for fine-tuning is not. The quality of your training data, the formatting of your prompts, and the decoding strategies you choose can make or break your fine-tuned model. This article is a curated collection of 10 powerful prompts that you can use (and adapt) for your own LoRA/QLoRA fine-tuning projects. Each prompt is designed to help you with a specific step: from data preparation to evaluation, from avoiding catastrophic forgetting to mixing data sources. Let's dive in.
1. Data Generation: Creating Synthetic Training Data
Task: Generate diverse, high-quality training examples from a small seed set.
Prompt:
You are an expert data engineer. I have a dataset of {N} examples for fine-tuning a model to {task}. Each example is in the format:
### Instruction: {user query}
### Response: {assistant answer}
Please generate {M} new, diverse examples that cover edge cases, variations in phrasing, and different difficulty levels. Ensure the responses are factually correct, helpful, and follow the same format. Here are the seed examples:
{seed_examples}
Output the new examples in a JSON array, each with 'instruction' and 'response' fields.
Example Result:
[
{
"instruction": "What is the capital of France?",
"response": "The capital of France is Paris."
},
{
"instruction": "Explain the concept of supply and demand.",
"response": "Supply and demand is an economic model that determines the price of a good ..."
}
]
Why it works: This prompt leverages the model's own knowledge to expand your dataset, which is especially useful when you have limited human-labeled data. By specifying the format and asking for diversity, you get a richer training set.
2. Formatting: Converting Raw Text to Instruction-Response Pairs
Task: Transform unstructured text into a structured instruction dataset.
Prompt:
Convert the following raw text into a set of instruction-response pairs suitable for fine-tuning a language model. The instructions should be clear and self-contained, and the responses should directly address the instruction without extra fluff.
Raw text:
{raw_text}
Output as JSON: [{"instruction": "...", "response": "..."}]
Example Result:
[
{
"instruction": "What are the main features of the new smartphone?",
"response": "The new smartphone features a 6.7-inch OLED display, a 108MP camera, and a 5000mAh battery."
},
{
"instruction": "How does the battery life compare to previous models?",
"response": "The battery life is improved by 20% compared to the previous model due to a more efficient processor."
}
]
Why it works: Many fine-tuning tasks start with messy data (e.g., scraped docs). This prompt helps you structure it automatically, saving hours of manual work.
3. Data Quality: Cleaning and Deduplicating Your Dataset
Task: Identify and remove duplicates, near-duplicates, and low-quality examples.
Prompt:
You are a data quality specialist. I have a dataset of {N} instruction-response pairs. Please:
1. Remove exact duplicates.
2. Identify near-duplicates (e.g., same instruction but slightly different response) and keep the best one.
3. Flag examples where the response is off-topic, incomplete, or contains factual errors.
4. Suggest improvements for flagged examples.
Dataset:
{dataset}
Output a cleaned dataset in JSON format, along with a report of removed/flagged examples.
Example Result:
{
"cleaned_dataset": [...],
"report": {
"duplicates_removed": 15,
"near_duplicates_merged": 5,
"flagged_for_review": [
{
"instruction": "What is the speed of light?",
"response": "About 300,000 km/s.",
"issue": "Missing context: in a vacuum"
}
]
}
}
Why it works: Quality beats quantity. Removing noisy samples prevents the model from learning hallucinated patterns.
4. Hyperparameter Selection: Choosing the Right LoRA Rank and Alpha
Task: Decide the optimal LoRA configuration for your task.
Prompt:
I am fine-tuning a {base_model} (e.g., Llama-3-8B) for {task} using LoRA. My training set has {N} examples. Suggest appropriate values for:
- lora_r (rank)
- lora_alpha (scaling parameter)
- lora_dropout
- target_modules (which layers to apply LoRA to)
- learning_rate
- batch_size
- num_epochs
Also explain the trade-offs for each choice.
Example Result:
For a task with 5k examples, a common starting point is:
- lora_r=16, lora_alpha=32 (alpha = 2*r is a good rule of thumb)
- lora_dropout=0.05
- target_modules: ["q_proj", "v_proj"] (for Llama architecture)
- learning_rate=2e-4 (for AdamW)
- batch_size=8 (adjust based on VRAM)
- num_epochs=3
Higher rank allows more expressiveness but risks overfitting on small datasets. Alpha scales the LoRA weights; a higher alpha gives more influence to the adapter.
Why it works: This prompt gives you a starting point based on best practices, and the explanation helps you understand the underlying principles.
5. Prompt Formatting: Designing a Consistent Template for Training
Task: Create a consistent prompt template that the model will learn to follow.
Prompt:
Design a prompt template for fine-tuning a model to {task}. The template should be clear, consistent, and include placeholders for variables. For instance, for a question-answering task, you might use:
### Instruction: {question}
### Context: {context}
### Response: {answer}
Provide 10 examples of how different types of inputs would look with this template.
Example Result:
### Instruction: What is the capital of France?
### Context: France is a country in Europe.
### Response: The capital is Paris.
### Instruction: Summarize the article.
### Context: [article text]
### Response: The article discusses ...
Why it works: A consistent template helps the model learn to map inputs to outputs effectively. It also makes inference easier later.
6. Data Mixing: Combining Multiple Datasets Effectively
Task: Blend different datasets (e.g., general and task-specific) to prevent catastrophic forgetting.
Prompt:
I want to fine-tune {base_model} for {task} but also want to preserve its general knowledge. I have two datasets: A (task-specific, {N_A} examples) and B (general instruction data, {N_B} examples). Recommend a mixing ratio and sampling strategy. Also suggest whether to use a learning rate scheduler with warmup.
Example Result:
A common approach is to mix in 10-20% general data. For example, with 5k task-specific and 50k general examples, you might sample 10% of the general data per epoch. Use a cosine decay scheduler with a warmup of 10% of steps. This helps the model retain general capabilities while learning the new task.
Why it works: Mixing data prevents the model from overfitting to the narrow task distribution and losing its pretrained knowledge.
7. Training: Writing the Code for LoRA/QLoRA Fine-Tuning
Task: Generate a training script using Hugging Face PEFT and Transformers.
Prompt:
Write a Python script to fine-tune a model using QLoRA with the following specifications:
- Base model: {model_name} (e.g., meta-llama/Llama-2-7b-hf)
- Use 4-bit quantization (BitsAndBytesConfig)
- LoRA config: r=16, alpha=32, dropout=0.05, target_modules=['q_proj','v_proj']
- Training arguments: batch_size=4, learning_rate=2e-4, num_epochs=3
- Use Hugging Face Trainer
Include code for loading the dataset, tokenization, and saving the adapter.
Example Result:
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
model_name = "meta-llama/Llama-2-7b-hf"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=bnb_config)
model = prepare_model_for_kbit_training(model)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules=['q_proj', 'v_proj'],
task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)
# ... load dataset, tokenize, and train
training_args = TrainingArguments(
output_dir="./lora-llama2",
per_device_train_batch_size=4,
learning_rate=2e-4,
num_train_epochs=3,
logging_steps=10,
save_strategy="epoch",
fp16=True
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=dataset,
)
trainer.train()
model.save_pretrained("./lora-adapter")
Why it works: This prompt gives you a working starting point, and you can adapt it to your specific model and data.
8. Evaluation: Testing Your Fine-Tuned Model
Task: Generate a set of evaluation prompts to test the model's performance on your task.
Prompt:
Create 20 evaluation prompts to test a model fine-tuned for {task}. The prompts should cover:
- Basic cases
- Edge cases
- Adversarial cases (e.g., ambiguous queries, out-of-scope requests)
Provide the prompts in a JSON array.
Example Result:
[
"What is the capital of France?",
"What is the capital of a country that does not exist?",
"Explain quantum physics in simple terms.",
"Write a poem about a cat."
]
Why it works: A robust evaluation set helps you catch weaknesses and ensure your fine-tuning achieved the desired behavior.
9. Debugging: Analyzing Failure Cases
Task: Identify why the model fails on certain inputs and suggest fixes.
Prompt:
I have a fine-tuned model that fails on the following examples: {list of failures}. Analyze the possible reasons (e.g., overfitting, underfitting, data imbalance, prompt format mismatch) and suggest concrete steps to fix them, such as adding more data, adjusting hyperparameters, or changing the prompt template.
Example Result:
The failures suggest that the model has overfit to the training data and cannot generalize to paraphrased queries. Try:
1. Increasing lora_dropout to 0.1.
2. Adding more diverse training examples.
3. Using a lower learning rate (1e-4) and training for fewer epochs.
4. Incorporating a validation set to monitor overfitting.
Why it works: This prompt turns the model into a debugging assistant, helping you iterate faster.
10. Deployment: Exporting and Using Your LoRA Adapter
Task: Generate instructions for merging the LoRA adapter with the base model and deploying it.
Prompt:
Explain how to merge a LoRA adapter with a base model for deployment. Include code for merging using the PEFT library, saving the merged model, and loading it for inference. Also discuss considerations for serving (e.g., quantization, batching).
Example Result:
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model = PeftModel.from_pretrained(base_model, "./lora-adapter")
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./merged-model")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
inputs = tokenizer("What is the capital of France?", return_tensors="pt")
outputs = merged_model.generate(**inputs)
print(tokenizer.decode(outputs[0]))
For serving, you can use vLLM or TGI with the merged model for high throughput.
**Why it works:** Deployment is often the last mile; this prompt ensures you don't get stuck.
## Extra: Combining LoRA with RLHF
**Task:** Get guidance on using LoRA for reinforcement learning from human feedback (RLHF).
**Prompt:**
I want to fine-tune a model using RLHF but with LoRA to save compute. Explain the steps: supervised fine-tuning (SFT), reward modeling, and proximal policy optimization (PPO), and how to integrate LoRA at each stage. Provide any code snippets for the PPO step using TRL.
**Example Result:**
```python
from trl import PPOConfig, PPOTrainer
# ... load model, tokenizer, and reward model
config = PPOConfig(
model_name="./lora-sft",
learning_rate=1.4e-5,
batch_size=16,
)
ppo_trainer = PPOTrainer(config=config, model=model, tokenizer=tokenizer)
for query, response in dataset:
# compute reward and run PPO step
Why it works: RLHF with LoRA is a hot topic; this prompt gives you a high-level roadmap.
Conclusion
Fine-tuning LLMs with LoRA and QLoRA is no longer a black art. With the right prompts, you can systematically approach data preparation, training, and evaluation. These 10 prompts are just a starting point — adapt them to your specific use case, and you'll be fine-tuning like a pro in no time. Remember, the key is to iterate: use the evaluation prompt to test, the debugging prompt to fix, and the data generation prompt to expand your dataset when needed.
If you want to go deeper, there are excellent resources: the Hugging Face PEFT documentation and the QLoRA paper are must-reads. And if you're looking for a guided learning path, consider courses that cover these topics in detail. Happy fine-tuning!
Comments