Resources/Technical Guide

Technical Deep Dive

AI Skills & Fine-Tuning Guide

A complete guide to teaching AI models new skills: supervised fine-tuning (SFT), LoRA/QLoRA, RLHF, DPO, GRPO, model distillation, model merging, and evaluation. From concept to production — with working code at every step.

11 Sections

45 min read

Production-Ready Code

March 2026

The Fine-Tuning Landscape

Pretraining gives a model broad knowledge of the world, but only one skill: predicting the next token. The model has seen Wikipedia, code, books, and the web — but it doesn't know to be helpful, to follow instructions, or to refuse dangerous requests. Fine-tuning is the process of teaching these behaviors after pretraining.

The industry has converged on a standard training ladder that all major frontier models (GPT-4o, Claude Opus 4.6, Llama 4, Gemini 2.5) follow. Each stage builds on the previous — you cannot skip SFT and jump straight to RLHF.

The Training Ladder

graph LR
  A[Raw Text Corpus] -->|Pretraining cross-entropy| B[Base Model]
  B -->|Supervised Fine-Tuning| C[Instruction-Following Model]
  C -->|RLHF / DPO / GRPO| D[Aligned Model]
  D -->|Evaluation & Red-teaming| E[Production Model]

Pretraining

Self-supervised next-token prediction on massive corpora. Encodes world knowledge.

SFT

Supervised fine-tuning on instruction-response pairs. Teaches the model to be helpful.

Preference Alignment

RLHF, DPO, or GRPO on human preference data. Makes outputs safe and preferred.

Evaluation

Automated benchmarks + red-teaming. Catch regressions before shipping.

Fine-tuning vs Prompt Engineering

Prompt engineering makes behaviors conditional (they only appear when the prompt says so). Fine-tuning makes behaviors default — the model exhibits them consistently without being told. At scale, this reliability difference is significant.

Supervised Fine-Tuning (SFT)

SFT trains the model to predict assistant tokens given a conversation context. The key detail is loss masking: the cross-entropy loss is computed only on assistant tokens, not on the system prompt or user turns. This prevents the model from “learning” the user's side of the conversation.

Data Formats

Three formats dominate the SFT landscape. ChatML has become the most widely adopted due to its unambiguous special tokens.

text (ChatML format)

<|im_start|>system
You are a helpful AI assistant specialized in European AI regulation.
<|im_end|>
<|im_start|>user
What are the key obligations under the EU AI Act for high-risk systems?
<|im_end|>
<|im_start|>assistant
High-risk AI systems under the EU AI Act (in force August 2024) must comply with...
<|im_end|>

Key Hyperparameters

Parameter	Typical Value	Notes
Learning rate	2e-5	Lower than pretraining; cosine decay
Epochs	2–3	More epochs → overfitting on small datasets
Batch size (effective)	64–128	Use gradient accumulation for small GPU memory
Warmup ratio	0.1	10% of steps for LR warmup
Max sequence length	2048–8192	Match your inference context window

SFT with trl SFTTrainer

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer
from datasets import load_dataset
import torch

model_name = "meta-llama/Llama-4-Scout-17B-16E-Instruct"  # 2026: Llama 4 Scout replaces Llama 3.1 8B
model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.bfloat16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

dataset = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")

sft_config = SFTConfig(
    output_dir="./sft-llama-4-scout",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=2e-5,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    logging_steps=10,
    save_strategy="epoch",
    bf16=True,
)

trainer = SFTTrainer(
    model=model,
    args=sft_config,
    train_dataset=dataset,
    processing_class=tokenizer,
)
trainer.train()
trainer.save_model()

Data Quality Beats Quantity

1,000 high-quality, diverse instruction-response pairs consistently outperform 100,000 noisy examples. The top instruction-tuning datasets (Alpaca 52K, WizardLM 196K, OpenHermes 1M, UltraChat 200K) succeed because of curation, not raw size.

Parameter-Efficient Fine-Tuning: LoRA

Full fine-tuning modifies all ~7 billion parameters of a 7B model. At bfloat16 that's 14 GB just for parameter storage, plus gradients and optimizer states. LoRA (Low-Rank Adaptation, Hu et al. 2021) exploits a key empirical observation: weight changes during fine-tuning are low-rank.

Instead of learning a full weight update ΔW ∈ ℝ^(d×k), LoRA learns two small matrices: A ∈ ℝ^(d×r) and B ∈ ℝ^(r×k) where r ≪ min(d, k). At inference, the adapter is folded back: W′ = W + αAB/r. Once merged, there is zero inference overhead.

r = 4

Minimal adaptation (tone, style)

~21M (0.3%)

r = 8

Default — balanced quality

~42M (0.6%)

r = 16

More capacity, domain tasks

~83M (1.0%)

r = 64

Near full fine-tune quality

~335M (4.1%)

Alpha/Rank Ratio

Keep lora_alpha = 2 × r as a starting point (e.g., r=16, alpha=32). This controls the effective learning rate of the adapter. Higher alpha = stronger adaptation; too high = instability.

LoRA with PEFT

python

from peft import LoraConfig, TaskType, get_peft_model

config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    bias="none",
)

model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable params: 83,886,080 || all params: 8,030,261,248 || trainable%: 1.044

# After training, merge adapter back into the base weights
merged = model.merge_and_unload()
merged.save_pretrained("./my-lora-merged")

LoRA vs Full Fine-Tuning Comparison

Method	Trainable Params	GPU RAM (8B)	Quality	Training Speed
Full Fine-Tuning	7B (100%)	~80 GB	Best	Slowest
LoRA r=4	~21M (0.3%)	~16 GB	Good	Fast
LoRA r=16	~83M (1.0%)	~18 GB	Very Good	Fast
LoRA r=64	~335M (4.1%)	~24 GB	Near Full FT	Moderate

DoRA: Weight-Decomposed LoRA

DoRA (Liu et al. 2024) decomposes weight updates into magnitude and direction components, applying separate learning rates to each. It consistently achieves 1–2% better benchmark scores than standard LoRA with no additional inference cost. Available in PEFT via use_dora=True in LoraConfig.

QLoRA: 4-bit Fine-Tuning

Even with LoRA, the base model loaded at bfloat16 requires 16 GB for a 8B model — beyond consumer GPU budgets. QLoRA (Dettmers et al. 2023) solves this by quantizing the frozen base model to 4-bit NormalFloat (NF4) and training LoRA adapters at bfloat16 precision.

NF4 Quantization

NormalFloat4 is information-theoretically optimal for normally-distributed neural network weights. Less error than int4 or fp4.

Paged Optimizers

Optimizer states automatically page to CPU RAM when GPU memory fills, preventing OOM crashes during training.

Double Quantization

Quantizes the quantization constants themselves, saving an extra ~0.5 bits per parameter.

Hardware Requirements

Model	FP16 VRAM	QLoRA VRAM	Min GPU
Llama 4 Scout (17B)	34 GB	10 GB	RTX 4090 24GB
Llama 4 Maverick (70B-class)	140 GB	40 GB	2× A100 40GB
Llama 4 Behemoth (frontier)	800+ GB	~200 GB	8× H100 80GB

QLoRA with bitsandbytes

python

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-4-Maverick-17B-128E-Instruct",  # 2026: Llama 4 Maverick replaces Llama 3.1 70B
    quantization_config=bnb_config,
    device_map="auto",
)
# Now apply LoRA to the 4-bit model — same LoraConfig + get_peft_model as before

Unsloth for Single-GPU Workloads

Unsloth provides custom CUDA kernels for QLoRA that achieve 2× faster training and 50% less VRAM than standard bitsandbytes QLoRA. It supports Llama 4, Llama 3, Mistral, Qwen, and Gemma families and is the go-to choice for single-GPU fine-tuning.

Alignment: RLHF

Reinforcement Learning from Human Feedback (RLHF) was the breakthrough that turned GPT-3 into InstructGPT and eventually GPT-4o. It aligns model behavior to human preferences — not just instruction following, but making outputs genuinely preferred, safe, and helpful.

The Three-Stage Pipeline

Stage 1

SFT Warmup

Fine-tune the base model on a curated set of high-quality instruction-following demos. This creates the starting policy that RLHF will improve.

Stage 2

Reward Model Training

Train a classifier on pairwise human preferences: given two completions (y_w, y_l) to the same prompt, which is better? Loss: log σ(r(x, y_w) − r(x, y_l)).

Stage 3

PPO Optimization

Use Proximal Policy Optimization to maximize the reward model score while staying close to the SFT policy (KL divergence penalty prevents reward hacking).

RLHF Pipeline Diagram

graph LR
  A[Base Model] -->|SFT on demos| B[SFT Model]
  B -->|Sample completions| C[Completion Pairs]
  C -->|Human labelers rank| D[Preference Dataset]
  D -->|Train| E[Reward Model]
  B -->|Initialize policy| F[Policy Model]
  F -->|Rollout + PPO| G[RL Optimization]
  E -->|Score rollouts| G
  G -->|Converged| H[RLHF Model]

PPO Complexity

RLHF with PPO requires four models simultaneously: the policy, the reference policy (frozen SFT model), the reward model, and the value model. This makes RLHF memory-intensive and notoriously difficult to stabilize. Reward hacking (the policy finds ways to score highly without being actually good) is a persistent challenge. This is why DPO has become widely preferred.

Alignment: DPO & GRPO

DPO (Direct Preference Optimization) (Rafailov et al. 2023) eliminates the reward model entirely. It showed mathematically that the optimal RLHF policy can be expressed directly as a function of the preference data, collapsing a three-stage pipeline into a single fine-tuning step.

The DPO loss directly optimizes the policy on preference pairs (prompt, chosen, rejected) using the SFT model as a frozen reference. No PPO, no reward model, no separate RM training data collection.

DPO with trl DPOTrainer

python

from trl import DPOConfig, DPOTrainer
from datasets import load_dataset

# Dataset needs: prompt, chosen, rejected columns
dataset = load_dataset("HuggingFaceH4/ultrafeedback_binarized", split="train_prefs")

dpo_config = DPOConfig(
    output_dir="./dpo-output",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=5e-7,   # much smaller than SFT lr
    beta=0.1,             # KL penalty coefficient
    bf16=True,
)

trainer = DPOTrainer(
    model=sft_model,          # your SFT fine-tuned model
    ref_model=sft_ref_model,  # frozen reference
    args=dpo_config,
    train_dataset=dataset,
    processing_class=tokenizer,
)
trainer.train()

GRPO: DeepSeek's Approach

Group Relative Policy Optimization (GRPO) (used in DeepSeek-R1) eliminates the reference model. For each prompt, it samples multiple outputs and uses the group mean reward as the baseline for advantage estimation. This is cheaper than PPO (no value model) and better suited for reasoning tasks where you can verify correctness programmatically.

Key GRPO advantage:

No reference model required + group-relative rewards = efficient training for verifiable tasks (math, code, structured output).

Alignment Methods Comparison

Method	Compute	Stability	Data Requirements	Notes
RLHF (PPO)	Very High	Low	Human rankings	4 models in memory; reward hacking risk
DPO	Low	High	Preference pairs	No reward model; simpler pipeline
GRPO	Medium	Medium	Rollout samples	No reference model; good for reasoning
SimPO	Low	High	Preference pairs	No reference model; avg log prob reward

Model Distillation

Knowledge distillation trains a small “student” model to mimic a large “teacher” model. The key insight is that the teacher provides soft probability distributions over the vocabulary (logits) rather than one-hot labels. These soft targets encode far more information — they reveal which tokens are semantically similar to the correct answer, giving the student a richer training signal.

The combined loss: L = α × L_CE(hard labels) + (1 − α) × L_KL(student logits ‖ teacher logits). Temperature scaling T > 1 softens the teacher distribution, spreading probability mass across more tokens and making the soft labels even more informative.

Distillation Pipeline

graph TB
  A["Large Teacher (70B)"] -->|"Generate on training data"| B[Soft Logits]
  C[Input Prompt] --> A
  C --> D["Small Student (7B)"]
  B -->|KL Loss| D
  E[Ground Truth] -->|CE Loss| D
  D -->|Both losses| F[Distilled Student]

Response Distillation

Student imitates teacher outputs — generate teacher completions, train student to reproduce them. Used by DeepSeek-R1-Distill to transfer reasoning traces.

Feature Distillation

Match intermediate representations (hidden states, attention patterns) between teacher and student layers. Transfers structural knowledge, not just surface outputs.

Speculative Decoding

A small draft model proposes token sequences; the large model verifies them in parallel. Achieves 2–4x inference speedup with no quality loss.

On-Policy Distillation

The student generates tokens; the teacher scores them. Avoids exposure bias (train-test distribution mismatch) common in offline distillation.

Real-World Distillation Examples

Phi-3 / Phi-4 (Microsoft): distilled from GPT-4 on curated synthetic data
Gemma 2 (Google): distilled from Gemini Ultra; 9B matches much larger models
DeepSeek-R1-Distill: reasoning traces from R1 distilled into 7B / 14B Qwen2.5 models

Model Merging

Model merging combines multiple fine-tuned checkpoints into a single model without any additional training. It's cheap, fast, and surprisingly effective for combining specialized skills — code, math, instruction following — into one deployable model. Merged models frequently appear at the top of the HuggingFace Open LLM Leaderboard.

SLERP— Spherical Linear Interpolation

Smooth interpolation between two model checkpoints in weight space. Treats weights as points on a hypersphere. Best for blending two closely-related models.

Task Arithmetic— Add/Subtract Fine-Tuning Deltas

Compute ΔW = W_FT − W_base for each fine-tuned model, then add deltas together. Lets you compose capabilities or subtract undesirable behaviors.

TIES-Merging— Trim, Elect Signs, Merge

Resolves conflicts between models: trim small-magnitude parameters, elect the dominant sign for each weight, then merge. Handles 3+ models cleanly.

DARE— Drop and Rescale

Randomly drops fine-tuning weight deltas (with probability p) and rescales the survivors to preserve the norm. Reduces interference between models.

MergeKit Configuration (TIES)

yaml

# mergekit config.yaml
models:
  - model: meta-llama/Llama-4-Scout-17B-16E
    parameters:
      weight: 0.4
  - model: ./llama-4-scout-code-finetuned
    parameters:
      weight: 0.3
  - model: ./llama-4-scout-math-finetuned
    parameters:
      weight: 0.3
merge_method: ties
base_model: meta-llama/Llama-4-Scout-17B-16E
parameters:
  density: 0.7
  normalize: true

bash

mergekit-yaml config.yaml ./merged-model --cuda

Frankenmerge (Layer Stacking)

A more radical technique: stack different layers from different model checkpoints — e.g., layers 0–16 from model A, layers 17–32 from model B. Requires no training and can produce surprising capabilities, but needs experimentation to find good layer combinations. MergeKit supports this via the passthrough merge method.

Dataset Preparation

Dataset quality is often a major driver of fine-tuning results, but its importance relative to model choice, objective, training configuration, and evaluation is task-dependent. Poor curation raises failure risk; it does not by itself predict every outcome.

Human-WrittenHighest

Most expensive

Expert-authored examples; highest signal-to-noise ratio. Used for critical behaviors.

GPT-4 / Claude GeneratedHigh

Moderate

Synthetic generation with frontier models. Good for bootstrapping domain coverage at scale.

Evol-Instruct / MagpieGood

Low

Evolve seed instructions into harder, more diverse variants. Used in WizardLM and OpenHermes.

Internet-FilteredVariable

Cheapest

Requires aggressive quality filtering: deduplication, length filter, perplexity filter, safety filter.

ShareGPT Data Format

json

{
  "conversations": [
    {"from": "system", "value": "You are an expert in EU AI regulation."},
    {"from": "human", "value": "Explain the risk categories in the EU AI Act."},
    {"from": "gpt", "value": "The EU AI Act categorizes AI systems into four risk levels..."}
  ]
}

Synthetic Data Generation at Scale

python

from openai import OpenAI  # or use Mistral/Llama locally

client = OpenAI()

def generate_training_example(topic: str, difficulty: str) -> dict:
    prompt = (
        f"Generate a challenging {difficulty}-level question about {topic} "
        "and a comprehensive expert answer."
    )
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.8,
    )
    content = response.choices[0].message.content
    # Parse and structure output (question/answer split)...
    return {"instruction": topic, "response": content}

Recommended Instruction Diversity Distribution

Question Answering

30%

Writing & Summarization

20%

Code Generation & Debugging

20%

Analysis & Reasoning

15%

Other (Translation, Extraction, etc.)

15%

Data Contamination

Test set contamination is the #1 evaluation problem in fine-tuning. If any of your evaluation benchmarks (MT-Bench, HumanEval, MMLU) appear in your training data, your scores will be inflated and meaningless. Always run n-gram overlap checks between your training set and evaluation benchmarks before training.

Evaluation & Iteration

The fine-tuning loop is: train → evaluate on holdout → diagnose failure modes → improve data → retrain. Good evaluation is what transforms trial-and-error into systematic improvement.

MT-Bench

General Quality

80-question multi-turn benchmark across 8 categories (writing, math, coding, etc.). GPT-4 scores each response 1–10.

AlpacaEval

Instruction Following

Win rate of your model vs. a reference model (GPT-4o) as judged by GPT-4o. Fast automated evaluation of instruction-following quality.

IFEval

Format Compliance

Instruction-following accuracy on verifiable constraints (e.g., 'respond in fewer than 100 words'). Strict and loose scoring variants.

HumanEval / MBPP

Code Generation

Code generation benchmarks. Pass@k metric: fraction of problems solved in k attempts. Ground-truth executable test cases.

LLM-as-Judge Pattern

python

import json
from openai import OpenAI

client = OpenAI()

def evaluate_response(question: str, answer: str, judge_model: str = "gpt-4o") -> dict:
    prompt = f"""Rate the following AI assistant response on a scale of 1-10.

Question: {question}
Answer: {answer}

Evaluate: helpfulness (1-10), factuality (1-10), safety (1-10).
Return JSON: {{"helpfulness": N, "factuality": N, "safety": N, "rationale": "..."}}"""

    response = client.chat.completions.create(
        model=judge_model,
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)

Common Evaluation Pitfalls

Length bias: LLM judges tend to prefer longer responses regardless of quality. Calibrate your judge.
Sycophancy: Models score their own outputs higher. Use a different model as judge, or human validation.
Contamination: Benchmark data in training set inflates scores. Always check overlap.
Single-metric traps: Optimizing one metric often hurts others. Track a balanced scorecard.

Experiment Tracking Template

Run	Base Model	Method	Dataset	MT-Bench	AlpacaEval Win%	Notes
v1	Llama-4-Scout	SFT	UltraChat 200K	7.4	70%	Baseline
v2	Llama-4-Scout	SFT+DPO	+ UltraFeedback	8.0	76%	+DPO improved safety
v3	Llama-4-Scout	SFT+DPO (r=16)	+ UltraFeedback	8.1	77%	LoRA r=16 vs full FT

When to Fine-Tune vs RAG vs Prompt Engineering

Fine-tuning is powerful but not always the right tool. The decision depends on what you're trying to change: knowledge, behavior, format, or preferences. Choosing wrong costs weeks of engineering and compute.

Scenario	Best Approach	Why
Need to ground answers in company docs	RAG	Knowledge can change; FT can't update easily
Want consistent tone/style	SFT	Tone is format, not knowledge
Domain-specific terminology usage	SFT + small data	Change default behavior cheaply
Need to handle specific output formats	SFT	Schema adherence is a learned skill
Reduce harmful outputs	DPO / RLHF	Preference alignment directly targets this
Need reasoning capabilities	GRPO or distill from R1	Reasoning patterns are trainable
Add new factual knowledge	RAG (not FT)	FT memorizes, can't cite sources
Reduce API costs at scale	Fine-tune small model	Match big-model quality on narrow task
Prototype / quick experiment	Prompt engineering first	Zero training cost; validate concept first

The LLM Staircase

Start at the bottom. Only climb when the current level is genuinely insufficient — each step adds cost, complexity, and latency.

Prompt Engineering

Free, instant, zero training cost

Few-Shot Examples

Add examples in context

RAG

Ground answers in retrieved docs

SFT

Teach format, style, domain knowledge

DPO / RLHF

Align to preferences and safety

Distillation

Compress to task-specific small model

Fine-Tune When

Consistent tone/format at scale
Domain jargon must be default
Specific output schema required
Reducing API costs on narrow task
Preference/safety alignment needed

Use RAG When

Knowledge changes frequently
Answers need citations/sources
Private/proprietary knowledge base
Large document corpus (>1M tokens)
Need to update without retraining

Avoid Fine-Tuning When

Adding new factual knowledge (use RAG)
Quick prototype or PoC stage
Very small dataset (<100 examples)
No GPU budget available
Prompting already achieves target

Ready to Fine-Tune?

Build Your Custom AI Model

Whether you are evaluating a domain assistant, preference tuning, or distillation, start with a measurable baseline, dataset rights, held-out evaluation, and a deployment plan. Hyperion can scope that evidence path without presenting this guide as proof of shipped client systems.

More Guides