How LLMs Work: From Tokens to Transformers

Tokenization Algorithms

Tokenization Examples

Vocabulary Size by Model

Training Objective: Cross-Entropy Loss

Given a sequence of tokens [t₁, t₂, ..., tₙ], the model is trained to maximise the log-likelihood of each token given all preceding tokens:

Each forward pass processes a full sequence and produces a loss at every position in parallel (teacher forcing). During inference, tokens are generated autoregressively, one at a time.

Chinchilla Scaling Laws

Hoffmann et al. (2022) showed that previous large models (GPT-3, Gopher) were undertrained — too many parameters for too few tokens. The Chinchilla optimal ratio is:

Major Models at a Glance

Pretraining Data

Distributed Training

Stage 1: Supervised Fine-Tuning (SFT)

The base model is fine-tuned on a dataset of (instruction, ideal response) pairs, written or curated by human annotators. This teaches the model the instruction-following format. The training objective is identical to pretraining (cross-entropy), but the dataset is small (tens of thousands of examples) and high quality. After SFT, the model can follow instructions but may still be untruthful or harmful.

The Full RLHF Pipeline

graph LR
  A[Pretrained LLM] --> B[SFT on Instruction Data]
  B --> C[SFT Model]
  C --> D[Generate Completions]
  D --> E[Human Preference Labels]
  E --> F[Train Reward Model]
  F --> G[RLHF with PPO]
  G --> H[Aligned LLM]

DPO: Direct Preference Optimisation

Rafailov et al. (2023) showed that the RLHF objective can be optimised directly on the policy model without a separate reward model or RL loop. Given pairs of preferred and rejected responses, DPO reparametrises the reward as a function of the policy and reference model log-probabilities:

DPO is simpler, more stable, and cheaper than RLHF with PPO. Most open-source aligned models (Llama 4 Instruct, Mistral Instruct) use DPO or a variant (SimPO, IPO) for the preference alignment stage.

KV-Cache: Making Inference Practical

During autoregressive generation, the model recomputes attention for every token at every step — naively O(n²) per step. The KV-cache stores the key and value tensors from all previous tokens. On each new step, only the new token's Q, K, V are computed and the cached K, V are appended. This reduces generation from O(n²) to O(n) per new token.

Context Window Comparison

*Effective length: the window over which the model reliably retrieves information. “Lost in the middle” research shows performance degrades for content placed in the middle of very long contexts.

Quantisation: Smaller Models, Similar Quality

Reducing the numerical precision of model weights shrinks VRAM and speeds up computation. Modern techniques (GPTQ, AWQ, GGUF) apply quantisation non-uniformly, protecting the most sensitive weights.

Inference Servers

PagedAttention & Continuous Batching

vLLM's PagedAttention (Kwon et al., 2023) manages KV-cache memory like virtual memory in an OS — splitting it into fixed-size pages allocated non-contiguously. This eliminates fragmentation and wasted reservation, enabling 2–24× more throughput than HuggingFace Transformers at equal GPU memory.

Continuous batching allows new requests to join an in-flight batch as soon as a sequence finishes, rather than waiting for the entire batch. This dramatically improves GPU utilisation under variable-length workloads.

Standard Benchmarks

Model Benchmark Scores

LLM-as-Judge Pattern

For open-ended tasks where reference answers don't exist, a powerful LLM can score responses using a structured rubric. MT-Bench and Chatbot Arena use this approach. The key risk is position bias (the judge prefers answers appearing first) and verbosity bias (longer answers score higher regardless of quality).

from openai import OpenAI

client = OpenAI()

def llm_judge(question: str, answer: str, rubric: str) -> dict:
    prompt = f"""You are an expert evaluator. Score the following answer on a 1-10 scale.

Question: {question}

Answer: {answer}

Rubric: {rubric}

Respond with JSON: {{"score": <int>, "reasoning": "<str>", "strengths": ["..."], "weaknesses": ["..."]}}"""

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
        temperature=0.0,
    )
    import json
    return json.loads(response.choices[0].message.content)

result = llm_judge(
    question="Explain the attention mechanism in transformers.",
    answer="Attention computes a weighted sum of values...",
    rubric="Accuracy (4pt), Clarity (3pt), Completeness (3pt)",
)
print(f"Score: {result['score']}/10 — {result['reasoning']}")