Techniques

Top-p Sampling (Nucleus Sampling)

Definition

A decoding strategy that restricts the model's next-token choices to the smallest set of tokens whose cumulative probability exceeds a threshold p. Used alongside temperature, top-p sampling balances output diversity and coherence in production LLM deployments.

Related Terms

Temperature (LLM)

A sampling parameter that controls the randomness of an LLM's output. A temperature of 0 produces deterministic, focused responses; higher values introduce creative variability. Selecting the right temperature is part of operationalising LLMs in production workflows.

Large Language Model (LLM)

AI models trained on vast amounts of text data that can understand and generate human-like text. Examples include GPT-4, Claude, and Llama. LLMs power modern chatbots, content generation, and code assistance tools.

Inference

The process of running a trained model on new data to produce predictions or generated outputs. Inference cost and latency are the dominant operational concerns in production AI, particularly for large generative models that can cost cents per request at scale.

Knowing the Terms Is Step One. Applying Them Is Step Two.

Book a 30-minute call to discuss how these AI concepts translate to your specific industry and business challenges.