Architecture

Residual Connection

Definition

A shortcut connection that adds a layer's input directly to its output, allowing gradients to flow more easily during training and enabling very deep networks. Residual connections are essential in transformers and were pioneered in the ResNet image model family.

Related Terms

Layer Normalization

A technique that normalises activations across the feature dimension within a single layer, stabilising training of deep networks. Layer norm is standard in transformer architectures, enabling effective training at billion-parameter scale.

Deep Learning

A subset of machine learning based on artificial neural networks with multiple layers. Deep learning can learn complex patterns from large amounts of data and is particularly effective for image recognition, speech processing, and natural language understanding.

Transformer

The dominant neural network architecture for language, vision, and multimodal AI, introduced in the 2017 "Attention Is All You Need" paper. Transformers use self-attention to process all tokens in parallel, enabling training on internet-scale data and powering every major LLM in use today.

Knowing the Terms Is Step One. Applying Them Is Step Two.

Book a 30-minute call to discuss how these AI concepts translate to your specific industry and business challenges.