Architektur

Feed-Forward Network (FFN)

Definition

The fully connected sublayer within each transformer block that applies two linear transformations with a non-linearity between them. FFN layers are responsible for storing factual associations and account for the majority of a transformer's parameters.

Verwandte Begriffe

Transformer

The dominant neural network architecture for language, vision, and multimodal AI, introduced in the 2017 "Attention Is All You Need" paper. Transformers use self-attention to process all tokens in parallel, enabling training on internet-scale data and powering every major LLM in use today.

Residual Connection

A shortcut connection that adds a layer's input directly to its output, allowing gradients to flow more easily during training and enabling very deep networks. Residual connections are essential in transformers and were pioneered in the ResNet image model family.

Layer Normalization

A technique that normalises activations across the feature dimension within a single layer, stabilising training of deep networks. Layer norm is standard in transformer architectures, enabling effective training at billion-parameter scale.

Hilfe beim Verständnis von KI Benötigt?

Buchen Sie eine Beratung, um zu besprechen, wie KI-Konzepte auf Ihre Herausforderungen anwendbar sind.