The Transformer is a neural network architecture introduced in the paper Attention Is All You Need. It relies primarily on attention mechanisms rather than recurrent connections to process sequential information.
A traditional recurrent network processes a sequence step by step. A Transformer can process many sequence positions in parallel during training because its main attention operations do not require each position to wait for the previous position's hidden state.
The original Transformer contains an encoder and a decoder:
1. Encoder: Processes the input sequence and creates contextual representations.
2. Decoder: Generates output tokens using previously generated tokens and information from the encoder.
Important components include:
- Token embeddings: Convert discrete tokens into numerical vectors.
- Positional encodings: Provide information about token positions because attention alone does not inherently represent sequence order.
- Multi-head self-attention: Allows each position to combine information from other relevant positions.
- Masked self-attention: Prevents decoder positions from accessing future target tokens during autoregressive training.
- Cross-attention: Allows decoder representations to use information from encoder outputs.
- Feedforward networks: Apply transformations independently to each sequence position.
- Residual connections and layer normalization: Support stable information flow and training.
Many modern Transformer models use only an encoder, only a decoder, or modified combinations of the original architecture. Encoder-only models are commonly used for contextual representation and understanding tasks. Decoder-only models are commonly used for autoregressive text generation. Encoder-decoder models are useful for sequence-to-sequence tasks such as translation.
Although Transformers support parallel training across sequence positions, attention computation and memory usage can become expensive for long sequences. Modern implementations use optimizations and alternative attention methods to address these limitations.
Transformer Architecture
The Transformer is a neural network architecture based primarily on attention mechanisms, enabling efficient sequence processing and learning relationships between distant elements.
Attention(Q,K,V) = softmax(QKᵀ / √d_k)V
Explanation
Variables
| Symbol | Meaning |
|---|---|
Q | Query matrix |
K | Key matrix |
V | Value matrix |
d_k | Key-vector dimension |
softmax | Attention-score normalization function |
Example or application
A Transformer-based translation system can process a source sentence through an encoder and use a decoder to generate its translation one token at a time, using attention to access relevant source information.
Text
Transformers use attention layers, feedforward networks, positional information, and residual connections to process sequences. Their architecture supports parallel processing of sequence positions during training.
Key Points
Transformers are based primarily on attention mechanisms.
The original architecture contains an encoder and a decoder.
Positional information helps represent token order.
Multi-head attention captures different relationships.
Masked attention prevents access to future target tokens.
Residual connections and normalization support training.
Modern Transformer models may use only part of the original architecture.
Applications
Large language models
Machine translation
Text summarization
Question answering
Code generation
Image understanding
Multimodal artificial intelligence
Learning Outcomes
Identify the major components of a Transformer.
Explain the role of positional encoding.
Differentiate encoder-only, decoder-only, and encoder-decoder models.
Understand the function of masked and cross-attention.
Practice Questions
Why were Transformers developed as an alternative to recurrent architectures?
What is the purpose of positional information?
How does masked self-attention work?
What is the difference between encoder-only and decoder-only Transformers?
Related Concepts
Attention Mechanisms
Self-Attention
Positional Encoding
Large Language Models
Neural Machine Translation
Common Mistakes
Assuming every Transformer must contain both an encoder and a decoder.
Assuming attention automatically understands token order without positional information.
Confusing masked self-attention with cross-attention.
Assuming parallel training means autoregressive generation produces every token simultaneously.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Vaswani, A. et al., Attention Is All You Need, Advances in Neural Information Processing Systems, 2017.
- Goodfellow, Bengio, and Courville, Deep Learning, MIT Press, 2016.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning