✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-119

Transformer Architecture

The Transformer is a neural network architecture based primarily on attention mechanisms, enabling efficient sequence processing and learning relationships between distant elements.

Attention(Q,K,V) = softmax(QKᵀ / √d_k)V

Explanation

The Transformer is a neural network architecture introduced in the paper Attention Is All You Need. It relies primarily on attention mechanisms rather than recurrent connections to process sequential information.

A traditional recurrent network processes a sequence step by step. A Transformer can process many sequence positions in parallel during training because its main attention operations do not require each position to wait for the previous position's hidden state.

The original Transformer contains an encoder and a decoder:

1. Encoder: Processes the input sequence and creates contextual representations.
2. Decoder: Generates output tokens using previously generated tokens and information from the encoder.

Important components include:

- Token embeddings: Convert discrete tokens into numerical vectors.
- Positional encodings: Provide information about token positions because attention alone does not inherently represent sequence order.
- Multi-head self-attention: Allows each position to combine information from other relevant positions.
- Masked self-attention: Prevents decoder positions from accessing future target tokens during autoregressive training.
- Cross-attention: Allows decoder representations to use information from encoder outputs.
- Feedforward networks: Apply transformations independently to each sequence position.
- Residual connections and layer normalization: Support stable information flow and training.

Many modern Transformer models use only an encoder, only a decoder, or modified combinations of the original architecture. Encoder-only models are commonly used for contextual representation and understanding tasks. Decoder-only models are commonly used for autoregressive text generation. Encoder-decoder models are useful for sequence-to-sequence tasks such as translation.

Although Transformers support parallel training across sequence positions, attention computation and memory usage can become expensive for long sequences. Modern implementations use optimizations and alternative attention methods to address these limitations.

Variables

SymbolMeaning

Q

Query matrix

K

Key matrix

V

Value matrix

d_k

Key-vector dimension

softmax

Attention-score normalization function

Example or application

A Transformer-based translation system can process a source sentence through an encoder and use a decoder to generate its translation one token at a time, using attention to access relevant source information.

Text

Transformers use attention layers, feedforward networks, positional information, and residual connections to process sequences. Their architecture supports parallel processing of sequence positions during training.

Key Points

  • Transformers are based primarily on attention mechanisms.

  • The original architecture contains an encoder and a decoder.

  • Positional information helps represent token order.

  • Multi-head attention captures different relationships.

  • Masked attention prevents access to future target tokens.

  • Residual connections and normalization support training.

  • Modern Transformer models may use only part of the original architecture.

Applications

  • Large language models

  • Machine translation

  • Text summarization

  • Question answering

  • Code generation

  • Image understanding

  • Multimodal artificial intelligence

Learning Outcomes

  • Identify the major components of a Transformer.

  • Explain the role of positional encoding.

  • Differentiate encoder-only, decoder-only, and encoder-decoder models.

  • Understand the function of masked and cross-attention.

Practice Questions

  • Why were Transformers developed as an alternative to recurrent architectures?

  • What is the purpose of positional information?

  • How does masked self-attention work?

  • What is the difference between encoder-only and decoder-only Transformers?

Related Concepts

  • Attention Mechanisms

  • Self-Attention

  • Positional Encoding

  • Large Language Models

  • Neural Machine Translation

Common Mistakes

  • Assuming every Transformer must contain both an encoder and a decoder.

  • Assuming attention automatically understands token order without positional information.

  • Confusing masked self-attention with cross-attention.

  • Assuming parallel training means autoregressive generation produces every token simultaneously.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Vaswani, A. et al., Attention Is All You Need, Advances in Neural Information Processing Systems, 2017.
  • Goodfellow, Bengio, and Courville, Deep Learning, MIT Press, 2016.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning