✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-118

Attention Mechanisms

Attention mechanisms allow neural networks to assign different importance weights to different parts of an input when generating a representation or prediction.

Attention(Q,K,V) = softmax(QKᵀ / √d_k)V

Explanation

An attention mechanism is a neural network operation that dynamically determines which parts of available information are most relevant to a particular computation.

In sequence processing, a model may need to use information from different positions in an input sequence. Instead of relying only on a single fixed-size representation, attention can calculate weights that determine how strongly each position contributes to the current output.

A widely used attention formulation uses three components:

1. Query (Q): Represents the information currently being searched for.
2. Key (K): Represents information used to calculate how relevant each available item is to the query.
3. Value (V): Contains the information that is combined using the calculated attention weights.

The query is compared with keys using a similarity or compatibility function. The resulting scores are normalized, commonly using softmax, to produce attention weights. These weights are then used to calculate a weighted sum of the values.

Scaled dot-product attention divides query-key dot products by the square root of the key dimension before applying softmax. This scaling helps prevent large dot-product values from making the softmax distribution excessively concentrated.

Attention can be used in different ways. Self-attention relates positions within the same sequence, while cross-attention relates queries from one sequence or representation to keys and values from another. Multi-head attention performs several attention operations in parallel, allowing the model to learn different relationships.

Variables

SymbolMeaning

Q

Query matrix

K

Key matrix

V

Value matrix

d_k

Dimension of each key vector

Kᵀ

Transpose of the key matrix

softmax

Function that converts scores into normalized weights

Example or application

In a language translation task, when generating the translation of a particular word, an attention mechanism can assign higher weights to relevant words in the source sentence rather than treating every word as equally important.

Text

Attention computes relationships between a query and a collection of keys, then uses the resulting weights to combine corresponding values. This allows a model to focus on the most relevant information.

Key Points

  • Attention dynamically weights available information.

  • Queries are compared with keys.

  • Values are combined using attention weights.

  • Softmax commonly converts scores into normalized weights.

  • Scaling helps stabilize dot-product attention.

  • Self-attention and cross-attention serve different purposes.

  • Multi-head attention learns multiple representation relationships.

Applications

  • Machine translation

  • Text summarization

  • Language modeling

  • Image captioning

  • Speech processing

  • Multimodal learning

Learning Outcomes

  • Explain the roles of queries, keys, and values.

  • Describe how attention weights are calculated.

  • Understand scaled dot-product attention.

  • Differentiate self-attention from cross-attention.

Practice Questions

  • What are queries, keys, and values?

  • Why is softmax used in attention?

  • What is the purpose of scaling by the key dimension?

  • How does self-attention differ from cross-attention?

Related Concepts

  • Transformer Architecture

  • Softmax

  • Neural Networks

  • Sequence Modeling

  • Multi-Head Attention

Common Mistakes

  • Assuming attention weights are learned as fixed parameters for every input rather than computed dynamically.

  • Confusing keys with values.

  • Assuming attention automatically provides a complete explanation of a model's reasoning.

  • Confusing self-attention with cross-attention.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Vaswani, A. et al., Attention Is All You Need, Advances in Neural Information Processing Systems, 2017.
  • Bahdanau, D., Cho, K., and Bengio, Y., Neural Machine Translation by Jointly Learning to Align and Translate, 2015.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning