✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-114

Gated Recurrent Units (GRUs)

Gated Recurrent Units are recurrent neural network architectures that use update and reset gates to regulate information flow and learn sequential patterns.

z_t = σ(W_z x_t + U_z h_(t-1) + b_z)
r_t = σ(W_r x_t + U_r h_(t-1) + b_r)
h̃_t = tanh(W_h x_t + U_h(r_t ⊙ h_(t-1)) + b_h)
h_t = (1 - z_t) ⊙ h_(t-1) + z_t ⊙ h̃_t

Explanation

A Gated Recurrent Unit (GRU) is a type of recurrent neural network introduced as an alternative to more complex recurrent architectures such as LSTMs. It was designed to improve the learning of sequential dependencies while using a relatively simple structure.

A GRU uses two main gates:

1. Update gate: Controls how much information from the previous hidden state should be retained and how much new candidate information should be incorporated.
2. Reset gate: Controls how much previous hidden-state information is used when calculating the candidate state.

Unlike a standard LSTM, a GRU does not maintain a separate cell state. Instead, it uses a single hidden state to carry information through the sequence. It also combines some of the memory-control operations found in LSTMs, resulting in fewer gating components.

The reset gate can help the network reduce the influence of previous information when processing a new input. The update gate controls the balance between the previous hidden state and the candidate hidden state.

GRUs are often compared with LSTMs in sequence-learning tasks. GRUs may have fewer parameters and can train faster in some situations, but neither architecture is universally better. Their relative performance depends on the dataset, task, model configuration, and computational resources.

Variables

SymbolMeaning

z_t

Update gate activation

r_t

Reset gate activation

h̃_t

Candidate hidden state

h_t

Current hidden state

h_(t-1)

Previous hidden state

x_t

Current input

σ

Sigmoid activation function

⊙

Element-wise multiplication

Example or application

A GRU used for time-series forecasting can process previous temperature readings and the latest measurement to update its hidden representation before predicting a future temperature.

Text

GRUs simplify the LSTM architecture by combining memory-related operations into a smaller gating structure. They use update and reset gates to control how previous information and new input contribute to the current hidden state.

Key Points

  • GRUs are gated recurrent architectures.

  • They use update and reset gates.

  • GRUs maintain a hidden state without a separate LSTM-style cell state.

  • The update gate controls the balance between old and candidate information.

  • The reset gate influences candidate-state calculation.

  • GRUs can be computationally simpler than LSTMs.

Applications

  • Speech recognition

  • Time-series forecasting

  • Text classification

  • Language modeling

  • Sequence prediction

  • Sensor data processing

Learning Outcomes

  • Identify the two main GRU gates.

  • Explain how a GRU updates its hidden state.

  • Compare GRU and LSTM architectures.

  • Understand why simpler recurrent architectures may be useful.

Practice Questions

  • What are the two main gates in a GRU?

  • How does the reset gate influence information processing?

  • How does the update gate affect the hidden state?

  • What is one structural difference between a GRU and an LSTM?

Related Concepts

  • Recurrent Neural Networks

  • Long Short-Term Memory

  • Hidden States

  • Gating Mechanisms

  • Sequence Modeling

Common Mistakes

  • Assuming GRUs have a separate cell state identical to LSTMs.

  • Confusing the reset gate with the update gate.

  • Assuming GRUs always outperform LSTMs.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Cho, K. et al., Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation, 2014.
  • Chung, J. et al., Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, 2014.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning