✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-112

Recurrent Neural Networks (RNNs)

Recurrent Neural Networks are neural architectures designed to process sequential data by maintaining information from previous time steps through recurrent connections.

h_t = tanh(W_xh x_t + W_hh h_{t-1} + b_h)

Explanation

A Recurrent Neural Network (RNN) is a neural network designed for data in which the order of elements matters. Examples include sentences, speech recordings, time-series measurements, and sequences of events.

Unlike a basic feedforward neural network, an RNN contains recurrent connections. At each time step, it receives the current input and a hidden state from the previous time step. It combines these to calculate a new hidden state, which acts as a representation of information gathered from the sequence so far.

A typical RNN operation follows these steps:

1. Receive the current input element.
2. Combine it with the previous hidden state.
3. Apply a nonlinear transformation to calculate the new hidden state.
4. Generate an output when required.
5. Pass the new hidden state to the next time step.

The same weights are reused across time steps, allowing the network to process sequences of different lengths. RNNs can support several input-output arrangements, including one-to-one, one-to-many, many-to-one, and many-to-many tasks.

RNNs are trained using Backpropagation Through Time (BPTT), which applies gradient calculations across the unfolded sequence. However, standard RNNs can experience vanishing or exploding gradients, making it difficult to learn long-term dependencies. Architectures such as LSTMs and GRUs were developed to improve the handling of longer sequences.

Variables

SymbolMeaning

x_t

Input at time step t

h_t

Hidden state at time step t

h_(t-1)

Previous hidden state

W_xh

Input-to-hidden weight matrix

W_hh

Hidden-to-hidden recurrent weight matrix

b_h

Hidden-layer bias

tanh

Hyperbolic tangent activation function

Example or application

An RNN used for language modeling reads a sentence word by word. Its hidden state carries information from earlier words and helps predict which word is likely to come next.

Text

RNNs process sequences one element at a time. Their hidden state carries information from earlier elements, allowing the network to learn patterns that depend on sequence order.

Key Points

  • RNNs are designed for sequential data.

  • The hidden state carries information between time steps.

  • The same recurrent weights are reused throughout a sequence.

  • RNNs can process variable-length sequences.

  • Backpropagation Through Time is commonly used for training.

  • Vanishing gradients can limit long-term memory.

Applications

  • Language modeling

  • Speech recognition

  • Time-series forecasting

  • Sequence classification

  • Text generation

  • Sensor data analysis

Learning Outcomes

  • Explain how recurrent connections work.

  • Describe the purpose of a hidden state.

  • Understand how an RNN processes a sequence.

  • Identify limitations of standard RNN architectures.

Practice Questions

  • What makes an RNN different from a feedforward neural network?

  • What information does the hidden state contain?

  • What is Backpropagation Through Time?

  • Why can standard RNNs struggle with long sequences?

Related Concepts

  • Artificial Neural Networks

  • Hidden States

  • Backpropagation Through Time

  • LSTM

  • GRU

  • Sequence Modeling

Common Mistakes

  • Assuming an RNN processes all sequence elements independently.

  • Confusing hidden states with model parameters.

  • Assuming standard RNNs can easily remember information over arbitrarily long sequences.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Goodfellow, Bengio, and Courville, Deep Learning, MIT Press, 2016.
  • Elman, J. L., Finding Structure in Time, Cognitive Science, 1990.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning