A Recurrent Neural Network (RNN) is a neural network designed for data in which the order of elements matters. Examples include sentences, speech recordings, time-series measurements, and sequences of events.
Unlike a basic feedforward neural network, an RNN contains recurrent connections. At each time step, it receives the current input and a hidden state from the previous time step. It combines these to calculate a new hidden state, which acts as a representation of information gathered from the sequence so far.
A typical RNN operation follows these steps:
1. Receive the current input element.
2. Combine it with the previous hidden state.
3. Apply a nonlinear transformation to calculate the new hidden state.
4. Generate an output when required.
5. Pass the new hidden state to the next time step.
The same weights are reused across time steps, allowing the network to process sequences of different lengths. RNNs can support several input-output arrangements, including one-to-one, one-to-many, many-to-one, and many-to-many tasks.
RNNs are trained using Backpropagation Through Time (BPTT), which applies gradient calculations across the unfolded sequence. However, standard RNNs can experience vanishing or exploding gradients, making it difficult to learn long-term dependencies. Architectures such as LSTMs and GRUs were developed to improve the handling of longer sequences.
Recurrent Neural Networks (RNNs)
Recurrent Neural Networks are neural architectures designed to process sequential data by maintaining information from previous time steps through recurrent connections.
h_t = tanh(W_xh x_t + W_hh h_{t-1} + b_h)
Explanation
Variables
| Symbol | Meaning |
|---|---|
x_t | Input at time step t |
h_t | Hidden state at time step t |
h_(t-1) | Previous hidden state |
W_xh | Input-to-hidden weight matrix |
W_hh | Hidden-to-hidden recurrent weight matrix |
b_h | Hidden-layer bias |
tanh | Hyperbolic tangent activation function |
Example or application
An RNN used for language modeling reads a sentence word by word. Its hidden state carries information from earlier words and helps predict which word is likely to come next.
Text
RNNs process sequences one element at a time. Their hidden state carries information from earlier elements, allowing the network to learn patterns that depend on sequence order.
Key Points
RNNs are designed for sequential data.
The hidden state carries information between time steps.
The same recurrent weights are reused throughout a sequence.
RNNs can process variable-length sequences.
Backpropagation Through Time is commonly used for training.
Vanishing gradients can limit long-term memory.
Applications
Language modeling
Speech recognition
Time-series forecasting
Sequence classification
Text generation
Sensor data analysis
Learning Outcomes
Explain how recurrent connections work.
Describe the purpose of a hidden state.
Understand how an RNN processes a sequence.
Identify limitations of standard RNN architectures.
Practice Questions
What makes an RNN different from a feedforward neural network?
What information does the hidden state contain?
What is Backpropagation Through Time?
Why can standard RNNs struggle with long sequences?
Related Concepts
Artificial Neural Networks
Hidden States
Backpropagation Through Time
LSTM
GRU
Sequence Modeling
Common Mistakes
Assuming an RNN processes all sequence elements independently.
Confusing hidden states with model parameters.
Assuming standard RNNs can easily remember information over arbitrarily long sequences.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Goodfellow, Bengio, and Courville, Deep Learning, MIT Press, 2016.
- Elman, J. L., Finding Structure in Time, Cognitive Science, 1990.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning