Long Short-Term Memory (LSTM) networks are a specialized form of Recurrent Neural Network developed to address difficulties that standard RNNs experience when learning long-term dependencies.
A standard RNN repeatedly updates its hidden state. During training, gradients passing through many time steps can become extremely small or excessively large. This makes it difficult to preserve and learn information from distant parts of a sequence.
An LSTM introduces a memory cell and a collection of gates that regulate information flow. The main components are:
1. Forget gate: Determines which information from the previous cell state should be retained or discarded.
2. Input gate: Controls how much new information should be written into the cell state.
3. Candidate cell state: Represents potential new information to store.
4. Cell state: Carries information through the sequence.
5. Output gate: Controls which information from the cell state contributes to the new hidden state.
The forget gate uses a sigmoid function to produce values between zero and one. Values near zero indicate that information should largely be forgotten, while values near one indicate that information should be retained. The input and output gates provide additional control over memory updates and hidden-state generation.
LSTMs are trained through Backpropagation Through Time. Their gating structure improves the ability to preserve useful information, although it does not guarantee that every long-term dependency will be learned. They can also require more computation than simpler recurrent architectures.
Long Short-Term Memory Networks (LSTMs)
Long Short-Term Memory networks are a specialized type of recurrent neural network designed to learn long-term dependencies using a memory cell and gating mechanisms.
f_t = σ(W_f [h_(t-1), x_t] + b_f)
i_t = σ(W_i [h_(t-1), x_t] + b_i)
o_t = σ(W_o [h_(t-1), x_t] + b_o)
C_t = f_t ⊙ C_(t-1) + i_t ⊙ C̃_t
h_t = o_t ⊙ tanh(C_t)
Explanation
Variables
| Symbol | Meaning |
|---|---|
f_t | Forget gate activation |
i_t | Input gate activation |
o_t | Output gate activation |
C_t | Current cell state |
C_(t-1) | Previous cell state |
C̃_t | Candidate cell-state information |
h_t | Current hidden state |
x_t | Current input |
σ | Sigmoid activation function |
⊙ | Element-wise multiplication |
Example or application
In a language-processing task, an LSTM may retain the subject of a sentence while reading several words before encountering the verb. This helps it use information that appeared earlier in the sequence.
Text
LSTMs use input, forget, and output gates to control information flow through a memory cell. This architecture helps preserve useful information across many time steps.
Key Points
LSTMs are specialized recurrent neural networks.
A memory cell carries information across time steps.
The forget gate controls retention of previous information.
The input gate controls the addition of new information.
The output gate controls information exposed through the hidden state.
LSTMs help address, but do not completely eliminate, long-term dependency problems.
Applications
Speech recognition
Language modeling
Time-series forecasting
Text classification
Sequence labeling
Handwriting recognition
Learning Outcomes
Explain why LSTMs were developed.
Identify the three main gates.
Describe how the cell state is updated.
Compare LSTMs with standard RNNs.
Practice Questions
What problem do LSTMs address?
What is the function of the forget gate?
How do the input and output gates differ?
Why is the cell state important?
Related Concepts
Recurrent Neural Networks
GRU
Sigmoid Activation
Backpropagation Through Time
Sequence Modeling
Common Mistakes
Confusing the cell state with the hidden state.
Assuming the forget gate directly controls the output.
Claiming that LSTMs completely eliminate vanishing gradients.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Hochreiter, S., and Schmidhuber, J., Long Short-Term Memory, Neural Computation, 1997.
- Goodfellow, Bengio, and Courville, Deep Learning, MIT Press, 2016.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning