Deep Q-Networks extend Q-Learning by using a neural network as a function approximator. The network receives a state representation, such as a vector or processed image, and predicts Q-values for the available actions. Training aims to make these predictions closer to a target based on the observed reward and the maximum estimated value of the next state. Two important techniques from the original DQN approach are experience replay and a target network. Experience replay stores past transitions in a replay buffer and samples batches from that buffer, reducing dependence on the immediate order of experiences. A target network provides more stable target estimates because its parameters are updated less frequently than those of the main network. DQN was famously demonstrated on Atari games using visual observations. It is designed primarily for discrete action spaces; continuous control generally requires other methods or modifications. Although DQN can handle much larger state representations than a simple Q-table, training can still be unstable and sensitive to hyperparameters, reward design, and data distribution.
Deep Q-Networks (DQN)
A Deep Q-Network (DQN) combines Q-Learning with a deep neural network that approximates action values. It allows reinforcement learning to operate in larger state spaces, including environments with high-dimensional observations such as images.
y = r + γ max_{a'} Q(s',a'; θ⁻)
Explanation
Variables
| Symbol | Meaning |
|---|---|
y | Target Q-value used for training |
r | Observed reward |
γ | Discount factor |
s' | Next state |
a' | Possible next action |
Q | Neural network estimate of action value |
θ⁻ | Parameters of the target network |
Example or application
A DQN agent learns to play an Atari game by receiving screen images as state observations and predicting the value of available button actions. It improves its decisions using rewards and previously collected gameplay experiences.
Text
DQN uses a neural network to estimate Q-values instead of storing every state-action value in a table. It learns from collected experiences and uses techniques such as experience replay and a target network to improve training stability.
Key Points
DQN uses a neural network to approximate Q-values.
It extends Q-Learning to larger state representations.
Experience replay reuses stored transitions.
A target network helps stabilize learning targets.
The original DQN approach is primarily designed for discrete actions.
Training may remain unstable despite these techniques.
Applications
Atari game-playing systems
Simulated robot navigation
Discrete-action control tasks
Resource management
Scheduling simulations
Research in deep reinforcement learning
Learning Outcomes
Define a Deep Q-Network.
Explain how a neural network approximates Q-values.
Describe experience replay.
Explain the purpose of a target network.
Identify the limitations of DQN.
Practice Questions
How does DQN differ from tabular Q-Learning?
What is the purpose of a replay buffer?
Why does DQN use a target network?
What types of action spaces are suited to standard DQN?
Why can deep reinforcement learning training be unstable?
Related Concepts
Q-Learning
Deep Learning
Experience Replay
Target Network
Temporal-Difference Learning
Deep Reinforcement Learning
Common Mistakes
Assuming DQN stores every Q-value in a table.
Confusing the target network with the replay buffer.
Assuming standard DQN naturally handles continuous action spaces.
Believing experience replay alone guarantees stable convergence.
Ignoring the influence of reward scaling and hyperparameters.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Mnih, V. et al. (2015). Human-level Control through Deep Reinforcement Learning. Nature.
- Sutton, R. S., and Barto, A. G. (2018). Reinforcement Learning: An Introduction, Second Edition. MIT Press.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning