Q-Learning is a temporal-difference learning algorithm used to learn an action-value function without requiring a known environment transition model. The Q-value Q(s,a) estimates the expected discounted return obtained by taking action a in state s and then following an optimal policy. After taking an action, the agent receives a reward and observes the next state. It updates the previous Q-value using the difference between its current estimate and a target formed from the observed reward plus the best estimated future action value. This difference is called the temporal-difference error. The learning rate controls how strongly new information changes the existing estimate. The discount factor determines the importance of future rewards. In tabular Q-Learning, values are stored in a table indexed by states and actions. Exploration strategies such as epsilon-greedy action selection help the agent discover useful actions. Q-Learning is off-policy because its update uses the maximum estimated value of the next state's actions, even when the agent explores by selecting a different action. Tabular Q-Learning is most suitable for manageable state and action spaces. Large or continuous spaces often require function approximation, which introduces additional stability and convergence challenges.
Q-Learning
Q-Learning is a model-free, off-policy reinforcement learning algorithm that learns the expected value of taking an action in a particular state and then following an optimal policy.
Q(s,a) ← Q(s,a) + α[r + γ max_{a'} Q(s',a') - Q(s,a)]
Explanation
Variables
| Symbol | Meaning |
|---|---|
Q(s,a) | Current estimated value of taking action a in state s |
α | Learning rate |
r | Reward received after taking the action |
γ | Discount factor |
s' | Next state |
a' | Possible action in the next state |
max Q(s',a') | Highest estimated action value in the next state |
Example or application
A simulated robot learns to reach a target in a grid. It updates Q-values for movements such as left, right, up, and down. Over many episodes, movements that lead efficiently toward the target receive higher estimated values.
Text
Q-Learning stores or approximates action values called Q-values. It updates these values using rewards and estimates of the best future action, gradually learning which actions are most beneficial.
Key Points
Q-Learning is model-free and off-policy.
It learns action values rather than directly storing a policy.
Updates use observed rewards and estimated future values.
The learning rate controls the size of each update.
Exploration is necessary to discover useful actions.
Tabular methods become difficult in very large state spaces.
Applications
Grid-world navigation
Game-playing agents
Resource allocation
Simple robotic simulations
Scheduling problems
Adaptive decision systems
Learning Outcomes
Explain the purpose of Q-Learning.
Interpret the Q-value update equation.
Understand learning rate and discount factor.
Explain why Q-Learning is off-policy.
Describe the role of exploration.
Practice Questions
What does a Q-value represent?
Why is Q-Learning called model-free?
What is the role of the learning rate?
Why does the update use the maximum next-state Q-value?
How does epsilon-greedy exploration work?
Related Concepts
Reinforcement Learning
Temporal-Difference Learning
Q-Values
Exploration and Exploitation
Deep Q-Networks
Markov Decision Processes
Common Mistakes
Confusing Q-Learning with supervised learning.
Treating the reward as the complete Q-value.
Forgetting that the update uses the next state's estimated maximum value.
Assuming that exploration is unnecessary after the first update.
Assuming tabular Q-Learning scales easily to arbitrary continuous environments.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Watkins, C. J. C. H., and Dayan, P. (1992). Technical Note: Q-Learning. Machine Learning.
- Sutton, R. S., and Barto, A. G. (2018). Reinforcement Learning: An Introduction, Second Edition. MIT Press.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning