A policy describes how an agent selects actions based on its current state. In policy-based reinforcement learning, the policy is represented by a parameterized function, commonly a neural network. The learning algorithm adjusts these parameters to increase expected cumulative reward. A deterministic policy selects an action directly from the state, while a stochastic policy produces a probability distribution over available actions. Stochastic policies are useful when exploration is needed or when several actions may be appropriate. Policy Gradient methods estimate how changes to policy parameters affect expected returns and update the policy in a direction that improves performance. The REINFORCE algorithm is a classic Monte Carlo policy-gradient method. Policy-based approaches can naturally represent continuous actions, such as steering angles or motor-control values, where enumerating every possible action would be impractical. However, policy-gradient methods can have high variance and may require many interactions with the environment. Baselines and advantage estimates are commonly used to reduce variance and improve learning stability.
Policy-Based Reinforcement Learning
Policy-based reinforcement learning directly learns a policy that maps states to actions or action probabilities. It is especially useful for problems involving continuous actions or stochastic decision-making.
∇θ J(θ) = E[∇θ log πθ(a|s) G_t]
Explanation
Variables
| Symbol | Meaning |
|---|---|
θ | Parameters of the policy |
J(θ) | Expected performance or return of the policy |
πθ(a|s) | Probability of choosing action a in state s under policy parameters θ |
G_t | Return associated with the sampled action |
∇θ | Gradient with respect to policy parameters |
Example or application
A simulated robotic arm uses a policy network to select continuous joint-control values. Training adjusts the policy parameters so that the arm becomes more successful at reaching a target position.
Text
Instead of learning only action values, policy-based methods optimize the decision strategy itself. A policy may select a single action or produce a probability distribution over possible actions.
Key Points
Policy-based methods optimize the policy directly.
Policies can be deterministic or stochastic.
Policy gradients adjust parameters to improve expected returns.
Continuous action spaces are a major use case.
High gradient variance can make learning difficult.
Baselines and advantage estimates can improve training.
Applications
Robotic arm control
Continuous motor control
Autonomous navigation
Game-playing agents
Adaptive resource allocation
Simulated control environments
Learning Outcomes
Define a policy in reinforcement learning.
Explain policy-based learning.
Distinguish deterministic and stochastic policies.
Understand the basic policy-gradient idea.
Identify strengths and limitations of policy-based methods.
Practice Questions
What does a policy represent?
How do policy-based methods differ from value-based methods?
What is a stochastic policy?
Why are policy gradients useful for continuous actions?
What problem do baseline functions help reduce?
Related Concepts
Policy Gradient
REINFORCE
Value-Based Learning
Actor-Critic Methods
Continuous Control
Advantage Function
Common Mistakes
Assuming that a policy must always select deterministic actions.
Confusing a policy with a value function.
Assuming policy-gradient updates always improve performance on every training step.
Ignoring the variance of return-based gradient estimates.
Assuming policy-based methods are limited to discrete action spaces.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Williams, R. J. (1992). Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning.
- Sutton, R. S., and Barto, A. G. (2018). Reinforcement Learning: An Introduction, Second Edition. MIT Press.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning