✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-156

Policy-Based Reinforcement Learning

Policy-based reinforcement learning directly learns a policy that maps states to actions or action probabilities. It is especially useful for problems involving continuous actions or stochastic decision-making.

∇θ J(θ) = E[∇θ log πθ(a|s) G_t]

Explanation

A policy describes how an agent selects actions based on its current state. In policy-based reinforcement learning, the policy is represented by a parameterized function, commonly a neural network. The learning algorithm adjusts these parameters to increase expected cumulative reward. A deterministic policy selects an action directly from the state, while a stochastic policy produces a probability distribution over available actions. Stochastic policies are useful when exploration is needed or when several actions may be appropriate. Policy Gradient methods estimate how changes to policy parameters affect expected returns and update the policy in a direction that improves performance. The REINFORCE algorithm is a classic Monte Carlo policy-gradient method. Policy-based approaches can naturally represent continuous actions, such as steering angles or motor-control values, where enumerating every possible action would be impractical. However, policy-gradient methods can have high variance and may require many interactions with the environment. Baselines and advantage estimates are commonly used to reduce variance and improve learning stability.

Variables

SymbolMeaning

θ

Parameters of the policy

J(θ)

Expected performance or return of the policy

πθ(a|s)

Probability of choosing action a in state s under policy parameters θ

G_t

Return associated with the sampled action

∇θ

Gradient with respect to policy parameters

Example or application

A simulated robotic arm uses a policy network to select continuous joint-control values. Training adjusts the policy parameters so that the arm becomes more successful at reaching a target position.

Text

Instead of learning only action values, policy-based methods optimize the decision strategy itself. A policy may select a single action or produce a probability distribution over possible actions.

Key Points

  • Policy-based methods optimize the policy directly.

  • Policies can be deterministic or stochastic.

  • Policy gradients adjust parameters to improve expected returns.

  • Continuous action spaces are a major use case.

  • High gradient variance can make learning difficult.

  • Baselines and advantage estimates can improve training.

Applications

  • Robotic arm control

  • Continuous motor control

  • Autonomous navigation

  • Game-playing agents

  • Adaptive resource allocation

  • Simulated control environments

Learning Outcomes

  • Define a policy in reinforcement learning.

  • Explain policy-based learning.

  • Distinguish deterministic and stochastic policies.

  • Understand the basic policy-gradient idea.

  • Identify strengths and limitations of policy-based methods.

Practice Questions

  • What does a policy represent?

  • How do policy-based methods differ from value-based methods?

  • What is a stochastic policy?

  • Why are policy gradients useful for continuous actions?

  • What problem do baseline functions help reduce?

Related Concepts

  • Policy Gradient

  • REINFORCE

  • Value-Based Learning

  • Actor-Critic Methods

  • Continuous Control

  • Advantage Function

Common Mistakes

  • Assuming that a policy must always select deterministic actions.

  • Confusing a policy with a value function.

  • Assuming policy-gradient updates always improve performance on every training step.

  • Ignoring the variance of return-based gradient estimates.

  • Assuming policy-based methods are limited to discrete action spaces.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Williams, R. J. (1992). Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning.
  • Sutton, R. S., and Barto, A. G. (2018). Reinforcement Learning: An Introduction, Second Edition. MIT Press.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning