✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-157

Actor-Critic Methods

Actor-Critic methods combine policy-based and value-based reinforcement learning. The actor selects actions, while the critic evaluates how good states or actions are and provides feedback for improving the policy.

δ_t = r_{t+1} + γV(s_{t+1}) - V(s_t)

Explanation

Actor-Critic algorithms use two cooperating learning components. The actor represents the policy and selects actions according to the current state. The critic estimates a value function, such as the state value V(s) or action value Q(s,a), to evaluate the expected future return. The critic's estimate helps calculate a temporal-difference error or advantage signal, which guides the actor's policy update. A common advantage estimate compares the value of taking a particular action with the expected value of the state. This can reduce the variance of policy-gradient learning compared with using complete episode returns directly. Many actor-critic methods update both components during training. Examples include Advantage Actor-Critic (A2C), Asynchronous Advantage Actor-Critic (A3C), and Soft Actor-Critic (SAC), which uses an entropy-regularized objective. Actor-Critic methods can support discrete and continuous action settings depending on the algorithm. Their performance depends on the accuracy of the critic, the quality of the policy updates, and suitable training settings. If the critic estimates are inaccurate, the actor may receive misleading feedback.

Variables

SymbolMeaning

δ_t

Temporal-difference error

r_{t+1}

Reward received after the current action

γ

Discount factor

V(s_t)

Critic's estimated value of the current state

V(s_{t+1})

Critic's estimated value of the next state

Example or application

A simulated robot uses an actor to select movement commands and a critic to estimate whether its current position is likely to lead to successful navigation. The critic's feedback helps the actor improve its future decisions.

Text

The actor learns what action to take, and the critic estimates the value of the current state or action. Their cooperation can produce more efficient policy learning than relying on raw episode returns alone.

Key Points

  • Actor-Critic combines policy learning with value estimation.

  • The actor selects actions according to a policy.

  • The critic estimates state or action values.

  • Temporal-difference errors can guide policy improvement.

  • Advantage estimates can reduce gradient variance.

  • Critic errors can negatively affect actor learning.

Applications

  • Robotic control

  • Continuous-action learning

  • Autonomous navigation research

  • Game-playing agents

  • Simulated physical control

  • Adaptive decision systems

Learning Outcomes

  • Explain the actor and critic roles.

  • Compare Actor-Critic with purely value-based and policy-based methods.

  • Interpret the temporal-difference error.

  • Understand how critic feedback improves policy learning.

  • Recognize common Actor-Critic algorithms.

Practice Questions

  • What does the actor do?

  • What does the critic estimate?

  • What is a temporal-difference error?

  • How can advantage estimates help policy learning?

  • Name two Actor-Critic algorithms.

Related Concepts

  • Policy Gradient

  • Value Function

  • Temporal-Difference Learning

  • Advantage Function

  • A2C

  • SAC

Common Mistakes

  • Reversing the responsibilities of the actor and critic.

  • Assuming the critic directly chooses actions.

  • Treating the critic's estimate as a guaranteed true value.

  • Ignoring how critic estimation errors affect policy updates.

  • Assuming every Actor-Critic algorithm uses an identical objective.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Konda, V. R., and Tsitsiklis, J. N. (2000). Actor-Critic Algorithms. Advances in Neural Information Processing Systems.
  • Mnih, V. et al. (2016). Asynchronous Methods for Deep Reinforcement Learning. International Conference on Machine Learning.
  • Haarnoja, T. et al. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning