That would be very inefficient.
Instead, we can use the relationship between the Q and the V from the Bellman optimality equation:So, we can rewrite the advantage as:Then, we only have to use one neural network for the V function (parameterized by v above).
So we can rewrite the update equation as:This is the Advantage Actor Critic.
Advantage Actor Critic (A2C) v.
Asynchronous Advantage Actor Critic (A3C)The Advantage Actor Critic has two main variants: the Asynchronous Advantage Actor Critic (A3C) and the Advantage Actor Critic (A2C).
A3C was introduced in Deepmind’s paper “Asynchronous Methods for Deep Reinforcement Learning” (Mnih et al, 2016).
In essence, A3C implements parallel training where multiple workers independently in parallel environments to update a global value function—hence “asynchronous.
” The idea of having asynchronous actors is that it helps to effectively explore the state space.
High Level Architecture of A3C (image taken from GroundAI blog post)A2C is like A3C but without the asynchronous part; this means a single-worker variant of the A3C.
It was empirically found that A2C produces comparable performance to A3C while being more efficient.
According to this OpenAI blog post, researchers aren’t completely sure how the asynchronicity benefits learning:After reading the paper, AI researchers wondered whether the asynchrony led to improved performance (e.
“perhaps the added noise would provide some regularization or exploration?“), or if it was just an implementation detail that allowed for faster training with a CPU-based implementation.
Anyhow, we will implement the A2C in this post as it is more simple in implementation and better for the computer — multi-processes are daunting.
Implementing A2CSo, recall the new update equation, replacing the discounted cumulative award from vanilla policy gradients with the Advantage function:On each learning step, we update both the Actor parameter (with policy gradients and advantage value), and the Critic parameter (with minimizing the mean squared error with the Bellman update equation).
Let’s see how this looks like in code:Below are the includes and hyper-parameters:First, let’s start with implementing the Actor Critic Network with the following configurations:The main loop and update loop, as outlined above:Running the code, we can see how the performance improves:Blue: Raw rewards; Orange: Smoothed rewardsFind the full implementation here:https://github.
com/thechrisyoon08/Reinforcement-Learning/References:UC Berkeley CS294 Lecture SlidesCarnegie Mellon University CS10703 Lecture SlidesLilian Weng’s Post on Policy Gradient AlgorithmsJerry Liu’s answer on Quora Post “Why does the policy gradient method have a high variance”Naver D2 RLCode Lecture VideoOpenAI blog post on A2C and ACKTRDiagram from GroundAI’s blog post “Figure 4: Schematic representation of Asynchronous Advantage Actor Critic algorithm (A3C) algorithm.
”Next Post:Most likely on reinforcement learning for deterministic policies and implementing the DDPG (Deep deterministic policy gradients) algorithm.
Thanks for reading!.