Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods
Qing Li, Wen-gang Zhou, Zhenbo Lu
May 8, 2022
Citations
0
Citations
Journal
ArXiv
Full text
Semantic Scholar
Key Takeaway Key Takeaway: SDQ-CAL improves sample efficiency and mitigates overestimation bias in actor-critic Reinforcement Learning, achieving state-of-the-art performance in continuous control tasks.
Abstract
—Actor-critic Reinforcement Learning (RL) algo- rithms have achieved impressive performance in continuous control tasks. However, they still suffer two nontrivial obstacles, i.e., low sample efficiency and overestimation bias. To this end, we propose Simultaneous Double Q-learning with Conservative Advantage Learning (SDQ-CAL). Our SDQ-CAL boosts the Double Q-learning for off-policy actor-critic RL based on a modification of the Bellman optimality operator with Advantage Learning. Specifically, SDQ-CAL improves sample efficiency by modifying the reward to facilitate the distinction from experience between the optimal actions and the others. Besides, it mitigates the overestimation issue by updating a pair of critics simultaneously upon double estimators. Extensive experiments reveal that our algorithm realizes less biased value estimation and achieves state-of-the-art performance in a range of continuous control benchmark tasks. We release the source code of our method at: