Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics

The overestimation bias is one of the major impediments to accurate\noff-policy learning. This paper investigates a novel way to alleviate the\noverestimation bias in a continuous control setting. Our method---Truncated\nQuantile Critics, TQC,---blends three ideas: distributional representation of a\ncritic, truncation of critics prediction, and ensembling of multiple critics.\nDistributional representation and truncation allow for arbitrary granular\noverestimation control, while ensembling provides additional score\nimprovements. TQC outperforms the current state of the art on all environments\nfrom the continuous control benchmark suite, demonstrating 25% improvement on\nthe most challenging Humanoid environment.\n